search

microsoft / table-transformer

Table Transformer (TATR) is a deep learning model for extracting tables from unstructured documents (PDFs and images). This is also the official repository for the PubTables-1M dataset and GriTS evaluation metric.
MIT License
2.01k stars 231 forks source link

Table is not getting identified completely after training. #162

Open Nikhilsonawane07 opened 6 months ago

Nikhilsonawane07 commented 6 months ago

I am trying to fine tune table structure recognition model on 10000 financial tables. I am resuming my training from TATR-v1.1-All weights. I have done annotations for all six classes : table, row, column, column headers, span cell, row headers. I have also performed postprocessing on annotated bboxes to align them properly with table. Using padding of 30 for training table images. When I am running inference pipeline with newly trained weights, tables are not fully getting detected. It is detecting table from line below Table column headers as shown in fig. Rows, columns and other things are getting identified correctly. What could be the issue? How can i improve my model? @bsmock Can you please help 1_0_fig_tables

srivatsan-sridhar99 commented 4 months ago

Hey @Nikhilsonawane07 , I am also trying to fine tune the TSR model (microsoft/table-structure-recognition-v1.1-all) using a proprietary annotated dataset of financial tables with just table row and table column as the classes. The results are extremely bad.

I used the fine tuning notebook mentioned in https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR. How exactly did you fine tune the model, using the script in this repo or the ones provided by hugging face?

Also, does the retrained model 'microsoft/table-structure-recognition-v1.1-all' also require table images and bounding boxes to have some sort of padding for fine tuning? is this just applicable for the entire table boundary or even for the cell boundaries? my cell boundaries are tightly cropped around the text.