search

microsoft / table-transformer

Table Transformer (TATR) is a deep learning model for extracting tables from unstructured documents (PDFs and images). This is also the official repository for the PubTables-1M dataset and GriTS evaluation metric.
MIT License
2.01k stars 231 forks source link

Postprocessing col/row refinement before alignment! #153

Open iamayushanand opened 8 months ago

iamayushanand commented 8 months ago

In the postprocessing script I think there should be an alignment before the refinement by nms as well? Because its possible that two table column objects may not overlap at all before alignment but they might overlap after it. We should remove these overlapping objects too.

https://github.com/microsoft/table-transformer/blob/16d124f616109746b7785f03085100f1f6247575/src/postprocess.py#L118-L134

I can confirm that I got a 0.01 increase in the GRiTS scores on my in-house dataset after adding the necessary changes.

bsmock commented 6 months ago

Hi, thanks for bringing this up, it's a great point.

If I understand you correctly, this situation can happen when two predicted columns are non-overlapping but stacked vertically, one above the other. Is that correct?

This situation seems like it would happen rarely or never for a model that is performing very well on a particular test dataset. But if the model is not performing well, and this situation did happen, then just thinking about it, yes it does seem to me like the current post-processing code could have an undesired outcome.

Can you share the GriTS score you had before and after making the change? An improvement of 0.01 is actually pretty significant. Because models that perform very well can already have GriTS scores above 0.98, I do suspect this improvement will only happen for models whose performance is less than desired on the test data. But until table structure recognition is completely solved, that's a valid case to consider!

Best, Brandon

iamayushanand commented 6 months ago

If I understand you correctly, this situation can happen when two predicted columns are non-overlapping but stacked vertically, one above the other. Is that correct?

Yup, that's right. It can also happen for rows stacked horizontally but not overlapping.

Can you share the GriTS score you had before and after making the change?

Before Change: Grits_Top: 0.9161 Grits_loc: 0.7638 After Change: Grits_Top:0.9257 Grits_loc: 0.7707 The test set consists of 1500 samples.

Thanks for taking note of this! Let me know if a PR is needed to add these changes, would be happy to open one.