search

microsoft / table-transformer

Table Transformer (TATR) is a deep learning model for extracting tables from unstructured documents (PDFs and images). This is also the official repository for the PubTables-1M dataset and GriTS evaluation metric.
MIT License
2.22k stars 247 forks source link

Loading Weights - not expected behaviour #161

Closed SquareGraph closed 9 months ago

SquareGraph commented 9 months ago

Hey, guys.

The following simple code:

image_processor = AutoImageProcessor.from_pretrained("microsoft/table-transformer-detection")
table_transformer = TableTransformerForObjectDetection.from_pretrained("microsoft/table-transformer-detection")

results in following warning:

Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Some weights of the model checkpoint at microsoft/table-transformer-detection were not used when initializing TableTransformerForObjectDetection: ['model.backbone.conv_encoder.model.layer4.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer2.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer3.0.downsample.1.num_batches_tracked']
- This IS expected if you are initializing TableTransformerForObjectDetection from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TableTransformerForObjectDetection from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

The model is generally working, but... I see a little downgrade in performance with respect to the legacy DERT implementation (the origin of this repo), and I was wondering if those missing weights are the reason. If so, do You know what is the real reason of the warning displayed, and how to initialise it properly? I thought this was the proper way (literally from the HF guide), but it seems there are a little turbulences.

bsmock commented 9 months ago

Hi, if you have any questions about the Hugging Face implementation of Table Transformer models, please ask them in the Hugging Face community forums.

Best, Brandon