DEIT -Some weights of the model checkpoint at facebook/deit-base-patch16-224 were not used when initializing DeiTMode

kanlions commented 2 years ago

System Info

Nvidia 3080
Windows 11

Who can help?

@NielsRogge The warning I get is this and a big list of layers You are using a model of type vit to instantiate a model of type deit. This is not supported for all configurations of models and can yield errors. Some weights of the model checkpoint at facebook/deit-base-patch16-224 were not used when initializing DeiTModel:

Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)

Reproduction

I did the following from transformers import DeiTFeatureExtractor, DeiTModel feature_extractor = DeiTFeatureExtractor.from_pretrained("facebook/deit-base-patch16-224") model = DeiTModel.from_pretrained("facebook/deit-base-patch16-224") inputs_ref = feature_extractor(images=im_ref, return_tensors="pt") with torch.no_grad(): outputs_ref = model(**inputs_ref) last_hidden_states_ref = outputs_ref.last_hidden_state

The warning I get is this and a big list of layers You are using a model of type vit to instantiate a model of type deit. This is not supported for all configurations of models and can yield errors. Some weights of the model checkpoint at facebook/deit-base-patch16-224 were not used when initializing DeiTModel:

The images are simple scenes only.

Expected behavior

Is this warning something I should take seriously. I need only weights from a pre trained model to use.
The warning is 'You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference'

I am new to Hugging face community and I appreciate any help. I tried to follow the guidelines, please revert to me in case of more information

NielsRogge commented 2 years ago

Hi,

That's because the checkpoint you are loading (facebook/deit-base-patch16-224) needs to be loaded in a ViTModel/ViTForImageClassification rather than a DeiTModel. As explained in the docs of DeiT, the authors also trained more efficient ViT models:

The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into ViTModel or ViTForImageClassification. Techniques like data augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset (while only using ImageNet-1k for pre-training). There are 4 variants available (in 3 different sizes): facebook/deit-tiny-patch16-224, facebook/deit-small-patch16-224, facebook/deit-base-patch16-224 and facebook/deit-base-patch16-384. Note that one should use DeiTFeatureExtractor in order to prepare images for the model.

kanlions commented 2 years ago

@NielsRogge

Thank you very much for responding and I have got my mistake but still I have a confusion. Because Initially I started with 'facebook/deit-base-distilled-patch16-224' from the tutorial mentioned https://huggingface.co/docs/transformers/v4.19.2/en/model_doc/deit#transformers.DeiTFeatureExtractor

Still I get this issue. I understand I am not doing any classification so I get some warning but still I am not able to comprehend this. Any direction will be appreciated. Thanks in advance

Some weights of the model checkpoint at facebook/deit-base-distilled-patch16-224 were not used when initializing DeiTModel: ['distillation_classifier.bias', 'cls_classifier.weight', 'distillation_classifier.weight', 'cls_classifier.bias']

This IS expected if you are initializing DeiTModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing DeiTModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of DeiTModel were not initialized from the model checkpoint at facebook/deit-base-distilled-patch16-224 and are newly initialized: ['deit.pooler.dense.weight', 'deit.pooler.dense.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

NielsRogge commented 2 years ago

Hi,

When initializing a DeiTModel, it won't include the heads on top. For that, you'll need to instantiate a DeiTForImageClassification or DeiTForImageClassificationWithTeacher model.

huggingface / transformers