Training for ZeroShotImageClassification

[hxydxn]

System Info

• transformers version: 4.35.2
• Platform: Linux-5.15.0-83-generic-x86_64-with-glibc2.29
• Python version: 3.8.10
• Huggingface_hub version: 0.19.1
• Safetensors version: 0.4.0
• Accelerate version: 0.24.1
• Accelerate config: not found
• PyTorch version (GPU?): 2.1.0+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Who can help?

@amyeroberts @pacman100 @muellerz

Information

• [X] The official example scripts
• [X] My own modified scripts

Tasks

• [X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [X] My own task or dataset (give details below)

Reproduction

https://colab.research.google.com/drive/1ugxSI63fQd7YvO4IX5H9YrxyY7NDu2il?usp=sharing

Expected behavior

I am trying to finetune a ZeroShotImageClassification model, particularly geolocal/StreetCLIP. I'm encountering the following error:

I'm restricted to a batch size of 1 for train and eval set, otherwise I will get an error: Attention mask should be of size (1, 1, 8, 8), but is torch.Size([8, 1, 8, 8])

I'm currently finetuning on 1M images across test and train sets and hope to train with 8 A40 GPUs.

I based the flow very heavily from https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb and https://stackoverflow.com/questions/75802931/i-cant-fine-tune-clip-model-from-huggingface

Thank you for the help! Any sample code or advice is appreciated!

[NielsRogge]

Hi,

The example you refer to is not meant for zero-shot image classification. It's meant for supervised image classification, when you have a dataset of (image, label) pairs.

CLIP can be fine-tuned using this script: https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text if you want to improve zero-shot image classification on a certain domain.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.