search

facebookresearch / MetaCLIP

ICLR2024 Spotlight: curation/training code, metadata, distribution and pre-trained models for MetaCLIP; CVPR 2024: MoDE: CLIP Data Experts via Clustering
Other
1.17k stars 49 forks source link

Finetune the released MetaCLIP models on my own dataset #31

Open ShuxunoO opened 9 months ago

ShuxunoO commented 9 months ago

Hello~

How can I finetune the released MetaCLIP models on my own dataset(Img-Txt paires) and how do I organize the data?

howardhsu commented 9 months ago

Thx, for your interests in this project. It can be very similar to pre-training and here's a suggested steps:

  1. check the pre-training setup: https://github.com/facebookresearch/MetaCLIP#training
  2. derive a new config from b32_400m and add `pretrained="metaclip_400m"
def b32_400m_ft():
    return b32_400m(
        pretrained="metaclip_400m",  # should download pre-trained MetaCLIP ViT-B/32 400M
    )

(we will have a new config system released soon)

  1. depends on your fine-tuning set format, properly set train_data for https://github.com/facebookresearch/MetaCLIP/blob/main/src/training/data.py and we highly recommend use your own dataloader to control randomness instead of built-in one.