search

rom1504 / clip-retrieval

Easily compute clip embeddings and build a clip retrieval system with them
https://rom1504.github.io/clip-retrieval/
MIT License
2.41k stars 211 forks source link

How to use my finetuned open-clip model? #294

Closed czczup closed 1 year ago

czczup commented 1 year ago

Thanks for your awesome work!

I use the open-clip model ('xlm-roberta-base-ViT-B-32', 'laion5b_s13b_b90k') as initialization, and fine-tune the model on my custom image-text dataset.

After that, I got many checkpoints like open_clip/work_dirs/2023_07_24-23_01_12-model_xlm-roberta-base-ViT-B-32-lr_1e-05-b_512-j_8-p_amp/checkpoints/epoch_30.pt. How can I use this model to build a retrieval backend.

czczup commented 1 year ago

As far as I know, I should run clip inference first, like this:

clip-retrieval inference --input_dataset ./data --output_folder ./data_embeddings --clip_model "open_clip:xlm-roberta-base-ViT-B-32"

Is there any way I can load my own finetuned weights here?

czczup commented 1 year ago

Maybe I should modify some code to load my weights.

image
fabiozappo commented 1 year ago

Hello there! I'm currently attempting to execute the end-to-end clip retrieval utilizing a finetuned checkpoint of openclip/xlm-roberta-base-vitb32. However, I'm encountering unusual outcomes on the frontend page. Did you encounter any bugs in the code or face similar challenges that you could provide insights on or recommend how to address?

IF-chan commented 8 months ago

hello how do you load your finetuned weights?