Multi-Lingual Clip

[yaman]

Hi @vinid, what would happen if we have used a multilingual vit model like: clip-ViT-B-32-multilingual to run the same training on farfetch dataset?

I have a use case of using fashionclip on 6 different languages.

[vinid]

Hello!

It could work. Albeit, it depends on how much you tune the text encoder. What I would do is tuning it by freezing the text encoder part, in this way, embedded representations remain untouched.

You could also try to translate some examples and see how that works!

[yaman]

I think I need to learn more about llm training. Do you have any blog post or other materials to recommend for retraining fashion-clip for multilingual inference?

[vinid]

Of course! A couple of starting points that might be useful to get into CLIP code and CLIP training:

• https://github.com/openai/CLIP/issues/83
• HF Training Pipeline
• OpenCLIP training pipeline (this is the one we recently started using for training, due to the fact that it's easier to scale to multi-GPU setups with this one)

Some blog posts:

• https://towardsdatascience.com/clip-the-most-influential-ai-model-from-openai-and-how-to-use-it-f8ee408958b1
• https://towardsdatascience.com/how-to-train-your-clip-45a451dcd303 (note shameless self plug: I wrote this)

[yaman]

You're terrific, @vinid!

I stumbled upon your article last week; it's in an open tab and on my must-read list. I skimmed through it and appreciate how you've distilled complex fundamentals so effortlessly. It's like a CLIP for dummies :D Thanks for the article.

By the way, I'm crafting a Rust version of fashion-clip (despite being new to Rust) using onnx-runtime, with your model converted via Hugging Face's Optimum (not quantized though) for rapid embedding creation. Here it is; RustEmbed.

I'd value your feedback on whether I've utilized the fashion-clip model correctly and if RustEmbed piques your interest.

[vinid]

Thanks so much!

RustEmbed looks great! I can't give much feedback cause I don't know Rust, but I love the idea! I have added it to our readme: https://github.com/patrickjohncyh/fashion-clip/blob/master/README.md#fun-related-projects