Open michaeljelly opened 8 months ago
Hi @michaeljelly, thanks a lot for the great suggestion! We are currently focused on making LLMs work better and faster, and I haven't really put much thought on what expanding to embedding models would entail. It could potentially be an awesome contribution from the community! Maybe @xenova and @Vaibhavs10 have more insight on what the effort could look like.
I actually think that embedding models are some of the easiest to add support for. 😅 Once you have BERT models supported, you automatically are able to run most of the models on the MTEB leaderboard. I see the repo already supports the BERT tokenizer, so the only additional step is to add a pooling method (typically mean or CLS pooling) to get sentence embeddings (as opposed to per-token embeddings).
Multimodal embedding models, like CLIP, can come at a later stage :)
IMO, it is okay to open it to the community! I think in terms of overall impact, focusing on LLMs makes more sense since, from an On-device perspective, they'd be used way more used than an embedding model.
I believe, most of the embedding models run quite competitively on CPU already.
Thanks a lot @xenova and @Vaibhavs10 for your comments!
I assigned the good first issue
label just to increase visibility, but it doesn't mean this will be a straightforward PR 😅 Would you be interested in drafting a PR @michaeljelly, if you have the time? We can guide you along the process and help where needed 😄
any updates on this one? @pcuenca, @michaeljelly
most of the embedding models run quite competitively on CPU already
sorry if I can't find this: is there anything like huggingface swift-embedders ?
Hi @pannous! Unfortunately, there's not. This is a very interesting issue that remains open for the community to explore :)
While not huggingface, I found this for embeddings in swift: https://github.com/ZachNagengast/similarity-search-kit
Maybe it's useful for others too and parts can be copied to huggingface? (license compatible?)
cc @ZachNagengast in case he has any feedback on his experience :)
Embeddings models are very useful, and can easily be run on device in terms of hardware specs.
It would be awesome if swift-transformers would work with them.