search

weaviate / t2v-transformers-models

This is the repo for the container that holds the models for the text2vec-transformers module
BSD 3-Clause "New" or "Revised" License
40 stars 27 forks source link

Use Sentence Transformers lib instead of Transformers #31

Closed raam86 closed 7 months ago

raam86 commented 2 years ago

This PR suggests a move to a higher level library (sentence transformers) the behaviour stays the same while increasing performance by about an order of a magnitude.

This is probably due to the promise of this library to be highly optimized for such workloads, they also claim to use and distribute to all gpus automatically but I haven't tested that yet.

I also added support for batching in the form of an input array (instead of a single string) this should help with decoupling the transformation work for large workloads if needed.

weaviate-git-bot commented 2 years ago

To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

beep boop - the SeMI bot 👋🤖

kmulka-bloomberg commented 1 year ago

@raam86 The performance promises of this sound great. Anything stopping this from moving forward? It looks like this needs CLA acceptance from you. But, maybe something else is blocking this?

bobvanluijt commented 1 year ago

Looping in @antas-marcin who might have more context (mostly because this is an older PR)

erikvullings commented 1 year ago

Would be great if the t2v performance could be improved, as I am getting a lot of timeouts when importing and searching at the same time. So is it possible to merge this PR?

kl-thamm commented 1 year ago

Did not look into it this ins great detail but came across a related issue. Afaik sentence-transformers vecetorizes each text in a list as a whole. This PR calls encode with just one string, which sentence-transformers will then just put into a list. From my understanding it will then encode this as a whole. If the input is too long, it will not be split into batches or anything but truncate the input (see e.g. https://github.com/UKPLab/sentence-transformers/issues/181#issuecomment-614693712).

Could it be that the gain of speed would come (at least partly) from the fact that the way this is currently implemented would just encode parts of the text?

antas-marcin commented 7 months ago

The SentenceTransformer vectorizer is already a part of transformers project. Closing this PR.