weaviate / t2v-transformers-models

This is the repo for the container that holds the models for the text2vec-transformers module
BSD 3-Clause "New" or "Revised" License
40 stars 27 forks source link

Batching along different texts, producing one vector per text #59

Open JubilantJerry opened 1 year ago

JubilantJerry commented 1 year ago

Does the logic here implement batch processing of many different independent texts (i.e. independent entries in the Weaviate database)?

I see batching in the sense of splitting a text into sentences, computing vectors for each sentence, and averaging the vectors. I imagine this is done when a single entry contains multiple concepts. But if I understand the code correctly, there is no way to get multiple independent vectors from several independent texts.