Open kl-thamm opened 1 year ago
To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.
beep boop - the Weaviate bot 👋🤖
PS:
Are you already a member of the Weaviate Slack channel?
@kl-thamm if you want us to be able to merge your PR you need to agree to CLA. Simply replying here in a comment "I agree to CLA" will let us to merge to your PR.
@antas-marcin Thanks! I agree to CLA. The problem is that the smoke test runs fine for me locally with the model that I use and some tests passed here as well. If now after the additional commits I made, the tests fail, I would be unsure as how to proceed :)
I had an issue with the t2v-transformers today:
I create embeddings using a sentence-transformers model. One time using the sentence-transformers python library and one time using the t2v-transformers container. The cosine distance of the vectors was up to 0.16.
@antas-marcin greatly and quickly helped me by suggesting setting "T2V_TRANSFORMERS_DIRECT_TOKENIZE=true". This reduced the cosine distance to almost 0.
When looking into what it does i noticed two things:
Regarding 1: Tokenize in the context of this program means splitting the input into sentences and using the transformers tokenizer. I suggest changing
direct_tokenize
toshall_split_in_sentences
or something similar. Actuallyshall_embed_sentence_per_sentence
might even be more precise but that is a bit verbose. Other suggestions very welcome but its just the general idea. Therefore the environment variable becomesT2V_SHALL_SPLIT_IN_SENTENCES
. (see the commit)Regarding 2: For me this setting seems to be important and should be documented somewhere. I don't know how to suggest edits for the documentation so I am writing down what I think what would be helpful here:
Environment Settings _T2V_SHALL_SPLIT_INSENTENCES: If not set, will use true. If set to false, use raw input.
By default all t2v-transformers split the input into sentences using nltk with english interpunctuation and calculates the mean over the sentence embeddings. This allows to embed inputs of arbitrary length. But it will produce unexpected results if your text does not have the expected interpunctuation. Embedding on a per sentence level could at least theoretically degrade the embedding model's performance in case it produces better results with longer inputs.
(Also could this be significantly slower? Doing it sentence by sentence than doing a larger input at once?).