Closed davidgxue closed 7 months ago
Evaluation is on hold as the cloud cluster (not local) is on an older version of Weaviate that does not support the new model. I am trying to get access to the remote weaviate cluster from IT. Will be blocked until then
Latest commit: |
e3727ab
|
Status: | ✅ Deploy successful! |
Preview URL: | https://af944af7.ask-astro.pages.dev |
Branch Preview URL: | https://upgrade-text-embedding-model.ask-astro.pages.dev |
new_embed_model_comparison.csv I upgraded the weaviate cluster version and ran some tests using the new embedding model. Very few questions out of this quick test question set had different links/documents retrieved. The vast majority of the documents retrieved had either the same links or links that are highly similar. The ones where there are changes, it made very small differences and generally saw improvement in document relevancy when it comes to retrieval. This is likely due to the fact that we use a hybrid search approach and also use a reranker and prompting an LLM to filter at the end, so changing the embedding model which is only one part of the retrieval process did not make a significant impact.
Since this new model generally performs better according to OpenAI's own metrics as well other researchers AND has significantly lower cost compared to the older V2 ada embedding model, I will go ahead and upgrade this into the newer version.
Description
Technical Changes
schema.json
-- this changes the vectorizer used during ingestion AND during the retrievalTests & Evaluation
Related Issues
closes #286 partially completes #295