Support specifying a similarity threshold

wagtail / wagtail-vector-index

Store Wagtail pages & Django models as embeddings in vector databases

https://wagtail-vector-index.readthedocs.io/en/latest/

MIT License

15 stars 10 forks source link

Support specifying a similarity threshold #21

Open tomusher opened 9 months ago

tomusher commented 9 months ago

For each of the vector index methods, similar, search and query, support specifying a similarity threshold where anything below that value will be excluded from the results/the documents included in the query.

I'm not sure if this will need normalising across backends.

tm-kn commented 9 months ago

We'd expect backends to support this.

Qdrant has score_threshold (https://qdrant.github.io/qdrant/redoc/index.html#tag/points/operation/search_points)
We can filter pgvector distance.
Weaviate has a distance filtering as well (https://weaviate.io/developers/weaviate/search/similarity#distance-threshold).

Otherwise it should have no effect. We use the limit and ordering so that should be enough to not return too many results.