jobergum
commented
3 years ago Hello @SaiKiranBurle and thank you for your interest in Vespa and its approximate nearest neighbor search operator.
It's correct that dotproduct is not listed as it's not a proper distance metric.
The common approach is to transform dotproduct to euclidean space, see Speeding Up the Xbox Recommender System Using a Euclidean Transformation for Inner-Product Spaces
We use the mentioned transformation in this dense retrieval sample app where the original DPR representation uses the dot product and the representation on Vespa has transformed the data to euclidean space where we can use ANN.
See also https://blog.vespa.ai/efficient-open-domain-question-answering-on-vespa/
SaiKiranBurle
Hello, I intend to use Vespa along with this Approximate nearest neighbor capabilities for large scale recommender systems.
After some initial search, the supported distance metrics (listed at https://docs.vespa.ai/documentation/reference/schema-reference.html#distance-metric) does not include dot product function. Because of various intricacies in our model, we cannot use cosine similarity. We need to use dot product where the vectors are not normalized. Is there any way I can leverage the HNSW index of Vespa for my use-case?