mikemccand
commented
2 years ago +1, this would be awesome.
We would need some sort of (new) gold file I think, recording the "brute force correct KNN results" for each query vector?
Open jtibshirani opened 2 years ago
mikemccand
commented
2 years ago +1, this would be awesome.
We would need some sort of (new) gold file I think, recording the "brute force correct KNN results" for each query vector?
Would it be possible to report vector search recall in addition to QPS? When assessing vector search, users are interested in both computational performance and accuracy.
Tracking recall through nightly benchmarks would also help guard against subtle, unintentional changes to the algorithm. In my experience developing statistical algorithms, it's easy to make an accidental change while refactoring! With this type of algorithm, unit tests don't guard perfectly against regressions.