ENH: information retrieval for embedding models - Githubissues

Striveworks / valor

Valor is a centralized evaluation store which makes it easy to measure, explore, and rank model performance.

https://striveworks.github.io/valor/

Other

38 stars 4 forks source link

ENH: information retrieval for embedding models #480

Open ekorman opened 8 months ago

ekorman commented 8 months ago

Feature Type

[ ] Adding new functionality to valor
[ ] Changing existing functionality in valor
[ ] Removing existing functionality in valor

Problem Description

We should have support for information retrieval metrics for evaluating embedding models. Standard metrics:

recall at k
precision at k
f1 at k
mAP at k
mRR at k
NDCG at k

see https://www.pinecone.io/learn/offline-evaluation/ and retrieval section of torchmetrics https://lightning.ai/docs/torchmetrics/stable/

Feature Description

New evaluate_embedding mehtod

Additional Context

No response

ntlind commented 8 months ago

3/11: no pressing need here; fine to backlog