search

UKPLab / sentence-transformers

Multilingual Sentence & Image Embeddings with BERT
https://www.SBERT.net
Apache License 2.0
14.72k stars 2.43k forks source link

Reproduce results on website #1531

Open luffycodes opened 2 years ago

luffycodes commented 2 years ago

hello, is there a script to run the evaluation as mentioned on the website: https://www.sbert.net/docs/pretrained_models.html#sentence-embedding-models what were the 14 datasets used to measue the performance of sentence embeddings and 6 datasets for semantic search?

nreimers commented 2 years ago

We currently work on to create a nice benchmark to compute these scores.

The 6 datasets from semantic search are from BEIR: https://github.com/beir-cellar/beir

luffycodes commented 2 years ago

Thanks a lot !

Just curious, what the other 14 datasets are? Are they STS12, etc?

nreimers commented 2 years ago

Only one of these is STSbenchmark, as STS is a horrible way to evaluate embedding models.

The others come from different domain and tasks (clustering, retrieval, duplicate detection)

afcruzs commented 2 years ago

@nreimers can you share the names of the 6 datasets from BEIR?

tholor commented 1 year ago

@nreimers would also be very interested in the names of the 6 datasets from BEIR as we want to reproduce some results and compare other models on the same benchmark :)