allenai / ir_datasets

Provides a common interface to many IR ranking datasets.
https://ir-datasets.com/
Apache License 2.0
306 stars 40 forks source link

beir evaluation quirks #219

Open seanmacavaney opened 1 year ago

seanmacavaney commented 1 year ago

Dataset(s)

Describe the proposed change

This twitter thread points out some evaluation quirks when using these datasets: https://twitter.com/nandan__thakur/status/1603920955679551488

It's worth adding these details to the documentation page. Maybe it's important enough that it should also be logged when you load those datasets?