search

xlang-ai / instructor-embedding

[ACL 2023] One Embedder, Any Task: Instruction-Finetuned Text Embeddings
Apache License 2.0
1.78k stars 131 forks source link

Are document instructions not used for evaluation? #111

Closed orionw closed 2 months ago

orionw commented 3 months ago

Hi there! Thanks for the great work!

I was trying to reproduce the MTEB evaluation and noticed that it seems like the instructions for documents are not used. Is that correct?

See: https://github.com/xlang-ai/instructor-embedding/blob/e749023b14d36cdff234398de57d9b6c519e776a/evaluation/MTEB/mteb/abstasks/AbsTaskRetrieval.py#L738 referencing sentences instead of new_sentences that contain the instructions.

Totally fine, if that is true (even if unintentional - just trying to reproduce it).

hongjin-su commented 2 months ago

Thanks for pointing out! See here for the latest version!

orionw commented 2 months ago

Ah nice, I see it was updated! Sorry, I had started using this repo before then and didn’t realize there had been a change.

Appreciate your time!