MTEB Evaluation -BOS/EOS markers

[raghavlite]

Hi, While evaluating on MTEB, are you not using user_bos, embed_bos and their eos versions?

Also, the default BASE_BOS, USER_BOS, Embed_BOS in run.py.

Are these to be used with all LLMs? I'm finetuning a mistral model which uses [INST] and [/INST] as instruction markers. How should I use these during training?

[raghavlite]

This line used during eval doesn't use the same format used during training here?

[Muennighoff]

the eval instruction is set here & should correspond to the training one https://github.com/ContextualAI/gritlm/blob/9883da1e77812e6ba2c107dc7b65d8c5ddc7396b/evaluation/eval_mteb.py#L1051

[raghavlite]

thanks