[x] Run tests locally to make sure nothing is broken using make test.
[x] Run the formatter to format the code using make lint.
Adding datasets checklist
Reason for dataset addition: ...
[ ] I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.
[ ] I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
[ ] If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
[ ] I have filled out the metadata object in the dataset file (find documentation on it here).
[ ] Run tests locally to make sure nothing is broken using make test.
[ ] Run the formatter to format the code using make lint.
Adding a model checklist
[ ] I have filled out the ModelMeta object to the extent possible
[ ] I have ensured that my model can be loaded using
[ ] mteb.get_model(model_name, revision) and
[ ] mteb.get_model_meta(model_name, revision)
[ ] I have tested the implementation works on a representative set of tasks.
see https://github.com/embeddings-benchmark/results/pull/43
Checklist
make test.make lint.Adding datasets checklist
Reason for dataset addition: ...
mteb -m {model_name} -t {task_name}command.sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2intfloat/multilingual-e5-smallself.stratified_subsampling() under dataset_transform()make test.make lint.Adding a model checklist
mteb.get_model(model_name, revision)andmteb.get_model_meta(model_name, revision)