search

embeddings-benchmark / mteb

MTEB: Massive Text Embedding Benchmark
https://arxiv.org/abs/2210.07316
Apache License 2.0
1.99k stars 277 forks source link

[mieb] Caltech101 not working #1381

Open Muennighoff opened 3 weeks ago

Muennighoff commented 3 weeks ago 
ERROR:mteb.evaluation.MTEB:Error while evaluating Caltech101: No module named 'datasets.tasks' 
Traceback (most recent call last):                                                             
  File "/data/niklas/mieb/mteb/scripts/run_mieb.py", line 77, in <module>                      
    results = evaluation.run(model, output_folder="/data/niklas/mieb/results-mieb-final", batch
_size=1)                                                                                       
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 464, in run                      
    raise e                                                                                    
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 394, in run                      
    task.load_data(eval_splits=task_eval_splits, **kwargs)                                     
  File "/data/niklas/mieb/mteb/mteb/abstasks/AbsTask.py", line 192, in load_data               
    self.dataset = datasets.load_dataset(**self.metadata_dict["dataset"])  # type: ignore      
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/datasets/load.py", line 2132, in l
oad_dataset                                                                                    
[....]
  File "/data/huggingface/modules/datasets_modules/datasets/HuggingFaceM4--Caltech-101/de531bad
1446ca30ea620aad43702100b105c0fc5dd44fac9f87567389186803/Caltech-101.py", line 24, in <module> 
    from datasets.tasks import ImageClassification                                             
ModuleNotFoundError: No module named 'datasets.tasks'
isaac-chung commented 3 weeks ago

This error seems to be associated with an older version of datasets. My setup has datasets==2.20.0 and I'm not able to reproduce this error. What version of datasets are you running?

Muennighoff commented 3 weeks ago

I had 3.0.2 ; you're right that it works now! Hmm means we'd have to pin a lower version in reqs https://github.com/embeddings-benchmark/mteb/blob/3a18fbdafa25696080fc4fa18c1875f64d6a4010/pyproject.toml#L28 or we fix the dataset somehow; Would probably be better to do the latter (and maybe reupload the fixed one to mteb on the hub & point to that instead) 🤔

KennethEnevoldsen commented 3 weeks ago

Related to #1363

I would probably fix the dataset since pinning datasets seems like a bad long-term solution.

A short-term solution is to add a warning specific to the dataset.

Muennighoff commented 3 weeks ago

Seems like the same issue exists with this dataset (part of MMTEB):

  File "/data/huggingface/modules/datasets_modules/datasets/SpellOnYou--kor_sarcasm/00d38c200d4d563ed94efb9ff4ca119ded94fe3cdf1e381ed95274de0a9d59f0/kor_sarcasm.py", line 21, in <module>
    from datasets.tasks import TextClassification
ModuleNotFoundError: No module named 'datasets.tasks'
KennethEnevoldsen commented 3 weeks ago

@Muennighoff I believe that was fixed in #1363