embeddings-benchmark / mteb

MTEB: Massive Text Embedding Benchmark
https://arxiv.org/abs/2210.07316
Apache License 2.0
1.97k stars 274 forks source link

[mieb] nyu-visionx/moco-v3-vit-b on CIFAR10Clustering: Unexpected type <class 'list'> #1479

Closed Muennighoff closed 10 hours ago

Muennighoff commented 1 day ago
ERROR:mteb.evaluation.MTEB:Error while evaluating CIFAR10Clustering: Unexpected type <class 'list'>      
Traceback (most recent call last):                                                                       
  File "/data/niklas/mieb/mteb/scripts/run_mieb_missed_results.py", line 22, in <module>                 
    results = evaluation.run(model, output_folder="/data/niklas/mieb/results-mieb-final")                
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 464, in run                                
    raise e                                                                                              
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 425, in run                                
    results, tick, tock = self._run_eval(                                                                
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 300, in _run_eval
    results = task.evaluate(
  File "/data/niklas/mieb/mteb/mteb/abstasks/AbsTask.py", line 126, in evaluate
    scores[hf_subset] = self._evaluate_subset(
  File "/data/niklas/mieb/mteb/mteb/abstasks/Image/AbsTaskImageClustering.py", line 54, in _evaluate_subs
et
    metrics = evaluator(model, encode_kwargs=encode_kwargs)
  File "/data/niklas/mieb/mteb/mteb/evaluation/evaluators/Image/ClusteringEvaluator.py", line 41, in __ca
ll__
    image_embeddings = model.get_image_embeddings(
  File "/data/niklas/mieb/mteb/mteb/models/moco_models.py", line 82, in get_image_embeddings
    self.processor(batch_images)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torchvision/transforms/transforms.py", line 
95, in __call__
    img = t(img)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _
wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _
call_impl
    return forward_call(*args, **kwargs)
    image_embeddings = model.get_image_embeddings(
  File "/data/niklas/mieb/mteb/mteb/models/moco_models.py", line 82, in get_image_embeddings
    self.processor(batch_images)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torchvision/transforms/transforms.py", line 
95, in __call__
    img = t(img)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _
wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _
call_impl
    return forward_call(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torchvision/transforms/transforms.py", line 
354, in forward
    return F.resize(img, self.size, self.interpolation, self.max_size, self.antialias)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torchvision/transforms/functional.py", line 
465, in resize
    _, image_height, image_width = get_dimensions(img)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torchvision/transforms/functional.py", line 
80, in get_dimensions
    return F_pil.get_dimensions(img)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torchvision/transforms/_functional_pil.py", 
line 31, in get_dimensions 
    raise TypeError(f"Unexpected type {type(img)}")
TypeError: Unexpected type <class 'list'>
isaac-chung commented 11 hours ago

I'm able to reproduce the error. Investigating.