embeddings-benchmark / mteb

MTEB: Massive Text Embedding Benchmark
https://arxiv.org/abs/2210.07316
Apache License 2.0
1.95k stars 271 forks source link

[mieb] google/siglip-large-patch16-384 fails on ImageNet10Clustering #1417

Open Muennighoff opened 6 days ago

Muennighoff commented 6 days ago
ERROR:mteb.evaluation.MTEB:Error while evaluating ImageNet10Clustering: mean must have 1 elements if it is an iterable, got 3
Traceback (most recent call last):
  File "/data/niklas/mieb/mteb/scripts/run_mieb.py", line 91, in <module>
    results = evaluation.run(model, output_folder="/data/niklas/mieb/results-mieb-final", batch_size=32)
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 464, in run
    raise e
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 425, in run
    results, tick, tock = self._run_eval(
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 300, in _run_eval
    results = task.evaluate(
  File "/data/niklas/mieb/mteb/mteb/abstasks/AbsTask.py", line 126, in evaluate
    scores[hf_subset] = self._evaluate_subset(
  File "/data/niklas/mieb/mteb/mteb/abstasks/Image/AbsTaskImageClustering.py", line 54, in _evaluate_subset
    metrics = evaluator(model, encode_kwargs=encode_kwargs)
  File "/data/niklas/mieb/mteb/mteb/evaluation/evaluators/Image/ClusteringEvaluator.py", line 41, in __call__
    image_embeddings = model.get_image_embeddings(
  File "/data/niklas/mieb/mteb/mteb/models/siglip_models.py", line 74, in get_image_embeddings
    inputs = self.processor(
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/models/siglip/processing_siglip.py", line 113, in __call__
    image_features = self.image_processor(images, return_tensors=return_tensors)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/image_processing_utils.py", line 41, in __call__
    return self.preprocess(images, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/utils/generic.py", line 852, in wrapper
    return func(*args, **valid_kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/models/siglip/image_processing_siglip.py", line 231, in preprocess
    images = [
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/models/siglip/image_processing_siglip.py", line 232, in <listcomp>
    self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/image_processing_utils.py", line 111, in normalize
    return normalize(
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/image_transforms.py", line 398, in normalize
    raise ValueError(f"mean must have {num_channels} elements if it is an iterable, got {len(mean)}")
ValueError: mean must have 1 elements if it is an iterable, got 3
gowitheflow-1998 commented 4 days ago

fixed in #1424