search

embeddings-benchmark / mteb

MTEB: Massive Text Embedding Benchmark
https://arxiv.org/abs/2210.07316
Apache License 2.0
1.97k stars 274 forks source link

[mieb] TIGER-Lab/VLM2Vec-LoRA fails with dtype issue #1456

Closed Muennighoff closed 1 week ago

Muennighoff commented 1 week ago 
ERROR:mteb.evaluation.MTEB:Error while evaluating BLINKIT2IMultiChoice: self and mat2 must have the same dtype,
 but got BFloat16 and Float                                                                                    
Traceback (most recent call last):                                                                             
  File "/data/niklas/mieb/mteb/scripts/run_mieb.py", line 91, in <module>                                      
    results = evaluation.run(model, output_folder="/data/niklas/mieb/results-mieb-final", batch_size=32)       
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 464, in run                                      
    raise e                                                                                                    
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 425, in run                                      
    results, tick, tock = self._run_eval(                                                                      
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 300, in _run_eval                                
    results = task.evaluate(                                                                                   
  File "/data/niklas/mieb/mteb/mteb/abstasks/Image/AbsTaskAny2AnyMultiChoice.py", line 273, in evaluate        
    scores[hf_subset] = self._evaluate_subset(                                                                 
  File "/data/niklas/mieb/mteb/mteb/abstasks/Image/AbsTaskAny2AnyMultiChoice.py", line 282, in _evaluate_subset
    results = retriever(corpus, queries, relevant_docs)                                                        
  File "/data/niklas/mieb/mteb/mteb/evaluation/evaluators/Image/Any2AnyMultiChoiceEvaluator.py", line 298, in _
_call__                                                                                                        
    return self.retriever.search(                                                                              
  File "/data/niklas/mieb/mteb/mteb/evaluation/evaluators/Image/Any2AnyMultiChoiceEvaluator.py", line 145, in s
earch                                                                                                          
    query_embeddings = self.model.get_fused_embeddings(                                                        
  File "/data/niklas/mieb/mteb/mteb/models/vlm2vec_models.py", line 274, in get_fused_embeddings               
    outputs = self.encode_input(inputs)                                                                        
  File "/data/niklas/mieb/mteb/mteb/models/vlm2vec_models.py", line 91, in encode_input                        
    hidden_states = self.mdl(**input, return_dict=True, output_hidden_states=True)                             
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrappe
d_call_impl                                                                                                    
    return self._call_impl(*args, **kwargs)                                                                    
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_i
mpl                                                                                                            
    return forward_call(*args, **kwargs)                                                                       
  File "/data/huggingface/modules/transformers_modules/microsoft/Phi-3.5-vision-instruct/4a0d683eba9f1d0cbfb615
1705d1ee73c25a80ca/modeling_phi3_v.py", line 1603, in forward                                                  
    outputs = self.model(                                                                                      
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrappe
d_call_impl                                                                                                    
    return self._call_impl(*args, **kwargs)                                                                    
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_i
mpl                                                                                                            
    return forward_call(*args, **kwargs)                                                                       
  File "/data/huggingface/modules/transformers_modules/microsoft/Phi-3.5-vision-instruct/4a0d683eba9f1d0cbfb615
1705d1ee73c25a80ca/modeling_phi3_v.py", line 1431, in forward                                                  
    inputs_embeds = self.vision_embed_tokens(input_ids, pixel_values=pixel_values, image_sizes=image_sizes)    
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, i[23/1864]
d_call_impl
    return self._call_impl(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_i
mpl
    return forward_call(*args, **kwargs)
  File "/data/huggingface/modules/transformers_modules/microsoft/Phi-3.5-vision-instruct/4a0d683eba9f1d0cbfb615
1705d1ee73c25a80ca/modeling_phi3_v.py", line 237, in forward
    img_features = self.get_img_features(pixel_values.flatten(0, 1)).reshape(
  File "/data/huggingface/modules/transformers_modules/microsoft/Phi-3.5-vision-instruct/4a0d683eba9f1d0cbfb615
1705d1ee73c25a80ca/modeling_phi3_v.py", line 212, in get_img_features
    img_processor_output = self.img_processor(img_embeds, output_hidden_states=True)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrappe
d_call_impl
    return self._call_impl(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_i
mpl
    return forward_call(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 1
171, in forward
    return self.vision_model(
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrappe
d_call_impl
    return self._call_impl(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_i
mpl
    return forward_call(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 1
097, in forward
    encoder_outputs = self.encoder(
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrappe
d_call_impl
    return self._call_impl(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_i
mpl
    return forward_call(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 8
77, in forward
    layer_outputs = encoder_layer(
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrappe
d_call_impl
    return self._call_impl(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 6
08, in forward
    hidden_states, attn_weights = self.self_attn(
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrappe
d_call_impl
    return self._call_impl(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_i
mpl
    return forward_call(*args, **kwargs)
  File "/data/huggingface/modules/transformers_modules/microsoft/Phi-3.5-vision-instruct/4a0d683eba9f1d0cbfb615
1705d1ee73c25a80ca/modeling_phi3_v.py", line 101, in forward
    query_states = self.q_proj(hidden_states).reshape(bsz, tgt_len, self.num_heads, self.head_dim)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrappe
d_call_impl
    return self._call_impl(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_i
mpl
    return forward_call(*args, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 117, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: self and mat2 must have the same dtype, but got BFloat16 and Float
isaac-chung commented 1 week ago

Closing to consolidate the issue in #1377