Error running eval_usmle.py

[hyesunyun]

Hello!

Thank you so much for the repo.

I am trying to run the eval_usmle.py after creating a conda environment and installing dependencies using requirements.txt.

When I get to the part of generating outputs in the Inferer, I get an error.

Here is the output:

Traceback (most recent call last):
  File "/scratch/yun.hy/clinical-llm-alignment/medAlpaca/eval/eval_usmle.py", line 177, in <module>
    fire.Fire(main)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/scratch/yun.hy/clinical-llm-alignment/medAlpaca/eval/eval_usmle.py", line 158, in main
    response = model(
  File "/scratch/yun.hy/clinical-llm-alignment/medAlpaca/eval/../medalpaca/inferer.py", line 161, in __call__
    generation_output = self.model.generate(
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/peft/peft_model.py", line 550, in generate
    return self.base_model.generate(**kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/transformers/generation/utils.py", line 1572, in generate
    return self.sample(
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/transformers/generation/utils.py", line 2619, in sample
    outputs = self(
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 689, in forward
    outputs = self.model(
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 579, in forward
    layer_outputs = decoder_layer(
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 293, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 195, in forward
    query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/peft/tuners/lora.py", line 421, in forward
    result += self.lora_B(self.lora_A(self.lora_dropout(x))) * self.scaling
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/yun.hy/.conda/envs/medalpaca/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: expected scalar type Half but found Float

Have you encountered this error before? I found some things somewhere that it could be related to bitsandbytes and running on v100. I tried another GPU but still getting this issue.

Thanks!

[kbressem]

This might be due to bnb, as I did not pin the version. The version I used is 0.37.2. Can you try with this version?

[hyesunyun]

Hi! Thank you for the response.

I changed the bnb version to 0.37.2 but still facing the same error, unfortunately.

My goal at the moment is to run the USMLE questions using the MedAlpaca model. Would you recommend just writing my own code and implement what you have done using huggingface for this since I am not able to get past this error using the Inferer.py?