Closed py4 closed 9 months ago
cc @younesbelkada I think this worked / works on main no?
FYI @gante and @tomaarsen we'll work on a fix with @younesbelkada
I'm hitting a similar error with Llava 1.6 + Mistral 7B. What was the fix previously and what changes (if any) need to be made for Mistral 7B?
return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument weight in method wrapper_CUDA__native_layer_norm)
The problem arises due to:
<bound method AlignDevicesHook.pre_forward of AlignDevicesHook(execution_device=1, offload=False, io_same_device=False, offload_buffers=False, place_submodules=True, skip_keys=['past_key_values', 'causal_mask'])>
which moves the the input tensors from the "correct" devices (in my setting, 0) to the wrong device (in my setting, 1).
Please @RylanSchaeffer do not re-open the same discussion on various threads: https://github.com/huggingface/transformers/pull/28051#issuecomment-1976118206 let's keep the discussion on a single issue
System Info
transformers
version: 4.36.0.dev0Who can help?
@pacman100 @ArthurZucker
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Expected behavior
It should produce the output but I get the following. I believe something similar to this is needed to fix