Gemma2: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:7 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

Terrencezzj commented 5 days ago

System Info

transformers version: 4.47.0.dev0
Platform: Linux-5.15.0-1052-oracle-x86_64-with-glibc2.35
Python version: 3.10.12
Huggingface_hub version: 0.25.2
Safetensors version: 0.4.5
Accelerate version: 1.1.1
Accelerate config: not found
PyTorch version (GPU?): 2.5.0+cu124 (True)
Tensorflow version (GPU?): 2.9.1 (False)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using distributed or parallel set-up in script?:
Using GPU in script?:
GPU type: NVIDIA H100 80GB HBM3

Who can help?

No response

Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)

Reproduction

model_id = 'google/gemma-2-2b'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = "Any Context"
input_ids = tokenizer.encode(messages, return_tensors="pt").to("cuda")

gen_tokens = model(input_ids)

Expected behavior

Loading checkpoint shards: 100%|██████████████████████████████████████████████████| 3/3 [00:03<00:00,  1.09s/it]
Traceback (most recent call last):
  File "/host/ckpts/transformers/script.py", line 34, in <module>
    gen_tokens = model(input_ids)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/host/ckpts/transformers/src/transformers/models/gemma2/modeling_gemma2.py", line 1052, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/host/ckpts/transformers/src/transformers/models/gemma2/modeling_gemma2.py", line 785, in forward
    inputs_embeds = self.embed_tokens(input_ids)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/sparse.py", line 190, in forward
    return F.embedding(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 2551, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:7 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

If set one gpu visible, no error

LysandreJik commented 3 days ago

Can you take a look at this @SunMarc or @MekkCyber ?

Terrencezzj commented 2 days ago

v4.43.4 doesn't have the issue

hchings commented 2 days ago

Hi @LysandreJik , this potential feature regression in Transformers have caused issues in our library (NVIDIA TensorRT Model Optimizer) when our users want to run quantization on multi-GPUs. Currently, our users need to revert back to Transformers v4.43.4. We appreciate if you can help prioritize this. Thanks!

MekkCyber commented 2 days ago

Hi @hchings @Terrencezzj, thanks for specifying that v4.43.4 does work, I am looking into the problem

edkamesh commented 10 hours ago

Hi, I found that there was no accelerate config. You can create an accelerate config file with multigpu config and try running the below hoping you loaded the model and tokenizer before.

Refer here to create config.yaml file to support multiGPU Inference https://huggingface.co/docs/accelerate/v1.1.0/en/package_reference/accelerator#accelerate.Accelerator https://huggingface.co/docs/accelerate/v1.1.0/en/package_reference/utilities#accelerate.DistributedType


`accelerator= Accelerator()
model = accelerator.prepare(model)
input_ids = tokenizer.encode("Any Context", return_tensors="pt").to(accelerator.device)
gen_tokens = model.generate(input_ids)`

Terrencezzj commented 1 minute ago

Hi, I found that there was no accelerate config. You can create an accelerate config file with multigpu config and try running the below hoping you loaded the model and tokenizer before.

Refer here to create config.yaml file to support multiGPU Inference https://huggingface.co/docs/accelerate/v1.1.0/en/package_reference/accelerator#accelerate.Accelerator https://huggingface.co/docs/accelerate/v1.1.0/en/package_reference/utilities#accelerate.DistributedType
`accelerator= Accelerator()
model = accelerator.prepare(model)
input_ids = tokenizer.encode("Any Context", return_tensors="pt").to(accelerator.device)
gen_tokens = model.generate(input_ids)` 

Hi, I got the same error with Accelerator. Please noticed that my script was not inference, it's model(input_ids)

huggingface / transformers