Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

[fightingaaa]

Hello, By setting the model as "auto", the following error occurred. How to solve it? Thanks in advance.

code : model = ValleyLlamaForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map='auto')

Error: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

infos: model.hf_device_map:

{'model.embed_tokens': 0, 'model.layers.0': 0, 'model.layers.1': 0, 'model.layers.2': 0, 'model.layers.3': 0, 'model.layers.4': 0, 'model.layers.5': 0, 'model.layers.6': 0, 'model.layers.7': 0, 'model.layers.8': 0, 'model.layers.9': 0, 'model.layers.10': 0, 'model.layers.11': 0, 'model.layers.12': 0, 'model.layers.13': 0, 'model.layers.14': 0, 'model.layers.15': 0, 'model.layers.16': 0, 'model.layers.17': 1, 'model.layers.18': 1, 'model.layers.19': 1, 'model.layers.20': 1, 'model.layers.21': 1, 'model.layers.22': 1, 'model.layers.23': 1, 'model.layers.24': 1, 'model.layers.25': 1, 'model.layers.26': 1, 'model.layers.27': 1, 'model.layers.28': 1, 'model.layers.29': 1, 'model.layers.30': 1, 'model.layers.31': 1, 'model.norm': 1, 'model.vision_tower': 1, 'model.mm_projector': 1, 'lm_head': 1}

[RupertLuo]

Try deleting this sentence "device_map='auto'"

[fightingaaa]

Try deleting this sentence "device_map='auto'"

Thanks for your answer. But when I delete this sentence, It will show cuda has insufficient memory. my device is V100-32G , while inference with valley-7B

[RupertLuo]

Could you try using my inference script? The file is valley/inference/run_valley.py

[fightingaaa]

Could you try using my inference script? The file is valley/inference/run_valley.py

yes. I used the code: python3 valley/inference/run_valley.py --model-name ../../weights/Valley2-7b/ --video-file sample_2.mp4 --query "Provide a brief description of the given video.\n<video>