bash scripts/forward.sh

[Veluriyam]

bash scripts/forward.sh in Llama-2-7b-chat-hf Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:06<00:00, 3.10s/it] [2024-09-23 17:22:48,125] [forward.py:111] Model name: Llama-2-7b-chat-hf [2024-09-23 17:22:48,132] [forward.py:112] Model size: 13.543948288 [2024-09-23 17:22:48,133] [utils.py:94] GPU 0: 6.88 GB / 32.00 GB [2024-09-23 17:22:48,133] [utils.py:94] GPU 1: 6.88 GB / 32.00 GB [2024-09-23 17:22:48,273] [forward.py:173] Running 0%| | 0/100 [00:00<?, ?it/s]../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [1,0,0], thread: [96,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [1,0,0], thread: [97,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ... ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [31,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. 0%| | 0/100 [00:01<?, ?it/s] Traceback (most recent call last): File "forward.py", line 212, in main() File "forward.py", line 177, in main hidden_states = forward(model, toker, messages) File "forward.py", line 52, in forward outputs = model( File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, *kwargs) File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/accelerate/hooks.py", line 170, in new_forward output = module._old_forward(args, kwargs) File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 1183, in forward outputs = self.model( File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, *kwargs) File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 1070, in forward layer_outputs = decoder_layer( File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, kwargs) File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/accelerate/hooks.py", line 170, in new_forward output = module._old_forward(*args, *kwargs) File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 798, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, *kwargs) File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/accelerate/hooks.py", line 170, in new_forward output = module._old_forward(args, **kwargs) File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 706, in forward query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids) File "/opt/conda/envs/onprompt/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 232, in apply_rotary_pos_emb cos = cos[position_ids].unsqueeze(unsqueeze_dim) RuntimeError: CUDA error: device-side assert triggered Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

What is the cause of this error?Thanks.

[chujiezheng]

Pls try to downgrade the transformers's version to ~4.37.

[Veluriyam]

Based on your previous issues reconfigured the environment python==3.10, torch==2.1.1, transformers==4.37, used cuda==11.6, and got a similar error as yesterday, but in a different place： Traceback (most recent call last): File "/root/yp/LLM-Safeguard-main/code/forward.py", line 212, in main() File "/root/yp/LLM-Safeguard-main/code/forward.py", line 177, in main hidden_states = forward(model, toker, messages) File "/root/yp/LLM-Safeguard-main/code/forward.py", line 77, in forward hidden_states = [e[0].detach().half().cpu() for e in outputs.hidden_states[1:]] File "/root/yp/LLM-Safeguard-main/code/forward.py", line 77, in hidden_states = [e[0].detach().half().cpu() for e in outputs.hidden_states[1:]] RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[Veluriyam]

事实上我检查每个hidden_state 的形状：outputs = model( input_ids, attention_mask=input_ids.new_ones(input_ids.size(), dtype=model.dtype), return_dict=True, output_hidden_states=True, )

检查每个 hidden_state 的形状

for i, hidden_state in enumerate(outputs.hidden_states):
    print(f"Layer {i} shape: {hidden_state.shape}")
    print(f"Layer{i}:{hidden_state}")发现在执行到此时，打印输出第一个Layer0~Layer32 shape：torch.Size([1, 22, 4096])，但是从Layer 19开始-Layer32，tensor变为0，如图，然后打印下一个Layer0时shape变成了[1,21,4096],但tensor均不为0，随后就出现了错误，是shape变化的原因导致的错误吗？

[Veluriyam]

Thanks for you,change the device_map of function "model" from auto into sequential had solve the problem.