Open Doufanfan opened 11 months ago
有朋友遇到过这种报错吗?
/opt/conda/conda-bld/pytorch_1682343995622/work/aten/src/ATen/native/cuda/Indexing.cu File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/peft/peft_model.py", line 678, in forward :1146: indexSelectLargeIndex: block: [85,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed. /opt/conda/conda-bld/pytorch_1682343995622/work/aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [85,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed. return self.base_model( ^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 809, in forward outputs = self.model( ^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 690, in forward layer_outputs = torch.utils.checkpoint.checkpoint( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/torch/utils/checkpoint.py", line 249, in checkpoint return CheckpointFunction.apply(function, preserve, *args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/torch/autograd/function.py", line 506, in apply return super().apply(*args, **kwargs) # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/torch/utils/checkpoint.py", line 107, in forward outputs = run_function(*args) ^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 686, in custom_forward return module(*inputs, past_key_value, output_attentions) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 413, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 310, in forward query_states = self.q_proj(hidden_states) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/pt2/lib/python3.11/site-packages/peft/tuners/lora.py", line 565, in forward result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)` 0%| | 0/5000 [00:01<?, ?it/s]
有朋友遇到过这种报错吗?