RuntimeError

[Aaron198513]

这是什么情况呀 Traceback (most recent call last): File "/home/pc/c_zk/UniFormerV2-main/tools/run_net_multi_node.py", line 69, in main() File "/home/pc/c_zk/UniFormerV2-main/tools/run_net_multi_node.py", line 50, in main launch_job(cfg=cfg, init_method=args.init_method, func=train) File "/home/pc/c_zk/UniFormerV2-main/slowfast/utils/misc.py", line 311, in launch_job func(cfg=cfg) File "/home/pc/c_zk/UniFormerV2-main/tools/train_net.py", line 420, in train misc.log_model_info(model, cfg, use_train_input=True) File "/home/pc/c_zk/UniFormerV2-main/slowfast/utils/misc.py", line 188, in log_model_info get_model_stats(model, cfg, "flop", use_train_input) File "/home/pc/c_zk/UniFormerV2-main/slowfast/utils/misc.py", line 166, in get_model_stats countdict, * = model_stats_fun(model, inputs) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/fvcore/nn/flop_count.py", line 149, in flop_count for op, flop in flop_counter.by_operator().items(): File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/fvcore/nn/jit_analysis.py", line 267, in by_operator stats = self._analyze() File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/fvcore/nn/jit_analysis.py", line 556, in _analyze graph = _get_scoped_trace_graph(self._model, self._inputs, self._aliases) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/fvcore/nn/jit_analysis.py", line 178, in _get_scoped_tracegraph graph, = _get_trace_graph(module, inputs) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/jit/_trace.py", line 1166, in _get_trace_graph outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, kwargs) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, kwargs) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/jit/_trace.py", line 127, in forward graph, out = torch._C._create_graph_by_tracing( File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/jit/_trace.py", line 118, in wrapper outs.append(self.inner(trace_inputs)) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1128, in _call_impl result = forward_call(input, kwargs) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1098, in _slow_forward result = self.forward(*input, kwargs) File "/home/pc/c_zk/UniFormerV2-main/slowfast/models/uniformerv2.py", line 108, in forward output = self.backbone(x) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1128, in _call_impl result = forward_call(*input, kwargs) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1098, in _slow_forward result = self.forward(*input, *kwargs) File "/home/pc/c_zk/UniFormerV2-main/slowfast/models/uniformerv2_model.py", line 329, in forward out = self.transformer(x) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1128, in _call_impl result = forward_call(input, kwargs) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1098, in _slow_forward result = self.forward(*input, kwargs) File "/home/pc/c_zk/UniFormerV2-main/slowfast/models/uniformerv2_model.py", line 255, in forward x = resblock(x, T_down) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1128, in _call_impl result = forward_call(*input, *kwargs) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1098, in _slow_forward result = self.forward(input, kwargs) File "/home/pc/c_zk/UniFormerV2-main/slowfast/models/uniformerv2_model.py", line 114, in forward x = x + self.drop_path(self.attention(self.ln_1(x))) File "/home/pc/c_zk/UniFormerV2-main/slowfast/models/uniformerv2_model.py", line 95, in attention return self.attn(x, x, x, need_weights=False, attn_mask=self.attn_mask)[0] File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1128, in _call_impl result = forward_call(*input, *kwargs) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1098, in _slow_forward result = self.forward(input, **kwargs) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/modules/activation.py", line 1038, in forward attn_output, attn_output_weights = F.multi_head_attention_forward( File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/functional.py", line 5245, in multi_head_attention_forward q, k, v = _in_projection_packed(query, key, value, in_proj_weight, in_proj_bias) File "/home/pc/anaconda3/envs/uniformer/lib/python3.9/site-packages/torch/nn/functional.py", line 4928, in _in_projection_packed return linear(q, w, b).chunk(3, dim=-1) RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

[Andy1621]

Hi! Can you provide the version of PyTorch and CUDA?

[Aaron198513]

加入这个命令就好了 export LD_LIBRARY_PATH=/home/$USER/.conda/envs/$ENVNAME/lib:/usr/local/cuda-11/lib64

[Aaron198513]

Hi! Can you provide the version of PyTorch and CUDA?

Cuda 11.3 torch 1.11.0