RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE exception

[longlimin]

environment: torch version： 1.13.0+cu117 cuda: 11.6

detail info:

_INFO:root:[Generating First Frame with CogView2]Raw text: 一个男人在滑雪 高清摄影 Traceback (most recent call last): File "cogvideo_pipeline.py", line 793, in main(args) File "cogvideo_pipeline.py", line 736, in main parent_given_tokens = process_stage1(model_stage1, raw_text, duration=4.0, video_raw_text=raw_text, video_guidance_text="视频", File "cogvideo_pipeline.py", line 611, in process_stage1 my_filling_sequence(model, args,seq_1st.clone(), File "cogvideo_pipeline.py", line 225, in my_filling_sequence logits, output_per_layers = model( File "/data/limin.long/CogVideo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(input, kwargs) File "/data/limin.long/CogVideo/venv/lib/python3.8/site-packages/SwissArmyTransformer/model/base_model.py", line 114, in forward return self.transformer(*args, *kwargs) File "/data/limin.long/CogVideo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(input, kwargs) File "/data/limin.long/CogVideo/venv/lib/python3.8/site-packages/SwissArmyTransformer/model/transformer.py", line 560, in forward layer_ret = layer(args, layer_id=torch.tensor(i), kw_args, output_cross_layer, File "/data/limin.long/CogVideo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(input, kwargs) File "/data/limin.long/CogVideo/venv/lib/python3.8/site-packages/SwissArmyTransformer/model/transformer.py", line 330, in forward return HOOKS_DEFAULT['layer_forward'](self, hidden_states, mask, args, kw_args) File "/data/limin.long/CogVideo/venv/lib/python3.8/site-packages/SwissArmyTransformer/transformer_defaults.py", line 134, in layer_forward_default attention_output = self.attention(attention_input, mask, kw_args) File "/data/limin.long/CogVideo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(input, kwargs) File "/data/limin.long/CogVideo/venv/lib/python3.8/site-packages/SwissArmyTransformer/model/transformer.py", line 105, in forward return self.hooks['attention_forward'](hidden_states, mask, *kw_args) File "/data/limin.long/CogVideo/models/cogvideo_cache_model.py", line 624, in attention_forward context_text, context_frame_local_text = attention_localframe_and_text_NAR( File "/data/limin.long/CogVideo/models/cogvideo_cache_model.py", line 461, in attention_localframe_and_text_NAR score_any2text = torch.matmul(q0 / math.sqrt(q0.shape[-1]), k0T[..., :text_len]) RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasGemmStridedBatchedExFix( handle, opa, opb, m, n, k, (void)(&falpha), a, CUDA_R_16F, lda, stridea, b, CUDA_R_16F, ldb, strideb, (void*)(&fbeta), c, CUDA_R_16F, ldc, stridec, num_batches, CUDA_R_32F, CUBLAS_GEMM_DEFAULT_TENSOROP)`

[wenyihong]

hi, what's your running command?