RuntimeError: CUDA error: unknown error

[JoyceMind]

Traceback (most recent call last): File "/mnt/d/code/SEMamba/mytrain.py", line 575, in main() File "/mnt/d/code/SEMamba/mytrain.py", line 572, in main train(0, args, cfg) File "/mnt/d/code/SEMamba/mytrain.py", line 378, in train mag_g, pha_g, com_g = generator(noisy_mag, noisy_pha) File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, *kwargs) File "/mnt/d/code/SEMamba/mytrain.py", line 83, in forward x = block(x) File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, kwargs) File "/mnt/d/code/SEMamba/models/mamba_block.py", line 109, in forward x = self.flinear( self.freq_mamba(x).permute(0,2,1) ).permute(0,2,1) + x File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) File "/mnt/d/code/SEMamba/models/mamba_block.py", line 65, in forward x_backward, resi_backward = layer(x_backward, resi_backward) File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/mamba_ssm/modules/block.py", line 67, in forward hidden_states = self.mixer(hidden_states, inference_params=inference_params, mixer_kwargs) File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, *kwargs) File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/mamba_ssm/modules/mamba_simple.py", line 189, in forward y = selective_scan_fn( File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/mamba_ssm/ops/selective_scan_interface.py", line 88, in selective_scan_fn return SelectiveScanFn.apply(u, delta, A, B, C, D, z, delta_bias, delta_softplus, return_last_state) File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/torch/autograd/function.py", line 553, in apply return super().apply(args, *kwargs) # type: ignore[misc] File "/home/ubuntu22/miniconda3/envs/mambase/lib/python3.9/site-packages/mamba_ssm/ops/selective_scan_interface.py", line 42, in forward out, x, rest = selective_scan_cuda.fwd(u, delta, A, B, C, D, z, delta_bias, delta_softplus) RuntimeError: CUDA error: unknown error CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[lslsdx]

Hello, may I ask you have you solved that problem, I meet this problem today.