search

MzeroMiko / VMamba

VMamba: Visual State Space Models，code is based on mamba
MIT License
1.82k stars 98 forks source link

Triton Error [CUDA]: context is destroyed #247

Closed DingjieFu closed 5 days ago

DingjieFu commented 6 days ago

Hello, Thankx for you great work! I met the error below while using VMamba as a backbone, how can I fix it?

Traceback (most recent call last): File "/home/fudingjie/VMamba/ZeroVMamba.py", line 241, in x_g = model.vim(data) File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/fudingjie/VMamba/models/vmamba.py", line 1489, in forward x = layer(x) File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward input = module(input) File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward input = module(input) File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/fudingjie/VMamba/models/vmamba.py", line 1233, in forward return self._forward(input) File "/home/fudingjie/VMamba/models/vmamba.py", line 1221, in _forward x = x + self.drop_path(self.op(self.norm(x))) File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/fudingjie/VMamba/models/vmamba.py", line 654, in forwardv2 y = self.forward_core(x) File "/home/fudingjie/VMamba/models/vmamba.py", line 600, in forward_corev2 xs = cross_scan_fn(x, in_channel_first=True, out_channel_first=True, scans=_scan_mode, force_torch=scan_force_torch) File "/home/fudingjie/VMamba/models/csm_triton.py", line 496, in cross_scan_fn return CSF.apply(x, in_channel_first, out_channel_first, one_by_one, scans) File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/torch/autograd/function.py", line 539, in apply return super().apply(args, *kwargs) # type: ignore[misc] File "/home/fudingjie/VMamba/models/csm_triton.py", line 418, in forward triton_cross_scan_flex[(NH NW, NC, B)]( File "", line 65, in triton_cross_scan_flex File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/triton/compiler/compiler.py", line 579, in getattribute self._init_handles() File "/home/fudingjie/miniconda3/envs/vmamba/lib/python3.10/site-packages/triton/compiler/compiler.py", line 570, in _init_handles mod, func, n_regs, n_spills = fn_load_binary(self.metadata["name"], self.asm[bin_path], self.shared, device) RuntimeError: Triton Error [CUDA]: context is destroyed