search

OpenGVLab / Vision-RWKV

Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures
https://arxiv.org/abs/2403.02308
Apache License 2.0
371 stars 14 forks source link

RuntimeError: CUDA error: invalid argument #7

Closed yxchng closed 7 months ago

yxchng commented 7 months ago 
File "/home/user/segmentation/mmseg_custom/models/backbones/base/vrwkv.py", line 64, in backward
    gw = torch.sum(gw, dim=0)
RuntimeError: CUDA error: invalid argument
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I am meeting this problem when trying to use your model on different datasets. Any idea why this might occur?

duanduanduanyuchen commented 7 months ago 
File "/home/user/segmentation/mmseg_custom/models/backbones/base/vrwkv.py", line 64, in backward
    gw = torch.sum(gw, dim=0)
RuntimeError: CUDA error: invalid argument
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I am meeting this problem when trying to use your model on different datasets. Any idea why this might occur?

I haven't seen this error before, but some Cuda errors may occur when out of memory. Maybe you can try to reduce the batch size.