search

Megvii-BaseDetection / BEVDepth

Official code for BEVDepth.
MIT License
722 stars 100 forks source link

void magma_queue_create_internal(magma_device_t, magma_queue**, const char*, const char*, int): Assertion `queue->dCarray__ != __null’ failed. #57

Open JiachengDeng opened 2 years ago

JiachengDeng commented 2 years ago

python: /opt/conda/conda-bld/magma-cuda111_1605822518874/work/interface_cuda/interface.cpp:901: void magma_queue_create_internal(magma_device_t, cudaStream_t, cublasHandle_t, cusparseHandle_t, magma_queue*, const char, const char*, int): Assertion `queue->dCarray != null’ failed.

微信图片_20220907103602

Has anyone encountered this problem? The problem occurs more randomly at different epochs. When it occurs, the training stops, but the memory is not automatically released.

My Environment: CUDA 11.2 cudatoolkit 11.1 torch 1.9.1+cu111 pytorch-lightning 1.6.0 python 3.7.13 mmdet3d 1.0.0rc4 mmcv 1.6.0 mmcv-full 1.6.1 mmsegmentation 0.27.0

friendship1 commented 1 year ago

I too had the same problem. From what I've found, this is a memory issue. (https://github.com/kcosta42/VQGAN-CLIP-Docker/issues/2) Try reducing the input batch size or reducing the model network depth.