Anyone knows why I am getting CUDA error: CUBLAS_STATUS_INVALID_VALUE ?

[gedance]

Hello,

I am getting below error when trying to run training:

python3 tools/train.py ./config/nanodet-plus-m_320.yml

Error:

File "/mnt/d/ml/nanodet/nanodet/model/head/gfl_head.py", line 66, in forward x = F.linear(x, self.project.type_as(x)).reshape(*shape[:-1], 4) RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemv(handle, op, m, n, &alpha, a, lda, x, incx, &beta, y, incy)

I am using Ubuntu 20.04 LTS Python 3.8.10 Pytorch '1.13.1+cu117' torchvision '0.14.1+cu117' NVIDIA-SMI 470.223.02 Driver Version: 470.223.02 CUDA Version: 11.4

I am using the default config file, and all defaults,, Does anyone know why I am getting this error and how to resolve it?

Thanks you!

[gedance]

Oh I figured it out :) The problem was the cuda runtime version non-matching,, Here's my solution using Anaconda, in case of anyone needs help with CUDA error:

export PATH= ~ / anaconda3/bin:~/anaconda3/condabin:$PATH conda create -n testnanodet python=3.8 -y conda activate testnanodet conda install pytorch torchvision cudatoolkit=11.1 -c pytorch -c conda-forge git clone https://github.com/RangiLyu/nanodet.git cd nanodet pip install -r requirements.txt python setup.py develop python demo/demo.py image --config config/nanodet-plus-m_320.yml --model nanodet-plus-m_320.pth --path 1.jpg

It works like a charm! Cheers!