search

theislab / scarches

Reference mapping for single-cell genomics
https://docs.scarches.org/en/latest/
BSD 3-Clause "New" or "Revised" License
331 stars 51 forks source link

CUDA error when running scPoli #153

Closed DzvinkaYarish closed 1 year ago

DzvinkaYarish commented 1 year ago

Hi!

I'm trying to run the example notebook for scPoli, scpoli_surgery_pipeline.ipynb, with pancreas data. The model trains without errors when running on CPU. However, when I run it on GPU, I get the following error :

RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

The full trace:

image

Reducing batch size didn't solve the issue. I'm using python3.8 with torch 1.13 and cuda 11.4

Any help would be greatly appreciated!

cdedonno commented 1 year ago

Hi, I am sorry you're having this issue. I have never encountered it during my work, it could be some issue with package versions. I have an environment with python=3.10.6, cudatoolkit=11.3.1 and torch=1.12.1 and everything seems to work fine. Can you try with these versions and report back? Then I can investigate more on what the issue might be.

DzvinkaYarish commented 1 year ago

I used the versions you specified and it solved the issue. Thanks!