RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)

chaitanya100100 / TailorNet

Code for our CVPR 2020 (ORAL) paper - TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style.

https://virtualhumans.mpi-inf.mpg.de/tailornet/

Other

408 stars 67 forks source link

RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle) #27

Closed aapatre closed 3 years ago

aapatre commented 3 years ago

Screenshot from 2021-03-18 11-52-25

So this happens while using weights & data for male shirt. Female/Male tshirt, Male Short Pants work absolutely fine.

chaitanya100100 commented 3 years ago

It seems like CUDA issue rather than the weights. Have you tried these solutions: link 1 and link 2 ?. Shirt is the garment with the highest number of vertices. So there may be GPU memory issues. What is your GPU memory size?

aapatre commented 3 years ago

Yeah, It seems to be a GPU memory issue since we have a 4GB 1650Ti. I tried removing the cuda() calls from L98, L99, L100, in run_tailornet.py and it displays a "CUDA out of memory" error now.

chaitanya100100 commented 3 years ago

So removing those .cuda() calls won't help because TailorNet model is hardcoded to run on GPU. See this line. You can remove such calls to run on CPU. Or you can make GPU unavailable for this particular command by setting CUDA_VISIBLE_DEVICES empty.

aapatre commented 3 years ago

Thanks a lot for your response @chaitanya100100 I will check... Apologies for the late reply.