search

IsoNet-cryoET / IsoNet

Self-supervised learning for isotropic cryoET reconstruction
https://www.nature.com/articles/s41467-022-33957-8
MIT License
70 stars 13 forks source link

CUDA_ERROR_NOT_INITIALIZED: initialization error #11

Closed shahpnmlab closed 3 years ago

shahpnmlab commented 3 years ago

Hi, I am trying to run IsoNet on a dataset. It is installed as described on the github page. I am using tensorflow version 2.5.0 with an Nividia 2080RTX 8GB GPU running CUDA 11.4. I made the deconvolved tomos, after which I made the masks and extracted 50 subtomos from every tomogram. After that, i tried to train the NN and ran into the following message

07-28 13:27:54, INFO

Isonet starts refining

07-28 13:27:55, WARNING The results folder already exists before the 1st iteration The old results folder will be renamed (to results~) 07-28 13:27:58, INFO Done preperation for the first iteration! 07-28 13:27:58, INFO Start Iteration1! /home/user/software/anaconda3/envs/isonet/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:374: UserWarning: The lr argument is deprecated, use learning_rate instead. warnings.warn( /home/user/software/anaconda3/envs/isonet/lib/python3.8/site-packages/tensorflow/python/keras/utils/generic_utils.py:494: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. warnings.warn('Custom mask layers require a config and must override ' 07-28 13:27:59, INFO Noise Level:0.0 2021-07-28 13:28:12.359664: F tensorflow/stream_executor/cuda/cuda_driver.cc:210] Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error 2021-07-28 13:28:12.552163: F tensorflow/stream_executor/cuda/cuda_driver.cc:210] Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error 2021-07-28 13:28:13.096353: F tensorflow/stream_executor/cuda/cuda_driver.cc:210] Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error 2021-07-28 13:28:13.298862: F tensorflow/stream_executor/cuda/cuda_driver.cc:210] Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error 2021-07-28 13:28:13.750324: F tensorflow/stream_executor/cuda/cuda_driver.cc:210] Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error 2021-07-28 13:28:13.800403: F tensorflow/stream_executor/cuda/cuda_driver.cc:210] Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error 2021-07-28 13:28:15.548473: F tensorflow/stream_executor/cuda/cuda_driver.cc:210] Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error 2021-07-28 13:28:15.595047: F tensorflow/stream_executor/cuda/cuda_driver.cc:210] Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error ^CProcess ForkPoolWorker-19: Process ForkPoolWorker-23: Process ForkPoolWorker-22: Process ForkPoolWorker-17: Process ForkPoolWorker-18: Can you suggest what the cause of the problem is?

Heng-Z commented 3 years ago

Hi, @shahpnmlab Thanks for trying our software! I think the problem is that tensorflow version 2.5 is compatible with CUDA 11.2, as we can see from here . CUDA 11.4 is probably ok but you need to build tensorflow 2.5 from source, which is more complicated. So, I recommend installing CUDA 11.2 + cudnn 8.1 + tensorflow 2.5. This combination works well for me.

shahpnmlab commented 3 years ago

Hi Heng-Z, Thanks for your suggestion, I now have installed everything in accordance with the link you shared and yet I see this error pop up. Got any more suggestions? It clearly is a tensorflow related issue.

shahpnmlab commented 3 years ago

Hi Heng, After a little more investigation I have found the following behaviour -

if I run CASE 1: isonet.py refine subtomo.star --gpuID 0 --preprocessing_ncpus 2 the CUDA_ERROR_NOT_INITIALIZED: initialization error pops up. BUT, if I run,

CASE 2: isonet.py refine subtomo.star --gpuID 0,0 --preprocessing_ncpus 2 the error message is suppressed. Interestingly, the processing does not run on GPUs To extend this experiment further, I ran

CASE 3: isonet.py refine subtomo.star --gpuID 1 --preprocessing_ncpus 2 Note that my box has only 1 GPU with GPUID 0, and the behaviour is the same as CASE 2. I hope this helps in debugging.

shahpnmlab commented 3 years ago

Ah! So I think I have fixed the problem, but purging the installation and the virtual env for it and reinstalling it from scratch. I will close this issue for now.