Open VolkerH opened 3 years ago
ooh that's good to know! @VolkerH I always get this issue when I run the script on a computer with 2GB GPU while Google Colab has no issues. The error message doesn't appear when I just run the script again without doing anything.
Would you mind explaining what TF_FORCE_GPU_ALLOW_GROWTH=true does? I also have a GPU memory issue such that I have to restart the kernel between training and prediction. Not to mention the above issue, which I am hoping your solution fixes my case.
Are you able to run N2V training and prediction at one run without having to restart the kernel? I am wondering if there is a way to clear GPU memory programmatically after training is complete.
There is some background to how the environment variable changes the behaviour here:
https://www.tensorflow.org/guide/gpu under the subheading "Limiting GPU memory growth".
I cannot both have the training and prediction notebook running at the same time due to memory limitations. However, if you still have the model in memory (after training) rather than trying to create a new additional model I believe this could fix it.
My understanding is that without this option, tensorflow grabs pretty much all available GPU memory initially. If you then even need to allocate a tiny bit more it will fail. I also don't understand what other parameters affect it. As I mentioned, on a different machine (windows) with less GPU VRAM I do not need to set any option like that and I do not see the error.
Hi @VolkerH,
Thank you for reporting this issue here as well. I am wondering if there might be an issue with the compatibility between keras, tensorflow, CUDA, and nvidia-driver versions. I will put a note into the README.md with your proposed fix.
@citypalmtree interesting that your issue disappears if you rerun the notebook without changes. Do you restart the kernel between runs?
Regarding training- and prediction-notebooks: It is like @VolkerH says, as long as you stay in the same kernel/notebook you can run training and prediction sequentially. But unfortunately tensorflow allocates all available GPU memory (if not told otherwise), which means that a second kernel/notebook will find no GPU memory left and fail.
One way to limit GPU memory would be this:
from csbdeep.utils.tf import limit_gpu_memory
limit_gpu_memory(0.5)
This would only allocate 50% of the GPU memory.
Doing this would allow you to train two models in parallel on one GPU, but most likely it will take much longer. First you can only use half of the data and second data for two independent models has to be transferred to the GPU, which could lead to inefficient processing.
I spent yesterday afternoon installing n2v on a new machine (Ryzen 5, RTX 2060, Ubuntu 20.04, conda) and ran into tensorflow-related issues with n2v. The error occurs when running training in any of the example notebooks. The error message was cuDNN related (see full traceback below), so I suspected a library version problem. I tried various versions of tensorflow-gpu such as 1.14, 1.15 and versions installed with pip or from conda using the anaconda and conda-forge channels. Also tried various versions of the CUDA toolkit and python 3.6 and 3.7. All without success.
While there was enough GPU VRAM available, it turns out that this is related to GPU memory management in tensorflow. Setting the following environment variable
fixed the issue. This is not specific to
n2v
, in fact I found the answer in a thread related to DeepLabCut (https://forum.image.sc/t/could-not-create-cudnn-handle/24276/17).I am putting this here so that others who run into the issue can find it. I am not sure how common it is (I did not encounter the issue when installing n2v on Windows) and whether it warrants mentioning in the README.md file.