juglab / n2v

This is the implementation of Noise2Void training.
Other
387 stars 107 forks source link

N2V in Jupyter Notebook and GPU Memory Issue #99

Open citypalmtree opened 4 years ago

citypalmtree commented 4 years ago

Hi,

I am currently running N2V in Jupyter Notebook. In the past, I would run the training module, and restart the kernel to free GPU memory so that I could run the predict module.

Now, I was wondering if I can run both modules simultaneously without having to restart the kernel. Is there something that terminates TF to conserve GPU memory in the code? something like 'tf.keras.backend.clear_session()'?

I also tried to run the training module in multiprocess, hoping that the session would end and the memory would be freed, but I had to close Jupyter Notebook entirely to free up that GPU in multiprocess.

Any advice would be greatly appreciated! Thank you.

tibuch commented 3 years ago

Hi @citypalmtree,

In my experience running training and predicition sequentially in the same script/notebook works.

The problem of running first the training notebook and then the prediction notebook is, that two kernels are started. But if you have training and prediction running on the same kernel, the GPU memory will be allocated by the same kernel and can be reused.