search

ninia / jep

Embed Python in Java
Other
1.27k stars 146 forks source link

Jep did not free GPU memory when it was closed #540

Open SHITINGCHEN opened 2 weeks ago

SHITINGCHEN commented 2 weeks ago

Describe the bug Using jep to run tensorflow model on a GPU machine. I suppose it will release GPU memory after jep interpreter was closed. But it was not. When running a single python script using python, the GPU memory will be released when python process finished. But now java was the process who retains the GPU usage. How can I release these GPU memory when Jep was closed but jvm is still running?

Environment (please complete the following information):

bsteffensmeier commented 2 weeks ago

Jep does not allocate or free any GPU memory and I do not think this is something we can do. Jep will free any unused python objects when they are no longer references but things like modules and native extensions may not be freed. You may be able to free additional memory by manually clearing sys.modules but a guick search shows that tensorflow does not free GPU until a process exits and that is not something jep can control.

SHITINGCHEN commented 1 week ago

Jep does not allocate or free any GPU memory and I do not think this is something we can do. Jep will free any unused python objects when they are no longer references but things like modules and native extensions may not be freed. You may be able to free additional memory by manually clearing sys.modules but a guick search shows that tensorflow does not free GPU until a process exits and that is not something jep can control.

Is it possible to call a new Process (by multiprocess) in SharedInterpreter? Running tensorflow in Process in pure python did free gpu memory. But I didn't make Process work in Jep. It will throw pickle problem.

bsteffensmeier commented 1 week ago

Multiprocessing has worked in the past but I do not think it works well with recent python versions. There may be ways to get it working but that is not something anyone has had time to investigate and document, See #228 for more detailed background background.