search

serizba / cppflow

Run TensorFlow models in C++ without installation and without Bazel
https://serizba.github.io/cppflow/
MIT License
779 stars 177 forks source link

Tensorflow doesn't release GPU memory #170

Closed ghost closed 2 years ago

ghost commented 2 years ago

It is something to think about but not necessarily have to be done in cppflow. Tensorflow never releases any GPU memory so the handles must be saved. Deletig them doesn't release the GPU memory. The solution is not to delete the tensor handles and let the application handle recycling them or to make a tensor manager code. If the size is the same you can just recycle it, if it isn't you have to find the closest and resize it.

I just modified cppflow with a dummy delete and keep track of the handles for recycling.

ghost commented 2 years ago

I'm not sure if the memory is reclaimed when you delete a context. I wondered if because the global context is static that memory is never released back to be used again. I created temporary operations and it quickly will use all memory because it is never returned to the GPU heap so I am recycling them.

dskkato commented 2 years ago

It seems that any tensor/handle are not deleted correctly in the cppflow, which may cause memory leaks.

I missed the part about calling the destructors through smart pointers.

ghost commented 2 years ago

How should it be deleted? There is a smart pointer that is supposed to delete the tensor. But, the GPU memory is not released and burns out quick.

ghost commented 2 years ago

I've come to the conclusion Tensorflow is just a waste of time for me. Thanks for the code so I could try it but I will be using Torch from now on as I already know Torch7 and Tensorflow is just way too cryptic and dense and I have no interest in working for google at all.

ghost commented 2 years ago

We can prove if it is cppflow or not by making a loop and allocating and releasing alot of tensors to see if Tensorflow recycle memory, but I am ditching tensorflow anyways. Thanks