Hello. Firstly, I would like to express my gratitude to the authors and contributors for their hard work on this project.
I've been training the model and have noticed that the GPU memory usage gradually increases after each epoch. I've gone through the code multiple times but haven't been able to identify any sections where memory isn't being released as expected. Could anyone share their insights on this issue? Or is it typical for memory usage to increase incrementally during this model network training, or should I be looking for a memory leak somewhere in the code?
Hello. Firstly, I would like to express my gratitude to the authors and contributors for their hard work on this project.
I've been training the model and have noticed that the GPU memory usage gradually increases after each epoch. I've gone through the code multiple times but haven't been able to identify any sections where memory isn't being released as expected. Could anyone share their insights on this issue? Or is it typical for memory usage to increase incrementally during this model network training, or should I be looking for a memory leak somewhere in the code?
Thank you in advance for your help!