uci-rendering / psdr-cuda

Path-space differentiable renderer
BSD 3-Clause "New" or "Revised" License
155 stars 11 forks source link

Memory leak when optimization #15

Closed l1346792580123 closed 2 years ago

l1346792580123 commented 2 years ago

When I try to optimize the vertices positions using the provided optimization code here #11. The GPU memory usages increases along iterations and eventually out of memory. How can I avoid memory leak?

andyyankai commented 2 years ago

you will need to clear the grad(like zerograd) / delete loss, or call torch.cuda.empty_cache() or ek.cuda_malloc_trim() every time you finish the opt.step()

l1346792580123 commented 2 years ago

I have deleted all the variable generated by enoki in backward operation and call ek.cuda_malloc_trim every step. However, the occupied video memory fluctuates up and down but still gradually increases. Is there any clue?

andyyankai commented 2 years ago

Sound pretty strange, I can take a look on your code if needs

l1346792580123 commented 2 years ago

Here is my optimization code code.zip. At the beginning, 3G gpu memory is required. As the number of iterations increased, the gpu memory usage fluctuates up and down but gradually increases to 5g.

joyDeng commented 2 years ago

It seems that,if you configure the scene,by scene.configure ever iteration,it will up load geometry to gpu which increase the memory graudually.

andyyankai commented 2 years ago

image

andyyankai commented 2 years ago

Please do the following update and see if the problem gone. Sorry for the late update

l1346792580123 commented 2 years ago

I have done the update as you said, and there is no memory leak now. Thank you very much!