med-air / SurRoL

[IROS'21] SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning
https://med-air.github.io/SurRoL/
MIT License
119 stars 19 forks source link

GPU memory overflow when execute reseting the environment for multiple times #9

Closed fujiawei0724 closed 11 months ago

fujiawei0724 commented 11 months ago

When running the following commands:

import gym

env = gym.make('NeedlePick-v0')

for _ in range(100000000):

env.reset()

My GPU occupied memory will gradually overflow. Would you please help me solve the problem? Thank you very much

TaoHuang13 commented 11 months ago

Hi Jiawei,

Thanks for your question. I have run the commands with 10K iterations, and it seems no overflow is found (mine is 24G memory capacity).

If you are running DEX or ViSkill, I remembered that 4G memory will be enough to handle 100K iterations without overflow problems.

So would you please try to use a GPU with more memory, or provide more details to see through that deeply?

fujiawei0724 commented 11 months ago

Thanks for your prompt answer. It seems like the pybullet version cause this problem. I want to know your pybullet version when you run the code, since the surrol only instructs the pybullet >= 3.0.7, which is not a specific one. I guess the problem will be solved if I hold the same version of pybullet with you. Thank you very much. Looking forward to your reply~

fujiawei0724 commented 11 months ago

I have figure out the problem, the memory leak is caused by the high version of pybullet. The version 3.0.9 has solved the problem, but in the following version, the problem reappear. In a word, the pybullet version should >= 3.0.7 and <= 3.0.9. Thanks for your help again.