search

leggedrobotics / raisimGym

Other
140 stars 45 forks source link

The environment is getting slower as training goes #30

Closed WSPeng closed 4 years ago

WSPeng commented 4 years ago

Hi,

First, it is an amazing project! the simulation is very fast. But the speed(fps) is getting slower and slower as the training goes. If I restart the environment, the speed back to very fast again. Is this a universal problem? Do you have an idea about the cause of it?

thanks Shupeng

jhwangbo commented 4 years ago

Hi Shupeng, I have the same issue and this has been bothering me for a week too. For me, the iteration time increases from 5.7 sec to 8 sec. It increases until 1500 iterations and becomes constant. I am looking into it. I am monitoring the memory usage and there is no evidence that memory is leaking (both from cpu and gpu). I can run more than 30,000 iterations without any issue.

I'll let you know if I figure this out. I'll also appreciate if you look into this issue.

jhwangbo commented 4 years ago

Can you elaborate on how is your training time varying over iterations?

jhwangbo commented 4 years ago

my issue was not a bug. I had a curriculum set up and that just made the environment harder to simulate. So your issue is also most likely to be related to your environment.

jhwangbo commented 4 years ago

cannot reproduce this bug. Closing until further posts

WSPeng commented 4 years ago

@jhwangbo Hi Jemin, I confirm that it is not a bug. It was my problem. I added a ramp into the quadropod environment, but I didn't reset the position of the ramp when reset() is called, so the simulation is getting slower...