search

devendrachaplot / Neural-SLAM

Pytorch code for ICLR-20 Paper "Learning to Explore using Active Neural SLAM"
http://www.cs.cmu.edu/~dchaplot/projects/neural-slam.html
MIT License
761 stars 144 forks source link

Memory Leak #21

Closed ghost closed 3 years ago

ghost commented 3 years ago

Hi!

We trained the model from scratch on Gibson dataset with the setting that num_scenes = 72, but we found that once the program is running, the avaliable memory of our machine is going lower and lower, and exhausted after about 2 hours. We found it leaks over time and stop leaking after we terminate the program.

The curve of avaliable memory is shown as follows.

image

We also found that if we reduce the number of scenes, the speed of memory leaking will also decrease.

It will be very helpful if someone can suggest the possible solution.

ghost commented 3 years ago

Hi!

Too long FIFOMemory length may be the cause of Memory Leaking.

Default value of slam_memory_size is set to 500000, the traning data will be appended to the FIFOMemory untill the FIFO length reaches the maximum of 500000, which continuously takes up a lot of memory.

We set slam_memory_size to 720 hence the queue stop growing and stop consuming the memory.

The large default value of slam_memory_size is confusing. Does it matter a lot to the performance of the model?

devendrachaplot commented 3 years ago

Yes, the memory increase is due to the FIFO memory. In our experiments, we found the memory helps in improving sample efficiency but not the final performance, if you reduce the memory size, the model might require more frames to reach the same performance.