search

dotchen / LearningByCheating

(CoRL 2019) Driving in CARLA using waypoint prediction and two-stage imitation learning
MIT License
308 stars 102 forks source link

Explicitly delete device-converted tensors. #19

Closed tomcur closed 4 years ago

tomcur commented 4 years ago

Fixes https://github.com/dianchen96/LearningByCheating/issues/4.

dotchen commented 4 years ago

I don't think this is a fix as the python garbage collector would automatically handle this.

tomcur commented 4 years ago

The garbage collector does not handle this, as the objects live at least until the variables fall out of scope or are reassigned. A for-loop doesn't create a new scope. That means in the current code memory usage of input tensors is doubled.

More information here: https://github.com/dianchen96/LearningByCheating/issues/4#issuecomment-615429508

dotchen commented 4 years ago

This would not be an issue unless your GPU does not have the mem to hold 2x of the data, no? A standard deep learning GPU (1080Ti/Titan) would be able to handle the default batch-size.

tomcur commented 4 years ago

That's right; it's likely only an issue when trying the code on simple development machines. Do note that train_birdview.py, for the code's default batch size of 256, "wastes" 3GB vmem. It's good practice to del tensors when you no longer need them.