search

ChanganVR / RelationalGraphLearning

[IROS20] Relational graph learning for crowd navigation
132 stars 41 forks source link

How to shorten the training time? #6

Closed Affan-Hillergrand closed 3 years ago

Affan-Hillergrand commented 3 years ago

Hello! Your excellent work is very helpful to me. However, I had some questions when using your code training strategy. Now I am using an NVIDIA gtx1080 graphics card, which takes about 20 hours for 10000 episodes. And the utilization rate of graphics card computing power in training is very low. May I ask you about the equipment you used when you were training and how long did it take you to train each time? And are there any techniques that can be used to improve training efficiency? Thank you!

ChanganVR commented 3 years ago

Hi, thank you for your interest. I trained the policy on CPU and the training time was roughly the same as yours or even less. I experimented with GPU training and found it did not speed up the trainig. I think the reason is that the crowd simulation is done on CPU and the bottleneck is on the crowd simulation. Some optimization should definitely help speed up the simulation & training, however, I don't have time to investigate this. But I encourage you to explore that as it would help you gain a deeper understanding of the code.

Affan-Hillergrand commented 3 years ago

Thank you for your helpful reply. I will continue to explore this code and try to speed up the training process.