[question] Training speed

Kaixhin / Rainbow

Rainbow: Combining Improvements in Deep Reinforcement Learning

MIT License

1.56k stars 282 forks source link

[question] Training speed #71

Closed stringie closed 4 years ago

stringie commented 4 years ago

Hey Kai, just a quick question:

I cloned your repo to try and run it myself and I'm just wondering if I am getting the correct numbers here:

When on cuda (using a RTX 2080 SUPER) I'm getting about 300it/s
When on cpu (using i5-9600) I'm getting about 100it/s

I just want to know if you are also getting similar numbers or if I'm doing something wrong and getting way slower GPU speeds.

Kaixhin commented 4 years ago

You're getting a 3x speedup on GPU, so it's reasonable? Bottleneck is that there's a lot of data transfer with RL, but speedup mainly comes with training on minibatches I believe. I'm running other stuff on my one PC atm so can't give you full speeds, but getting 53it/s on GPU (GTX 1080 Ti) vs. 13it/s on CPU (i7-3820), so similar speedup ratio on my side.

stringie commented 4 years ago

Thank you for your answer. If you ever decide to run Rainbow again could you please share those speeds here for me ... if you remember to of course. I'm asking all this because I find it worrisome that for the process to complete I have to wait (as it says next to the it/s) 66h ... which seems crazy to me. Do you also wait around that much when training your models? Thanks again.

Kaixhin commented 4 years ago

Not sure I'll have a chance any time soon, but training definitely takes on the order of a week. It's very sequential, takes pixel inputs, and has a lot of environment interactions, so it is naturally slow.