Closed zach-nervana closed 5 years ago
Thanks a lot @zach-nervana for the detailed analysis! Resolving the communication and serialization bottlenecks are great improvements to the distributed runs code path.
Having a transition added to to the replay buffer taking 1 msec, indeed sounds like too much. It might be the result of the the mutexes laid out throughout the replay buffer code, used to allow multiple workers to read/write to the same buffer, but need to debug to make sure.
The good news is that the previous bottleneck in serializing, communication and deserialization of the policies is no longer the primary bottleneck. The bad news is that given the inefficiencies of the
emulate_*
code path, we are still unable to benefit significantly from additional rollout workers in the case of PPO on the humanoid environment. It is likely that more difficult to simulate environments will see more benefit with this set up as is. More details are below.In order to run this code, run
make distributed
from thedocker
directory. You must also have access to a kubernetes cluster configured locally and coach installed locally.The total time between the end of the evaluation phase, and the processing of the first rollout in the next training phase is less than
200ms
. This includes the time required to serialize the policy, send the policy from the training worker to the rollout worker, deserialize the policy and load into tensorflow, evaluate a rollout, serialize the episode transitions, send the episode transitions back to the master, and finally run these episodes through the master trainingemulate_*
code path.No longer is policy serialization, communication and deserialization the primary bottleneck. It appears that the primary bottleneck is now the serial processing of transitions by the training process. You can see above that the training worker requires approximately
(32.185998 seconds - 31.177425 seconds) / (288270 steps - 287295 steps) * (1000 ms / 1 s) -> 1.03 ms / step
. At 4000 steps_between_evaluationperiods, that comes out to around 4s per training period. This isn't even including computing gradients and updating policy weights. This is only in pushing the transitions through the `emulate*` code path. Luckily, with further refactoring, this should be able to be greatly reduced as the processing of these transitions does not need to happen sequentially as it does now. In theory, it should be possible to process them in parallel. I suspect there is also overhead coming from somewhere else. It should not take 1 ms to add a transition to a replay buffer, which is essentially all that is happening here.