Imporove apex - Githubissues

ymd-h commented 3 years ago

This PR is for #117

Update cpprb and use the new class MPPrioritizedReplayBuffer (multi-process version PER), which doesn't requires manual lock
- MPPrioritizedReplayBuffer doesn't lock whole buffer but critical section only
Remove busy loop and use efficient waiting by queue's get() method.
Remove slow proxy, SyncManager
Reduce access for shared value, trained_steps
- Fix atomic increment, too. (+= requires manual lock. See doc)
Run learner at main process instead of background process
Remove writer.flush() (I just let TensorFlow flush. We might need to adjust flush timing for our needs)
Remove eps (small value) addition to TD, which is added in buffer.

This improvement have larger effect for small network and/or simple Env.

I tested by running example/run_apex_dqn.py with default "CartPole-v0" on CPU machine. Please test other Envs and/or on GPU.

P.S. Weights distribution with multiple queues seems to be inefficient because of multiple copying. I will continue to consider other solution.

keiohta commented 3 years ago

Hi @ymd-h , thanks for brilliant PR!! I would really appreciate your continuous support!

I checked changes of codes, and I think all of them would contribute improvement of ApeX performance.

P.S. Weights distribution with multiple queues seems to be inefficient because of multiple copying. I will continue to consider other solution.

Yeah, this is true. I also consider some workaround. Thanks!

ymd-h commented 2 years ago

@keiohta

I updated the PR.

On my local machine, (a part of) logs at example/run_apex_dqn.py are followings;

Before PR

After PR

It seems that Super Linter v3 is strangely broken and we need to upgrade to v4. (I will add soon.) https://github.com/github/super-linter/issues/2253

keiohta / tf2rl