search

google-research / batch_rl

Offline Reinforcement Learning (aka Batch Reinforcement Learning) on Atari 2600 games
https://offline-rl.github.io/
Apache License 2.0
520 stars 73 forks source link

Retraining the online agent #8

Closed n17s closed 3 years ago

n17s commented 3 years ago

I'd like to retrain the online DQN agent in order to log some additional data during online training. The README says

This data can be generated by running the online agents using batch_rl/baselines/train.py for 200 million frames (standard protocol)

However, this is not enough information to accurately replicate the setup. Could you share the gin file that was used? Is it the same as the one in dopamine?

Also, is there a faster way to accurately replicate the online training and logging via the RL unplugged project, or is it just the offline part that has become faster?

agarwl commented 3 years ago

Yes, it is the same as original gin config in dopamine and also present here. The agents in LoggedDQNAgent in this repo here were used to collect the data. You can just use the run_experiment.py script to launch these experiments.

Regarding the online training replication, Acme might have a faster version of training online agents too (not sure how fast it is since the bottleneck is online data collection from Atari environments). RL Unplugged creates TFRecord version of the offline Atari dataset, so it only focuses on the offline part. That said, logging experiments would take about 4-5 days per game run for online Atari with Dopamine.

n17s commented 3 years ago

This is very helpful! Thanks!