MillionIntegrals
commented
5 years ago Hi,
Thank you for using it, and asking questions ;) There isn't really any documentation yet, and probably will not be for some time but I'm very happy to answer any questions.
There are multiple ways in which you can access the output. During training, all metrics are stored and recorded in MongoDB (for long-term storage) and in visdom (for visualization). The way how I work with it, is that I use Visdom to visually inspect if training is doing what it should and then potentially load results with Python from MongoDB to process them for various reasons.
That's how the example Visdom chart looks like:
Metrics stored in the DB:
> db.metrics.find({'model_name': 'breakout_a2c'})
{ "_id" : ObjectId("5bc5310155cd970dcb4f25f5"), "epoch_idx" : 0, "frames" : 8000, "fps" : 1489, "PMM:episode_rewards" : 1.878787878787879, "P09:episode_rewards" : 3.8000000000000007, "P01:episode_rewards" : 0, "episode_length" : 197.0909090909091, "value_loss" : 0.02023520267427557, "policy_entropy" : 1.3809912633895873, "policy_loss" : 0.006739899950334802, "grad_norm" : 0.09930057158654704, "advantage_norm" : 1.4973756862804293, "explained_variance" : -0.25901189506053923, "model_name" : "breakout_a2c", "run_name" : "breakout_a2c/1" }
{ "_id" : ObjectId("5bcc22ee55cd974f28595b33"), "epoch_idx" : 1, "frames" : 6000, "fps" : 1391, "PMM:episode_rewards" : 1.2142857142857142, "P09:episode_rewards" : 3, "P01:episode_rewards" : 0, "episode_length" : 178.75, "value_loss" : 0.01603925398577303, "policy_entropy" : 1.3837194800376893, "policy_loss" : 0.0020585690875304864, "grad_norm" : 0.12393458011152558, "advantage_norm" : 0.9929843093454838, "explained_variance" : -1.671451603770256, "model_name" : "breakout_a2c", "run_name" : "breakout_a2c/2" }
{ "_id" : ObjectId("5bcc22f355cd974f28595b34"), "epoch_idx" : 2, "frames" : 12000, "fps" : 1311, "PMM:episode_rewards" : 1.2, "P09:episode_rewards" : 3, "P01:episode_rewards" : 0, "episode_length" : 178.06666666666666, "value_loss" : 0.018758475972190353, "policy_entropy" : 1.3858475637435914, "policy_loss" : 0.005217389368335716, "grad_norm" : 0.0655856565352905, "advantage_norm" : 1.2349432955682278, "explained_variance" : 0.013082603216171265, "model_name" : "breakout_a2c", "run_name" : "breakout_a2c/2" }I have also implemented to output metrics what I call in the "OpenAI" format, that is used in their baselines repository, but it is off by default - That's why you see output/openai directory empty. To enable that, two changes need to be done in the config file:
Set logging to true in the command definition will log metrics to the CSV file vel/output/openai/breakout_a2c/0/progress.csv
commands:
train:
name: vel.rl.commands.rl_train_command
total_frames: 1.0e6
batches_per_epoch: 20
openai_logging: trueIf you turn on monitor in the environment, some information about environment episodes will be written to the file vel/output/openai/breakout_a2c/0/0.monitor.csv etc:
env:
name: vel.rl.env.classic_atari
game: 'BreakoutNoFrameskip-v4'
env_settings:
default:
monitor: trueWhen you run the train command, the command vel.rl.commands.rl_train_command will run, which is a generic reinforcement learning training loop. What it does it trains the model on an environment rollouts, stores the metrics in a set of locations discussed above and stores model checkpoints in output/checkpoints directory.
To access models that were checkpointed to e.g. record video I've used different commands. So for example if you change your command line to
vel examples-configs/rl/atari/a2c/breakout_a2c.yaml recordThe record command will be invoked which is configured in this bit of config file:
record:
name: vel.rl.commands.record_movie_command
takes: 10
videoname: 'breakout_vid_{:04}.avi'
frame_history: 4
sample_args:
argmax_sampling: trueThat will run saved model on 10 environment episodes and store the videos from these runs into output/videos directory.
There are also some other commands defined, like evaluate:
evaluate:
name: vel.rl.commands.evaluate_env_command
takes: 100
frame_history: 4
sample_args:
argmax_sampling: truewhich does run model checkpoint on an environment 100 times to calculate some statistics over rewards/episode lengths to evaluate models to each other.
There is also a visdom command, which basically copies saved down metrics from MongoDB to visdom, as any data in visdom is lost on restart, so I needed a way to repopulate that data to view it:
visdom:
name: vel.commands.vis_store_commandI hope I've answered your question, feel free to let me know if anything is unclear ;) I just realized one of my refactorings broke movie recording in the latest master commit, I'll get that fixed soon.
yngtodd
Thanks for putting together this library!
I have installed the library on a headless server, along with MongoDB and Visdom. Is there a way to view the results after running from the .yaml configs?
I am testing it out with the example
Everything trains fine, but then when I look at the logfile at
vel/output/openai/breakout_a2c/0/log.txt, it only saves the following:And the
progress.csvat that directory level is empty.When looking at the yaml config at https://github.com/yngtodd/vel/blob/master/examples-configs/rl/atari/a2c/breakout_a2c.yaml#L57, I see that it is saving a video. Where is that stored?
Thanks!