Training browser

[a3ahmad]

Types of changes

Adds a simple browser to chart training results from multiple runs

• [ ] Docs change / refactoring / dependency upgrade
• [ ] Bug fix (non-breaking change which fixes an issue)
• [X] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Motivation and Context / Related issue

Adds a quick and easy way to browse/compare results

How Has This Been Tested (if it applies)

I ran a few different training runs, with different algorithms and use this to compare them

Checklist

• [ ] The documentation is up-to-date with the changes I made.
• [X] I have read the CONTRIBUTING document and completed the CLA (see CONTRIBUTING).
• [ ] All tests passed, and additional code has been covered with new tests.

[luisenp]

@a3ahmad This looks potentially pretty nice! Would you mind adding some brief instruction on how to run so that I can play around with it before making any suggestions?

Also, probably the best place to put this would be in the diagnostics folder (we might change later this folder to something like "visualizations").

[a3ahmad]

Sure, just run it with 'python training_browser.py' (you will need PyQt5 and PyQtChart installed -- pip and conda provide it)

You'll see a full size window with a (dockable) pane on the left listing all the available experiments in a table. Click on each to chart their results (or shift-click to select/overlay multiple results). The toolbar currently has check-boxes to use a logarithmic axis, and to filter out outliers (I don't like the current implementation as it removes the points, rather than just rescale the Y axis)

[a3ahmad]

Ok, I've moved it to diagnostics. So now you can run it with:

python -m mbrl.diagnostics.training_browser

I run it from the root mbrl_lib folder and it will infer current_working_directory/exp. Alternatively you can run:

python -m mbrl.diagnostics.training_browser --experiments [path to the experiment folder]

[luisenp]

This is awesome @a3ahmad, thanks a lot! A couple of suggestions:

• Rather than reading the experiment description from the path, it would be better to read the yaml config generated by Hydra, which is saved in .hydra/config.yaml, under the same folder that has results.csv. There we can get some of the same information (env, algorithm, experiment_name) except the date/time; we can also get other info like the seed used. This is to avoid users that have a different folder structure to get weird info in the browser. For example, when you run an algorithm with --multirun in Hydra, you'll get an extra folder with an integer index, and in this case the current browser code crashes. I could potentially add some date info in a metadata file (or maybe save it in the config) to keep it available for the browser, if you think that's useful.
• Can we add another tab to the browser that plots the mean and standard deviation plot over the different results, as shown below? We can probably pre-compute the data frame created in line line 117, and then serve depending on the index row clicked. This only makes sense for things that have the same experiment_name+env+algorithm config+datetime though.

Would you be OK with adding this changes?

[luisenp]

BTW, black was updated recently which broke the CI tests. I've already updated master, so you can just rebase and force push here.

[a3ahmad]

Those are great suggestions, I’ll get on them

[a3ahmad]

@luisenp, what kind of rolling window do you use for the mean/std-dev plots?

[luisenp]

@luisenp, what kind of rolling window do you use for the mean/std-dev plots?

@a3ahmad For the plot like the one I showed I usually just plot with no smoothing. Just average/std over multiple runs (in the case of that plot it was 10 seeds), for all time steps. We could add the window as an optional argument, seems like a useful feature.

[a3ahmad]

Ohh, I got you. I need to experiment with multiple runs!

[luisenp]

Hi @a3ahmad, thanks again for working on this! I noticed you added a checkbox labeled "Display As Distribution", but currently appears greyed out for me. Is there anything I need to do to enable it?

[a3ahmad]

Oh! You need you multiselect several runs of the same length.

The graph doesn’t look right to me, though. I need to investigate.

[a3ahmad]

Updated to swap from QtCharts to matplotlib. Simplifies code and works better.