search

aws-solutions-library-samples / guidance-for-training-an-aws-deepracer-model-using-amazon-sagemaker

DeepRacer workshop content. This Guidance demonstrates how software developers can use an Amazon SageMaker Notebook instance to directly train and evaluate AWS DeepRacer models with full control
https://aws.amazon.com/solutions/guidance/training-an-aws-deepracer-model-using-amazon-sagemaker/
MIT No Attribution
1.24k stars 714 forks source link

Improving the Log Analysis notebook #58

Closed sunil19m closed 4 years ago

sunil19m commented 4 years ago

This addresses the following

  1. The plot_track function seems to be really low resolution, so the heatmap in the ‘Analyze the reward distribution for your reward function’ block is showing the heat centers only centered on whole numbers, like (2,3) or (4,0). I’d modified the plot_track in my version (https://github.com/barberd/aws-deepracer-workshops/blob/d1bb8296d60e2fdb98e69734ca8062ab654bf386/log-analysis/DeepRacer%20Log%20Analysis.ipynb) to not do this.

  2. The track selected is now reInvent2019_track (in the ‘Load waypoints for the track you want to run analysis on’ block) but the track segments as defined in the track_segments variable in the ‘Action breakdown per iteration and historgram for action distribution for each of the turns - reinvent track’ block have not been updated to match

  3. In the same block ‘Action breakdown per iteration and historgram for action distribution for each of the turns - reinvent track’ the line for ax.text() adds ‘+20’ to the y-coordinate. This seems to not be needed and just makes all the labelsway off from where they should be.

  4. Same block, the action list doesn’t really make sense. Its currently set to ['LEFT', 'RIGHT', 'STRAIGHT', 'SLIGHT LEFT', 'SLIGHT RIGHT', 'SLOW'] which is 6 actions, but the ‘original deepracer’ has 9 nodes in its action space. The only way to get 6 nodes in the garage is a steering granularity of 3 and speed granularity of 2, and the action list should be more like ‘LEFT SLOW’,’LEFT FAST’,’STRAIGHT SLOW’,’STRAIGHT FAST’,’RIGHT SLOW’,’RIGHT FAST’ if this was selected. (I did not fix this in my notebook)

The other changes made on top of barberd code

  1. The ‘Download all the checkpoints (provided as an example)’ block no longer works. The s3_bucket and s3_prefix variables aren’t set anymore, and moreover, this technique no longer works after DeepRacer’s new service model changes. This shouldbe modified to use the new API for get_asset_url such as done in my notebook, https://github.com/barberd/aws-deepracer-workshops/blob/d1bb8296d60e2fdb98e69734ca8062ab654bf386/log-analysis/DeepRacer%20Log%20Analysis.ipynb (with box renamed to ‘Download and extract the trained model’)

  2. The ‘Evaluation Run Analyis’ is still relying on the old ‘simulation_name’ variable to grab from robomaker, which is nolonger accessible after the new service model. This section should either be modified to use the get_asset_url API such as my notebook does, or should be eliminated entirely. It seems the intent is the latter as I noticed the is_training Boolean up in the ‘Jobs run from AWS DeepRacer Console. Download the desired log file by providing model name’ block.

  3. Training metrics and evaluation metric graphs which the internal team used for analysing whether the training job has converged or some more training will help the job to converge. Along with percentage completion

wentzeld commented 4 years ago

Thanks sunil19m!