vitchyr
commented
5 years ago Did you modify the example script? The fact that the Average Returns start at -40 seems odd. Also, I wouldn't actually look at "Average return" too much. Distances in the latent space can be hard to interpret. Here's an example of what a run should look like, which also plots the most intuitive metrics (final hand/puck distance):
I made this plot using my version of viskit. I'll run more seeds now with the latest code, but this should work consistently.
tunglm2203
shivanikishnani
oracle.py works fine, by the way.
I ran the experiment with RIG + pusher with the original settings. Contrary to the paper, I cannot observe any improvement of the average return or success rate. How can I reproduce the original paper results?
Output after 100 epochs (388004 iterations)
hand_distance Mean 0.0369142 hand_distance Std 0.0125764 hand_distance Max 0.153462 hand_distance Min 0.0280608 Final hand_distance Mean 0.0406867 Final hand_distance Std 0.00171668 Final hand_distance Max 0.0432108 Final hand_distance Min 0.0369941 puck_distance Mean 0.14895 puck_distance Std 0.0542304 puck_distance Max 0.234878 puck_distance Min 0.0381613 Final puck_distance Mean 0.150567 Final puck_distance Std 0.0544439 Final puck_distance Max 0.234878 Final puck_distance Min 0.0528324 touch_distance Mean 0.0677295 touch_distance Std 0.00900345 touch_distance Max 0.105978 touch_distance Min 0.0565952 Final touch_distance Mean 0.0690568 Final touch_distance Std 0.00635284 Final touch_distance Max 0.0836148 Final touch_distance Min 0.0630353 success Mean 0 success Std 0 success Max 0 success Min 0 Final success Mean 0 Final success Std 0 Final success Max 0 Final success Min 0 QF1 Loss 0.171023 QF2 Loss 0.18778 Policy Loss 104.349 Q1 Predictions Mean -104.665 Q1 Predictions Std 88.4167 Q1 Predictions Max -4.22522 Q1 Predictions Min -336.314 Q2 Predictions Mean -104.558 Q2 Predictions Std 88.3845 Q2 Predictions Max -4.09563 Q2 Predictions Min -335.829 Q Targets Mean -104.699 Q Targets Std 88.4681 Q Targets Max -4.3236 Q Targets Min -336.498 Bellman Errors 1 Mean 0.171023 Bellman Errors 1 Std 0.416491 Bellman Errors 1 Max 3.98678 Bellman Errors 1 Min 1.80304e-06 Bellman Errors 2 Mean 0.18778 Bellman Errors 2 Std 0.355425 Bellman Errors 2 Max 2.20607 Bellman Errors 2 Min 3.32488e-06 Policy Action Mean 0.194368 Policy Action Std 0.731224 Policy Action Max 1 Policy Action Min -1 Test Rewards Mean -0.360232 Test Rewards Std 0.594497 Test Rewards Max -0.00478052 Test Rewards Min -5.77281 Test Returns Mean -36.0232 Test Returns Std 22.2522 Test Returns Max -13.6433 Test Returns Min -92.884 Test Actions Mean 0.0276301 Test Actions Std 0.247718 Test Actions Max 1 Test Actions Min -0.838328 Num Paths 10 Exploration Rewards Mean -1.33633 Exploration Rewards Std 0.762848 Exploration Rewards Max -0.225774 Exploration Rewards Min -5.57678 Exploration Returns Mean -133.633 Exploration Returns Std 60.5544 Exploration Returns Max -42.4458 Exploration Returns Min -214.126 Exploration Actions Mean 0.262311 Exploration Actions Std 0.44695 Exploration Actions Max 1 Exploration Actions Min -1 image_dist Mean 10.9785 image_dist Std 1.05965 image_dist Max 14.3582 image_dist Min 9.29165 Final image_dist Mean 10.8799 Final image_dist Std 1.03786 Final image_dist Max 12.6932 Final image_dist Min 9.39808 image_success Mean -0.727273 image_success Std 0.445362 image_success Max 0 image_success Min -1 Final image_success Mean -0.727273 Final image_success Std 0.445362 Final image_success Max 0 Final image_success Min -1 vae_dist Mean 0.360232 vae_dist Std 0.594497 vae_dist Max 5.77281 vae_dist Min 0.00478052 Final vae_dist Mean 0.216473 Final vae_dist Std 0.096578 Final vae_dist Max 0.361867 Final vae_dist Min 0.0465295 AverageReturn -36.0232 Number of train steps total 388004 Number of env steps total 101000 Number of rollouts total 1010 Train Time (s) 323.224 (Previous) Eval Time (s) 123.294 Sample Time (s) 115.072 Epoch Time (s) 561.59 Total Train Time (s) 29454.1 Epoch 100
Evolution of the average return: