Value loss is Increasing

yamatokataoka commented 4 years ago

The value loss is increasing while training compared to OpenAI Spinning Up implementation.

the current implementation is increasing from 178 to 451, whileSpinning Up's value loss is decreasing from 253 to 171 for example.

for now, I confirmed below parameters are the same between the implementations:

the number of the network parameters: policy: 4610, value function: 4545
network architecture (two hidden layers with 64 units)
total environment interactions
number of value function updates
learing rate both on policy and value function
gym environment: CartPole-v0

yamatokataoka commented 4 years ago

The current implementation stats

``` 2020-10-08 01:52:04,750 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 7 steps on the current episode 2020-10-08 01:52:04,916 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: 0.02 2020-10-08 01:52:04,917 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 179 2020-10-08 01:52:04,917 — rl_replicas.vpg.vpg — INFO — Epoch: 0 2020-10-08 01:52:04,917 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 18.9 2020-10-08 01:52:04,918 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 8.93 2020-10-08 01:52:04,918 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 50.0 2020-10-08 01:52:04,918 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 7.136378645896912 2020-10-08 01:52:04,918 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 18.9 2020-10-08 01:52:04,919 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 0.13 2020-10-08 01:52:04,919 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0668 2020-10-08 01:52:04,919 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 0.254 2020-10-08 01:52:04,919 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -0.0984 2020-10-08 01:52:04,920 — rl_replicas.vpg.vpg — INFO — Total env interactions: 4000 2020-10-08 01:52:04,920 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.02 2020-10-08 01:52:04,920 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 178.951 2020-10-08 01:52:04,920 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.672 2020-10-08 01:52:04,921 — rl_replicas.vpg.vpg — INFO — Time: 2.12 2020-10-08 01:52:06,857 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 11 steps on the current episode 2020-10-08 01:52:07,005 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: 0.0165 2020-10-08 01:52:07,006 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 198 2020-10-08 01:52:07,006 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.0035 2020-10-08 01:52:07,006 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 19 2020-10-08 01:52:07,006 — rl_replicas.vpg.vpg — INFO — Epoch: 1 2020-10-08 01:52:07,007 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 19.6 2020-10-08 01:52:07,007 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 9.67 2020-10-08 01:52:07,007 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 63.0 2020-10-08 01:52:07,007 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 8.0 2020-10-08 01:52:07,007 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 19.6 2020-10-08 01:52:07,008 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 0.198 2020-10-08 01:52:07,008 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0605 2020-10-08 01:52:07,008 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 0.279 2020-10-08 01:52:07,008 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -0.0763 2020-10-08 01:52:07,009 — rl_replicas.vpg.vpg — INFO — Total env interactions: 8000 2020-10-08 01:52:07,009 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.0182 2020-10-08 01:52:07,009 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 188.430 2020-10-08 01:52:07,009 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.674 2020-10-08 01:52:07,009 — rl_replicas.vpg.vpg — INFO — Time: 4.21 2020-10-08 01:52:08,954 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 8 steps on the current episode 2020-10-08 01:52:09,102 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: 0.0109 2020-10-08 01:52:09,102 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 183 2020-10-08 01:52:09,103 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.0056 2020-10-08 01:52:09,103 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -14.7 2020-10-08 01:52:09,103 — rl_replicas.vpg.vpg — INFO — Epoch: 2 2020-10-08 01:52:09,103 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 19.6 2020-10-08 01:52:09,104 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 8.94 2020-10-08 01:52:09,104 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 57.0 2020-10-08 01:52:09,104 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 8.218967497348785 2020-10-08 01:52:09,104 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 19.6 2020-10-08 01:52:09,105 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 0.267 2020-10-08 01:52:09,105 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0506 2020-10-08 01:52:09,105 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 0.324 2020-10-08 01:52:09,105 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.0641 2020-10-08 01:52:09,105 — rl_replicas.vpg.vpg — INFO — Total env interactions: 12000 2020-10-08 01:52:09,106 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.0158 2020-10-08 01:52:09,106 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 186.701 2020-10-08 01:52:09,106 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.676 2020-10-08 01:52:09,106 — rl_replicas.vpg.vpg — INFO — Time: 6.31 2020-10-08 01:52:11,010 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 1 steps on the current episode 2020-10-08 01:52:11,158 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: 0.0105 2020-10-08 01:52:11,158 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 224 2020-10-08 01:52:11,159 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000353 2020-10-08 01:52:11,159 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 40.3 2020-10-08 01:52:11,159 — rl_replicas.vpg.vpg — INFO — Epoch: 3 2020-10-08 01:52:11,159 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 20.2 2020-10-08 01:52:11,160 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 10.8 2020-10-08 01:52:11,160 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 100.0 2020-10-08 01:52:11,160 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 1.3601664006710052 2020-10-08 01:52:11,160 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 20.2 2020-10-08 01:52:11,161 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 0.34 2020-10-08 01:52:11,161 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0424 2020-10-08 01:52:11,161 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 0.382 2020-10-08 01:52:11,161 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.13 2020-10-08 01:52:11,161 — rl_replicas.vpg.vpg — INFO — Total env interactions: 16000 2020-10-08 01:52:11,162 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.0145 2020-10-08 01:52:11,162 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 195.922 2020-10-08 01:52:11,162 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.677 2020-10-08 01:52:11,162 — rl_replicas.vpg.vpg — INFO — Time: 8.36 2020-10-08 01:52:13,092 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 10 steps on the current episode 2020-10-08 01:52:13,236 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: 0.00519 2020-10-08 01:52:13,237 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 207 2020-10-08 01:52:13,237 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00535 2020-10-08 01:52:13,237 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -16.4 2020-10-08 01:52:13,237 — rl_replicas.vpg.vpg — INFO — Epoch: 4 2020-10-08 01:52:13,237 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 20.6 2020-10-08 01:52:13,238 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 10.1 2020-10-08 01:52:13,238 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 60.0 2020-10-08 01:52:13,238 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:52:13,238 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 20.6 2020-10-08 01:52:13,239 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 0.411 2020-10-08 01:52:13,239 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.038 2020-10-08 01:52:13,239 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 0.451 2020-10-08 01:52:13,239 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.19 2020-10-08 01:52:13,239 — rl_replicas.vpg.vpg — INFO — Total env interactions: 20000 2020-10-08 01:52:13,240 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.0126 2020-10-08 01:52:13,240 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 198.182 2020-10-08 01:52:13,240 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.68 2020-10-08 01:52:13,240 — rl_replicas.vpg.vpg — INFO — Time: 10.4 2020-10-08 01:52:15,155 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 19 steps on the current episode 2020-10-08 01:52:15,300 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: 0.00715 2020-10-08 01:52:15,301 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 248 2020-10-08 01:52:15,301 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00196 2020-10-08 01:52:15,301 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 40.8 2020-10-08 01:52:15,301 — rl_replicas.vpg.vpg — INFO — Epoch: 5 2020-10-08 01:52:15,301 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 21.4 2020-10-08 01:52:15,302 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 11.8 2020-10-08 01:52:15,302 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 85.0 2020-10-08 01:52:15,302 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 8.0 2020-10-08 01:52:15,302 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 21.4 2020-10-08 01:52:15,303 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 0.485 2020-10-08 01:52:15,303 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0354 2020-10-08 01:52:15,303 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 0.526 2020-10-08 01:52:15,303 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.297 2020-10-08 01:52:15,304 — rl_replicas.vpg.vpg — INFO — Total env interactions: 24000 2020-10-08 01:52:15,304 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.0117 2020-10-08 01:52:15,304 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 206.490 2020-10-08 01:52:15,304 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-08 01:52:15,304 — rl_replicas.vpg.vpg — INFO — Time: 12.5 2020-10-08 01:52:17,218 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 6 steps on the current episode 2020-10-08 01:52:17,366 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.000173 2020-10-08 01:52:17,367 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 243 2020-10-08 01:52:17,367 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00732 2020-10-08 01:52:17,367 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -5.52 2020-10-08 01:52:17,367 — rl_replicas.vpg.vpg — INFO — Epoch: 6 2020-10-08 01:52:17,368 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 21.5 2020-10-08 01:52:17,368 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 11.4 2020-10-08 01:52:17,368 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 103.0 2020-10-08 01:52:17,368 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 6.5881195068359375 2020-10-08 01:52:17,369 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 21.5 2020-10-08 01:52:17,369 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 0.552 2020-10-08 01:52:17,369 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0429 2020-10-08 01:52:17,369 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 0.602 2020-10-08 01:52:17,369 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.288 2020-10-08 01:52:17,370 — rl_replicas.vpg.vpg — INFO — Total env interactions: 28000 2020-10-08 01:52:17,370 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.01 2020-10-08 01:52:17,370 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 211.636 2020-10-08 01:52:17,370 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-08 01:52:17,370 — rl_replicas.vpg.vpg — INFO — Time: 14.6 2020-10-08 01:52:19,292 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 5 steps on the current episode 2020-10-08 01:52:19,439 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00221 2020-10-08 01:52:19,440 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 211 2020-10-08 01:52:19,440 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00204 2020-10-08 01:52:19,440 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -31.2 2020-10-08 01:52:19,440 — rl_replicas.vpg.vpg — INFO — Epoch: 7 2020-10-08 01:52:19,440 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 20.6 2020-10-08 01:52:19,441 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 10.5 2020-10-08 01:52:19,441 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 75.0 2020-10-08 01:52:19,441 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 5.665648579597473 2020-10-08 01:52:19,441 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 20.6 2020-10-08 01:52:19,442 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 0.626 2020-10-08 01:52:19,442 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0459 2020-10-08 01:52:19,442 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 0.676 2020-10-08 01:52:19,442 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.367 2020-10-08 01:52:19,443 — rl_replicas.vpg.vpg — INFO — Total env interactions: 32000 2020-10-08 01:52:19,443 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.00848 2020-10-08 01:52:19,443 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 211.595 2020-10-08 01:52:19,443 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.682 2020-10-08 01:52:19,443 — rl_replicas.vpg.vpg — INFO — Time: 16.6 2020-10-08 01:52:21,369 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 2 steps on the current episode 2020-10-08 01:52:21,518 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00324 2020-10-08 01:52:21,519 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 290 2020-10-08 01:52:21,519 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00102 2020-10-08 01:52:21,519 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 78.4 2020-10-08 01:52:21,519 — rl_replicas.vpg.vpg — INFO — Epoch: 8 2020-10-08 01:52:21,520 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 22.6 2020-10-08 01:52:21,520 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 13.4 2020-10-08 01:52:21,520 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 81.0 2020-10-08 01:52:21,520 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 2.7363826632499695 2020-10-08 01:52:21,521 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 22.6 2020-10-08 01:52:21,521 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 0.694 2020-10-08 01:52:21,521 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0519 2020-10-08 01:52:21,521 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 0.764 2020-10-08 01:52:21,522 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.386 2020-10-08 01:52:21,522 — rl_replicas.vpg.vpg — INFO — Total env interactions: 36000 2020-10-08 01:52:21,522 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.00718 2020-10-08 01:52:21,522 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 220.273 2020-10-08 01:52:21,522 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.683 2020-10-08 01:52:21,523 — rl_replicas.vpg.vpg — INFO — Time: 18.7 2020-10-08 01:52:23,417 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 20 steps on the current episode 2020-10-08 01:52:23,566 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00691 2020-10-08 01:52:23,567 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 310 2020-10-08 01:52:23,567 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00368 2020-10-08 01:52:23,567 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 20.4 2020-10-08 01:52:23,567 — rl_replicas.vpg.vpg — INFO — Epoch: 9 2020-10-08 01:52:23,568 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 25 2020-10-08 01:52:23,568 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 13.8 2020-10-08 01:52:23,568 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 77.0 2020-10-08 01:52:23,568 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:52:23,569 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 25 2020-10-08 01:52:23,569 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 0.771 2020-10-08 01:52:23,569 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.06 2020-10-08 01:52:23,569 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 0.849 2020-10-08 01:52:23,569 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.39 2020-10-08 01:52:23,570 — rl_replicas.vpg.vpg — INFO — Total env interactions: 40000 2020-10-08 01:52:23,570 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.00577 2020-10-08 01:52:23,570 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 229.254 2020-10-08 01:52:23,570 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.683 2020-10-08 01:52:23,570 — rl_replicas.vpg.vpg — INFO — Time: 20.8 2020-10-08 01:52:25,487 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 15 steps on the current episode 2020-10-08 01:52:25,635 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00761 2020-10-08 01:52:25,636 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 312 2020-10-08 01:52:25,636 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00069 2020-10-08 01:52:25,636 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 2.32 2020-10-08 01:52:25,636 — rl_replicas.vpg.vpg — INFO — Epoch: 10 2020-10-08 01:52:25,637 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 25.3 2020-10-08 01:52:25,637 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 13.9 2020-10-08 01:52:25,637 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 76.0 2020-10-08 01:52:25,637 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 8.0 2020-10-08 01:52:25,638 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 25.3 2020-10-08 01:52:25,638 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 0.838 2020-10-08 01:52:25,638 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0665 2020-10-08 01:52:25,638 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 0.927 2020-10-08 01:52:25,638 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.45 2020-10-08 01:52:25,639 — rl_replicas.vpg.vpg — INFO — Total env interactions: 44000 2020-10-08 01:52:25,639 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.00455 2020-10-08 01:52:25,639 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 236.812 2020-10-08 01:52:25,639 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.684 2020-10-08 01:52:25,639 — rl_replicas.vpg.vpg — INFO — Time: 22.8 2020-10-08 01:52:27,544 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 8 steps on the current episode 2020-10-08 01:52:27,695 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00865 2020-10-08 01:52:27,696 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 334 2020-10-08 01:52:27,696 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00105 2020-10-08 01:52:27,696 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 22 2020-10-08 01:52:27,696 — rl_replicas.vpg.vpg — INFO — Epoch: 11 2020-10-08 01:52:27,696 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 25.3 2020-10-08 01:52:27,697 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 14.7 2020-10-08 01:52:27,697 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 99.0 2020-10-08 01:52:27,697 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 8.95246708393097 2020-10-08 01:52:27,697 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 25.3 2020-10-08 01:52:27,698 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 0.92 2020-10-08 01:52:27,698 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0698 2020-10-08 01:52:27,698 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 1.01 2020-10-08 01:52:27,698 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.542 2020-10-08 01:52:27,698 — rl_replicas.vpg.vpg — INFO — Total env interactions: 48000 2020-10-08 01:52:27,699 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.00345 2020-10-08 01:52:27,699 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 244.945 2020-10-08 01:52:27,699 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.684 2020-10-08 01:52:27,699 — rl_replicas.vpg.vpg — INFO — Time: 24.9 2020-10-08 01:52:29,612 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 12 steps on the current episode 2020-10-08 01:52:29,760 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00653 2020-10-08 01:52:29,760 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 299 2020-10-08 01:52:29,761 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00212 2020-10-08 01:52:29,761 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -35.7 2020-10-08 01:52:29,761 — rl_replicas.vpg.vpg — INFO — Epoch: 12 2020-10-08 01:52:29,761 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 25.8 2020-10-08 01:52:29,762 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 13.3 2020-10-08 01:52:29,762 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 72.0 2020-10-08 01:52:29,762 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:52:29,762 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 25.8 2020-10-08 01:52:29,763 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 0.997 2020-10-08 01:52:29,763 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0752 2020-10-08 01:52:29,763 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 1.09 2020-10-08 01:52:29,763 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.556 2020-10-08 01:52:29,763 — rl_replicas.vpg.vpg — INFO — Total env interactions: 52000 2020-10-08 01:52:29,764 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.00269 2020-10-08 01:52:29,764 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 249.079 2020-10-08 01:52:29,764 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.685 2020-10-08 01:52:29,764 — rl_replicas.vpg.vpg — INFO — Time: 27 2020-10-08 01:52:31,696 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 18 steps on the current episode 2020-10-08 01:52:31,841 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0107 2020-10-08 01:52:31,841 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 346 2020-10-08 01:52:31,842 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.0042 2020-10-08 01:52:31,842 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 47 2020-10-08 01:52:31,842 — rl_replicas.vpg.vpg — INFO — Epoch: 13 2020-10-08 01:52:31,842 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 27.6 2020-10-08 01:52:31,843 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 14.8 2020-10-08 01:52:31,843 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 78.0 2020-10-08 01:52:31,843 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10.0 2020-10-08 01:52:31,843 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 27.6 2020-10-08 01:52:31,844 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 1.09 2020-10-08 01:52:31,844 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0772 2020-10-08 01:52:31,844 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 1.17 2020-10-08 01:52:31,844 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.583 2020-10-08 01:52:31,844 — rl_replicas.vpg.vpg — INFO — Total env interactions: 56000 2020-10-08 01:52:31,845 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.00173 2020-10-08 01:52:31,845 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 255.983 2020-10-08 01:52:31,845 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.685 2020-10-08 01:52:31,845 — rl_replicas.vpg.vpg — INFO — Time: 29 2020-10-08 01:52:33,765 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 10 steps on the current episode 2020-10-08 01:52:33,912 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00953 2020-10-08 01:52:33,913 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 384 2020-10-08 01:52:33,913 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.0012 2020-10-08 01:52:33,913 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 38.7 2020-10-08 01:52:33,913 — rl_replicas.vpg.vpg — INFO — Epoch: 14 2020-10-08 01:52:33,913 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 26.9 2020-10-08 01:52:33,914 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 16.6 2020-10-08 01:52:33,914 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 82.0 2020-10-08 01:52:33,914 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:52:33,914 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 26.8 2020-10-08 01:52:33,915 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 1.16 2020-10-08 01:52:33,915 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0861 2020-10-08 01:52:33,915 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 1.26 2020-10-08 01:52:33,915 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.646 2020-10-08 01:52:33,915 — rl_replicas.vpg.vpg — INFO — Total env interactions: 60000 2020-10-08 01:52:33,916 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.000976 2020-10-08 01:52:33,916 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 264.547 2020-10-08 01:52:33,916 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-08 01:52:33,916 — rl_replicas.vpg.vpg — INFO — Time: 31.1 2020-10-08 01:52:35,822 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 48 steps on the current episode 2020-10-08 01:52:35,968 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00999 2020-10-08 01:52:35,968 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 344 2020-10-08 01:52:35,969 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000453 2020-10-08 01:52:35,969 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -40.6 2020-10-08 01:52:35,969 — rl_replicas.vpg.vpg — INFO — Epoch: 15 2020-10-08 01:52:35,969 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 26.2 2020-10-08 01:52:35,970 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 15.3 2020-10-08 01:52:35,970 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 97.0 2020-10-08 01:52:35,970 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:52:35,970 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 26.1 2020-10-08 01:52:35,971 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 1.25 2020-10-08 01:52:35,971 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0911 2020-10-08 01:52:35,971 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 1.35 2020-10-08 01:52:35,971 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.786 2020-10-08 01:52:35,971 — rl_replicas.vpg.vpg — INFO — Total env interactions: 64000 2020-10-08 01:52:35,972 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: 0.000291 2020-10-08 01:52:35,972 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 269.505 2020-10-08 01:52:35,972 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-08 01:52:35,972 — rl_replicas.vpg.vpg — INFO — Time: 33.2 2020-10-08 01:52:37,931 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 2 steps on the current episode 2020-10-08 01:52:38,076 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0102 2020-10-08 01:52:38,077 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 295 2020-10-08 01:52:38,077 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000249 2020-10-08 01:52:38,077 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -48.6 2020-10-08 01:52:38,077 — rl_replicas.vpg.vpg — INFO — Epoch: 16 2020-10-08 01:52:38,078 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 27 2020-10-08 01:52:38,078 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 13 2020-10-08 01:52:38,078 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 82.0 2020-10-08 01:52:38,078 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 3.275852918624878 2020-10-08 01:52:38,079 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 27 2020-10-08 01:52:38,079 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 1.33 2020-10-08 01:52:38,079 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.088 2020-10-08 01:52:38,079 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 1.44 2020-10-08 01:52:38,079 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.797 2020-10-08 01:52:38,080 — rl_replicas.vpg.vpg — INFO — Total env interactions: 68000 2020-10-08 01:52:38,080 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.000328 2020-10-08 01:52:38,080 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 271.018 2020-10-08 01:52:38,080 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-08 01:52:38,080 — rl_replicas.vpg.vpg — INFO — Time: 35.3 2020-10-08 01:52:40,019 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 12 steps on the current episode 2020-10-08 01:52:40,167 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0112 2020-10-08 01:52:40,167 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 345 2020-10-08 01:52:40,168 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000978 2020-10-08 01:52:40,168 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 50.1 2020-10-08 01:52:40,168 — rl_replicas.vpg.vpg — INFO — Epoch: 17 2020-10-08 01:52:40,168 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 27.8 2020-10-08 01:52:40,169 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 15.1 2020-10-08 01:52:40,169 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 80.0 2020-10-08 01:52:40,169 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:52:40,169 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 27.8 2020-10-08 01:52:40,170 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 1.42 2020-10-08 01:52:40,170 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0934 2020-10-08 01:52:40,170 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 1.53 2020-10-08 01:52:40,170 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.826 2020-10-08 01:52:40,171 — rl_replicas.vpg.vpg — INFO — Total env interactions: 72000 2020-10-08 01:52:40,171 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.000933 2020-10-08 01:52:40,171 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 275.149 2020-10-08 01:52:40,171 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-08 01:52:40,171 — rl_replicas.vpg.vpg — INFO — Time: 37.4 2020-10-08 01:52:42,111 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 19 steps on the current episode 2020-10-08 01:52:42,260 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00934 2020-10-08 01:52:42,261 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 313 2020-10-08 01:52:42,261 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00187 2020-10-08 01:52:42,261 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -32.3 2020-10-08 01:52:42,261 — rl_replicas.vpg.vpg — INFO — Epoch: 18 2020-10-08 01:52:42,262 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 27.8 2020-10-08 01:52:42,262 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 13.8 2020-10-08 01:52:42,262 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 75.0 2020-10-08 01:52:42,262 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:52:42,263 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 27.8 2020-10-08 01:52:42,263 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 1.51 2020-10-08 01:52:42,263 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0968 2020-10-08 01:52:42,263 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 1.62 2020-10-08 01:52:42,264 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.955 2020-10-08 01:52:42,264 — rl_replicas.vpg.vpg — INFO — Total env interactions: 76000 2020-10-08 01:52:42,264 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00138 2020-10-08 01:52:42,264 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 277.147 2020-10-08 01:52:42,264 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.687 2020-10-08 01:52:42,265 — rl_replicas.vpg.vpg — INFO — Time: 39.5 2020-10-08 01:52:44,330 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0123 2020-10-08 01:52:44,331 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 413 2020-10-08 01:52:44,331 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00294 2020-10-08 01:52:44,331 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 99.8 2020-10-08 01:52:44,331 — rl_replicas.vpg.vpg — INFO — Epoch: 19 2020-10-08 01:52:44,331 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 30.5 2020-10-08 01:52:44,332 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 17.2 2020-10-08 01:52:44,332 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 110.0 2020-10-08 01:52:44,332 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10.0 2020-10-08 01:52:44,332 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 30.5 2020-10-08 01:52:44,333 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 1.6 2020-10-08 01:52:44,333 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.101 2020-10-08 01:52:44,333 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 1.72 2020-10-08 01:52:44,333 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.966 2020-10-08 01:52:44,334 — rl_replicas.vpg.vpg — INFO — Total env interactions: 80000 2020-10-08 01:52:44,334 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00192 2020-10-08 01:52:44,334 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 283.936 2020-10-08 01:52:44,334 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.685 2020-10-08 01:52:44,334 — rl_replicas.vpg.vpg — INFO — Time: 41.5 2020-10-08 01:52:46,254 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 3 steps on the current episode 2020-10-08 01:52:46,406 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0164 2020-10-08 01:52:46,406 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 386 2020-10-08 01:52:46,407 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.0041 2020-10-08 01:52:46,407 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -26.6 2020-10-08 01:52:46,407 — rl_replicas.vpg.vpg — INFO — Epoch: 20 2020-10-08 01:52:46,407 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 28.8 2020-10-08 01:52:46,408 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 16.7 2020-10-08 01:52:46,408 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 97.0 2020-10-08 01:52:46,408 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 4.712063193321228 2020-10-08 01:52:46,408 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 28.8 2020-10-08 01:52:46,409 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 1.69 2020-10-08 01:52:46,409 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.107 2020-10-08 01:52:46,409 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 1.81 2020-10-08 01:52:46,409 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.05 2020-10-08 01:52:46,410 — rl_replicas.vpg.vpg — INFO — Total env interactions: 84000 2020-10-08 01:52:46,410 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00261 2020-10-08 01:52:46,410 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 288.811 2020-10-08 01:52:46,410 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.685 2020-10-08 01:52:46,410 — rl_replicas.vpg.vpg — INFO — Time: 43.6 2020-10-08 01:52:48,370 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 46 steps on the current episode 2020-10-08 01:52:48,516 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.017 2020-10-08 01:52:48,517 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 452 2020-10-08 01:52:48,517 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00066 2020-10-08 01:52:48,517 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 65.6 2020-10-08 01:52:48,517 — rl_replicas.vpg.vpg — INFO — Epoch: 21 2020-10-08 01:52:48,518 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 31.3 2020-10-08 01:52:48,518 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 18.8 2020-10-08 01:52:48,518 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 103.0 2020-10-08 01:52:48,518 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:52:48,519 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 31.2 2020-10-08 01:52:48,519 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 1.8 2020-10-08 01:52:48,519 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0972 2020-10-08 01:52:48,519 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 1.91 2020-10-08 01:52:48,519 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.16 2020-10-08 01:52:48,520 — rl_replicas.vpg.vpg — INFO — Total env interactions: 88000 2020-10-08 01:52:48,520 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00327 2020-10-08 01:52:48,520 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 296.222 2020-10-08 01:52:48,520 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.685 2020-10-08 01:52:48,521 — rl_replicas.vpg.vpg — INFO — Time: 45.7 2020-10-08 01:52:50,449 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 45 steps on the current episode 2020-10-08 01:52:50,597 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.016 2020-10-08 01:52:50,597 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 360 2020-10-08 01:52:50,597 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00106 2020-10-08 01:52:50,597 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -92.2 2020-10-08 01:52:50,598 — rl_replicas.vpg.vpg — INFO — Epoch: 22 2020-10-08 01:52:50,598 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 29.6 2020-10-08 01:52:50,598 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 15.7 2020-10-08 01:52:50,598 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 84.0 2020-10-08 01:52:50,599 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:52:50,599 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 29.6 2020-10-08 01:52:50,600 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 1.9 2020-10-08 01:52:50,600 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.101 2020-10-08 01:52:50,600 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 2 2020-10-08 01:52:50,601 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.29 2020-10-08 01:52:50,601 — rl_replicas.vpg.vpg — INFO — Total env interactions: 92000 2020-10-08 01:52:50,601 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00382 2020-10-08 01:52:50,601 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 298.980 2020-10-08 01:52:50,601 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.685 2020-10-08 01:52:50,602 — rl_replicas.vpg.vpg — INFO — Time: 47.8 2020-10-08 01:52:52,525 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 16 steps on the current episode 2020-10-08 01:52:52,671 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.019 2020-10-08 01:52:52,672 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 331 2020-10-08 01:52:52,672 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00299 2020-10-08 01:52:52,672 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -28.9 2020-10-08 01:52:52,672 — rl_replicas.vpg.vpg — INFO — Epoch: 23 2020-10-08 01:52:52,673 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 27.8 2020-10-08 01:52:52,673 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 15.1 2020-10-08 01:52:52,673 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 90.0 2020-10-08 01:52:52,673 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:52:52,674 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 27.8 2020-10-08 01:52:52,674 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 1.99 2020-10-08 01:52:52,674 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.104 2020-10-08 01:52:52,674 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 2.12 2020-10-08 01:52:52,674 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.23 2020-10-08 01:52:52,674 — rl_replicas.vpg.vpg — INFO — Total env interactions: 96000 2020-10-08 01:52:52,675 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00445 2020-10-08 01:52:52,675 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 300.302 2020-10-08 01:52:52,675 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.685 2020-10-08 01:52:52,675 — rl_replicas.vpg.vpg — INFO — Time: 49.9 2020-10-08 01:52:54,599 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 13 steps on the current episode 2020-10-08 01:52:54,744 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0138 2020-10-08 01:52:54,744 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 415 2020-10-08 01:52:54,745 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00513 2020-10-08 01:52:54,745 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 84.2 2020-10-08 01:52:54,745 — rl_replicas.vpg.vpg — INFO — Epoch: 24 2020-10-08 01:52:54,745 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 29.6 2020-10-08 01:52:54,745 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 18.1 2020-10-08 01:52:54,746 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 112.0 2020-10-08 01:52:54,746 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10.0 2020-10-08 01:52:54,746 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 29.6 2020-10-08 01:52:54,746 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 2.09 2020-10-08 01:52:54,747 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.112 2020-10-08 01:52:54,747 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 2.22 2020-10-08 01:52:54,747 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.42 2020-10-08 01:52:54,747 — rl_replicas.vpg.vpg — INFO — Total env interactions: 100000 2020-10-08 01:52:54,747 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00483 2020-10-08 01:52:54,747 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 304.888 2020-10-08 01:52:54,748 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-08 01:52:54,748 — rl_replicas.vpg.vpg — INFO — Time: 51.9 2020-10-08 01:52:56,812 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0173 2020-10-08 01:52:56,813 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 444 2020-10-08 01:52:56,813 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00343 2020-10-08 01:52:56,813 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 28.7 2020-10-08 01:52:56,813 — rl_replicas.vpg.vpg — INFO — Epoch: 25 2020-10-08 01:52:56,813 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 32 2020-10-08 01:52:56,814 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 18.7 2020-10-08 01:52:56,814 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 113.0 2020-10-08 01:52:56,814 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:52:56,814 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 32 2020-10-08 01:52:56,814 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 2.2 2020-10-08 01:52:56,815 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.104 2020-10-08 01:52:56,815 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 2.33 2020-10-08 01:52:56,815 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.56 2020-10-08 01:52:56,815 — rl_replicas.vpg.vpg — INFO — Total env interactions: 104000 2020-10-08 01:52:56,815 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0053 2020-10-08 01:52:56,816 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 310.223 2020-10-08 01:52:56,816 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.685 2020-10-08 01:52:56,816 — rl_replicas.vpg.vpg — INFO — Time: 54 2020-10-08 01:52:58,733 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 48 steps on the current episode 2020-10-08 01:52:58,888 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0142 2020-10-08 01:52:58,888 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 548 2020-10-08 01:52:58,889 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00303 2020-10-08 01:52:58,889 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 105 2020-10-08 01:52:58,889 — rl_replicas.vpg.vpg — INFO — Epoch: 26 2020-10-08 01:52:58,889 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 34.8 2020-10-08 01:52:58,889 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 22 2020-10-08 01:52:58,890 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 145.0 2020-10-08 01:52:58,890 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10.0 2020-10-08 01:52:58,890 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 34.8 2020-10-08 01:52:58,890 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 2.32 2020-10-08 01:52:58,890 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0996 2020-10-08 01:52:58,891 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 2.44 2020-10-08 01:52:58,891 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.72 2020-10-08 01:52:58,891 — rl_replicas.vpg.vpg — INFO — Total env interactions: 108000 2020-10-08 01:52:58,891 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00564 2020-10-08 01:52:58,891 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 319.044 2020-10-08 01:52:58,892 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.684 2020-10-08 01:52:58,892 — rl_replicas.vpg.vpg — INFO — Time: 56.1 2020-10-08 01:53:00,783 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 3 steps on the current episode 2020-10-08 01:53:00,932 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0191 2020-10-08 01:53:00,932 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 470 2020-10-08 01:53:00,932 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00486 2020-10-08 01:53:00,933 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -78.1 2020-10-08 01:53:00,933 — rl_replicas.vpg.vpg — INFO — Epoch: 27 2020-10-08 01:53:00,933 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 31.8 2020-10-08 01:53:00,933 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 20.1 2020-10-08 01:53:00,934 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 117.0 2020-10-08 01:53:00,934 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 5.400299072265625 2020-10-08 01:53:00,934 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 31.7 2020-10-08 01:53:00,934 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 2.42 2020-10-08 01:53:00,934 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.104 2020-10-08 01:53:00,935 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 2.56 2020-10-08 01:53:00,935 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.76 2020-10-08 01:53:00,935 — rl_replicas.vpg.vpg — INFO — Total env interactions: 112000 2020-10-08 01:53:00,935 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00612 2020-10-08 01:53:00,935 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 324.447 2020-10-08 01:53:00,936 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.683 2020-10-08 01:53:00,936 — rl_replicas.vpg.vpg — INFO — Time: 58.1 2020-10-08 01:53:02,876 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 42 steps on the current episode 2020-10-08 01:53:03,025 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0132 2020-10-08 01:53:03,026 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 352 2020-10-08 01:53:03,026 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.0059 2020-10-08 01:53:03,026 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -119 2020-10-08 01:53:03,026 — rl_replicas.vpg.vpg — INFO — Epoch: 28 2020-10-08 01:53:03,026 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 30.6 2020-10-08 01:53:03,027 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 15.8 2020-10-08 01:53:03,027 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 88.0 2020-10-08 01:53:03,027 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12.0 2020-10-08 01:53:03,027 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 30.5 2020-10-08 01:53:03,028 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 2.55 2020-10-08 01:53:03,028 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0938 2020-10-08 01:53:03,028 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 2.66 2020-10-08 01:53:03,028 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.98 2020-10-08 01:53:03,028 — rl_replicas.vpg.vpg — INFO — Total env interactions: 116000 2020-10-08 01:53:03,029 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00636 2020-10-08 01:53:03,029 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 325.386 2020-10-08 01:53:03,029 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.683 2020-10-08 01:53:03,029 — rl_replicas.vpg.vpg — INFO — Time: 60.2 2020-10-08 01:53:04,938 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 13 steps on the current episode 2020-10-08 01:53:05,088 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0165 2020-10-08 01:53:05,088 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 531 2020-10-08 01:53:05,088 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00334 2020-10-08 01:53:05,089 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 179 2020-10-08 01:53:05,089 — rl_replicas.vpg.vpg — INFO — Epoch: 29 2020-10-08 01:53:05,089 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 36.4 2020-10-08 01:53:05,089 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 21.3 2020-10-08 01:53:05,089 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 103.0 2020-10-08 01:53:05,090 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11.0 2020-10-08 01:53:05,090 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 36.4 2020-10-08 01:53:05,090 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 2.66 2020-10-08 01:53:05,090 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.104 2020-10-08 01:53:05,090 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 2.78 2020-10-08 01:53:05,091 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 2.08 2020-10-08 01:53:05,091 — rl_replicas.vpg.vpg — INFO — Total env interactions: 120000 2020-10-08 01:53:05,091 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0067 2020-10-08 01:53:05,091 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 332.238 2020-10-08 01:53:05,091 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.683 2020-10-08 01:53:05,092 — rl_replicas.vpg.vpg — INFO — Time: 62.3 2020-10-08 01:53:06,997 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 9 steps on the current episode 2020-10-08 01:53:07,144 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0133 2020-10-08 01:53:07,145 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 470 2020-10-08 01:53:07,145 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00324 2020-10-08 01:53:07,145 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -60.5 2020-10-08 01:53:07,145 — rl_replicas.vpg.vpg — INFO — Epoch: 30 2020-10-08 01:53:07,146 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 35.4 2020-10-08 01:53:07,146 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 19.4 2020-10-08 01:53:07,146 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 123.0 2020-10-08 01:53:07,146 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11.0 2020-10-08 01:53:07,147 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 35.4 2020-10-08 01:53:07,147 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 2.78 2020-10-08 01:53:07,147 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0956 2020-10-08 01:53:07,147 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 2.9 2020-10-08 01:53:07,147 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 2.15 2020-10-08 01:53:07,148 — rl_replicas.vpg.vpg — INFO — Total env interactions: 124000 2020-10-08 01:53:07,148 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00691 2020-10-08 01:53:07,148 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 336.698 2020-10-08 01:53:07,148 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.682 2020-10-08 01:53:07,148 — rl_replicas.vpg.vpg — INFO — Time: 64.3 2020-10-08 01:53:09,036 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 13 steps on the current episode 2020-10-08 01:53:09,190 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0109 2020-10-08 01:53:09,191 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 491 2020-10-08 01:53:09,191 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00235 2020-10-08 01:53:09,191 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 20.4 2020-10-08 01:53:09,191 — rl_replicas.vpg.vpg — INFO — Epoch: 31 2020-10-08 01:53:09,191 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 37.1 2020-10-08 01:53:09,192 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 19.7 2020-10-08 01:53:09,192 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 99.0 2020-10-08 01:53:09,192 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11.0 2020-10-08 01:53:09,192 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 37 2020-10-08 01:53:09,193 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 2.89 2020-10-08 01:53:09,193 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.113 2020-10-08 01:53:09,193 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 3.01 2020-10-08 01:53:09,193 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 2.1 2020-10-08 01:53:09,193 — rl_replicas.vpg.vpg — INFO — Total env interactions: 128000 2020-10-08 01:53:09,194 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00704 2020-10-08 01:53:09,194 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 341.517 2020-10-08 01:53:09,194 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-08 01:53:09,194 — rl_replicas.vpg.vpg — INFO — Time: 66.4 2020-10-08 01:53:11,081 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 87 steps on the current episode 2020-10-08 01:53:11,229 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0185 2020-10-08 01:53:11,229 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 547 2020-10-08 01:53:11,230 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00752 2020-10-08 01:53:11,230 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 56 2020-10-08 01:53:11,230 — rl_replicas.vpg.vpg — INFO — Epoch: 32 2020-10-08 01:53:11,230 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 35.7 2020-10-08 01:53:11,231 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 22.6 2020-10-08 01:53:11,231 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 120.0 2020-10-08 01:53:11,231 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:53:11,231 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 35.7 2020-10-08 01:53:11,232 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 3.03 2020-10-08 01:53:11,232 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0888 2020-10-08 01:53:11,232 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 3.15 2020-10-08 01:53:11,232 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 2.25 2020-10-08 01:53:11,232 — rl_replicas.vpg.vpg — INFO — Total env interactions: 132000 2020-10-08 01:53:11,233 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00738 2020-10-08 01:53:11,233 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 347.741 2020-10-08 01:53:11,233 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-08 01:53:11,233 — rl_replicas.vpg.vpg — INFO — Time: 68.4 2020-10-08 01:53:13,148 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 38 steps on the current episode 2020-10-08 01:53:13,297 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0116 2020-10-08 01:53:13,297 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 452 2020-10-08 01:53:13,297 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00686 2020-10-08 01:53:13,297 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -94.7 2020-10-08 01:53:13,298 — rl_replicas.vpg.vpg — INFO — Epoch: 33 2020-10-08 01:53:13,298 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 36.1 2020-10-08 01:53:13,298 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 18.9 2020-10-08 01:53:13,298 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 93.0 2020-10-08 01:53:13,298 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11.0 2020-10-08 01:53:13,299 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 36 2020-10-08 01:53:13,299 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 3.15 2020-10-08 01:53:13,299 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0945 2020-10-08 01:53:13,299 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 3.26 2020-10-08 01:53:13,299 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 2.49 2020-10-08 01:53:13,300 — rl_replicas.vpg.vpg — INFO — Total env interactions: 136000 2020-10-08 01:53:13,300 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00751 2020-10-08 01:53:13,300 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 350.814 2020-10-08 01:53:13,300 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.682 2020-10-08 01:53:13,300 — rl_replicas.vpg.vpg — INFO — Time: 70.5 2020-10-08 01:53:15,213 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 29 steps on the current episode 2020-10-08 01:53:15,361 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0143 2020-10-08 01:53:15,361 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 404 2020-10-08 01:53:15,361 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00275 2020-10-08 01:53:15,361 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -48 2020-10-08 01:53:15,362 — rl_replicas.vpg.vpg — INFO — Epoch: 34 2020-10-08 01:53:15,362 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 35.7 2020-10-08 01:53:15,362 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 16.9 2020-10-08 01:53:15,362 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 89.0 2020-10-08 01:53:15,362 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10.0 2020-10-08 01:53:15,363 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 35.7 2020-10-08 01:53:15,363 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 3.27 2020-10-08 01:53:15,363 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0997 2020-10-08 01:53:15,363 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 3.4 2020-10-08 01:53:15,363 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 2.53 2020-10-08 01:53:15,364 — rl_replicas.vpg.vpg — INFO — Total env interactions: 140000 2020-10-08 01:53:15,364 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0077 2020-10-08 01:53:15,364 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 352.340 2020-10-08 01:53:15,364 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.682 2020-10-08 01:53:15,364 — rl_replicas.vpg.vpg — INFO — Time: 72.6 2020-10-08 01:53:17,275 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 7 steps on the current episode 2020-10-08 01:53:17,421 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0117 2020-10-08 01:53:17,422 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 726 2020-10-08 01:53:17,422 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00269 2020-10-08 01:53:17,422 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 322 2020-10-08 01:53:17,422 — rl_replicas.vpg.vpg — INFO — Epoch: 35 2020-10-08 01:53:17,423 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 38.5 2020-10-08 01:53:17,423 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 28.9 2020-10-08 01:53:17,423 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 200.0 2020-10-08 01:53:17,423 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10.0 2020-10-08 01:53:17,423 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 38.5 2020-10-08 01:53:17,424 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 3.4 2020-10-08 01:53:17,424 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0995 2020-10-08 01:53:17,424 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 3.53 2020-10-08 01:53:17,424 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 2.74 2020-10-08 01:53:17,424 — rl_replicas.vpg.vpg — INFO — Total env interactions: 144000 2020-10-08 01:53:17,425 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00781 2020-10-08 01:53:17,425 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 362.725 2020-10-08 01:53:17,425 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.682 2020-10-08 01:53:17,425 — rl_replicas.vpg.vpg — INFO — Time: 74.6 2020-10-08 01:53:19,332 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 30 steps on the current episode 2020-10-08 01:53:19,479 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0114 2020-10-08 01:53:19,479 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 697 2020-10-08 01:53:19,480 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.000256 2020-10-08 01:53:19,480 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -29.1 2020-10-08 01:53:19,480 — rl_replicas.vpg.vpg — INFO — Epoch: 36 2020-10-08 01:53:19,480 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 43.5 2020-10-08 01:53:19,480 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 26.4 2020-10-08 01:53:19,481 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 165.0 2020-10-08 01:53:19,481 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 14.0 2020-10-08 01:53:19,481 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 43.5 2020-10-08 01:53:19,481 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 3.52 2020-10-08 01:53:19,481 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.123 2020-10-08 01:53:19,482 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 3.66 2020-10-08 01:53:19,482 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 2.63 2020-10-08 01:53:19,482 — rl_replicas.vpg.vpg — INFO — Total env interactions: 148000 2020-10-08 01:53:19,482 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00791 2020-10-08 01:53:19,482 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 371.762 2020-10-08 01:53:19,483 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.682 2020-10-08 01:53:19,483 — rl_replicas.vpg.vpg — INFO — Time: 76.7 2020-10-08 01:53:21,392 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 20 steps on the current episode 2020-10-08 01:53:21,537 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0118 2020-10-08 01:53:21,537 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 644 2020-10-08 01:53:21,537 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000421 2020-10-08 01:53:21,538 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -52.6 2020-10-08 01:53:21,538 — rl_replicas.vpg.vpg — INFO — Epoch: 37 2020-10-08 01:53:21,538 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 39.6 2020-10-08 01:53:21,538 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 26 2020-10-08 01:53:21,538 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 193.0 2020-10-08 01:53:21,539 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:53:21,539 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 39.6 2020-10-08 01:53:21,539 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 3.64 2020-10-08 01:53:21,539 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.128 2020-10-08 01:53:21,539 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 3.79 2020-10-08 01:53:21,540 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 2.66 2020-10-08 01:53:21,540 — rl_replicas.vpg.vpg — INFO — Total env interactions: 152000 2020-10-08 01:53:21,540 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00801 2020-10-08 01:53:21,540 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 378.937 2020-10-08 01:53:21,540 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-08 01:53:21,541 — rl_replicas.vpg.vpg — INFO — Time: 78.7 2020-10-08 01:53:23,432 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 37 steps on the current episode 2020-10-08 01:53:23,578 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00957 2020-10-08 01:53:23,578 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 833 2020-10-08 01:53:23,579 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00225 2020-10-08 01:53:23,579 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 189 2020-10-08 01:53:23,579 — rl_replicas.vpg.vpg — INFO — Epoch: 38 2020-10-08 01:53:23,579 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 48.2 2020-10-08 01:53:23,579 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 30.2 2020-10-08 01:53:23,580 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 156.0 2020-10-08 01:53:23,580 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10.0 2020-10-08 01:53:23,580 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 48.2 2020-10-08 01:53:23,580 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 3.77 2020-10-08 01:53:23,581 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.146 2020-10-08 01:53:23,581 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 3.93 2020-10-08 01:53:23,581 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 2.94 2020-10-08 01:53:23,581 — rl_replicas.vpg.vpg — INFO — Total env interactions: 156000 2020-10-08 01:53:23,581 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00805 2020-10-08 01:53:23,581 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 390.580 2020-10-08 01:53:23,582 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-08 01:53:23,582 — rl_replicas.vpg.vpg — INFO — Time: 80.8 2020-10-08 01:53:25,464 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 20 steps on the current episode 2020-10-08 01:53:25,611 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0165 2020-10-08 01:53:25,611 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 586 2020-10-08 01:53:25,611 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00694 2020-10-08 01:53:25,612 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -247 2020-10-08 01:53:25,612 — rl_replicas.vpg.vpg — INFO — Epoch: 39 2020-10-08 01:53:25,612 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 42.6 2020-10-08 01:53:25,612 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 22.9 2020-10-08 01:53:25,612 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 133.0 2020-10-08 01:53:25,613 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12.0 2020-10-08 01:53:25,613 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 42.6 2020-10-08 01:53:25,613 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 3.92 2020-10-08 01:53:25,613 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.128 2020-10-08 01:53:25,614 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 4.06 2020-10-08 01:53:25,614 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 2.91 2020-10-08 01:53:25,614 — rl_replicas.vpg.vpg — INFO — Total env interactions: 160000 2020-10-08 01:53:25,614 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00827 2020-10-08 01:53:25,614 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 395.455 2020-10-08 01:53:25,615 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.679 2020-10-08 01:53:25,615 — rl_replicas.vpg.vpg — INFO — Time: 82.8 2020-10-08 01:53:27,541 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 16 steps on the current episode 2020-10-08 01:53:27,691 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00989 2020-10-08 01:53:27,692 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 515 2020-10-08 01:53:27,692 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00662 2020-10-08 01:53:27,692 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -70.5 2020-10-08 01:53:27,692 — rl_replicas.vpg.vpg — INFO — Epoch: 40 2020-10-08 01:53:27,693 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 40.9 2020-10-08 01:53:27,693 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 20.7 2020-10-08 01:53:27,693 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 103.0 2020-10-08 01:53:27,693 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12.0 2020-10-08 01:53:27,694 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 40.8 2020-10-08 01:53:27,694 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 4.05 2020-10-08 01:53:27,694 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.139 2020-10-08 01:53:27,694 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 4.18 2020-10-08 01:53:27,694 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 3.02 2020-10-08 01:53:27,695 — rl_replicas.vpg.vpg — INFO — Total env interactions: 164000 2020-10-08 01:53:27,695 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00831 2020-10-08 01:53:27,695 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 398.372 2020-10-08 01:53:27,695 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.678 2020-10-08 01:53:27,695 — rl_replicas.vpg.vpg — INFO — Time: 84.9 2020-10-08 01:53:29,602 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 32 steps on the current episode 2020-10-08 01:53:29,748 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0163 2020-10-08 01:53:29,748 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 597 2020-10-08 01:53:29,748 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00644 2020-10-08 01:53:29,749 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 81.5 2020-10-08 01:53:29,749 — rl_replicas.vpg.vpg — INFO — Epoch: 41 2020-10-08 01:53:29,749 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 41.7 2020-10-08 01:53:29,749 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 24.1 2020-10-08 01:53:29,749 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 152.0 2020-10-08 01:53:29,750 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10.0 2020-10-08 01:53:29,750 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 41.7 2020-10-08 01:53:29,750 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 4.19 2020-10-08 01:53:29,750 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.117 2020-10-08 01:53:29,750 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 4.34 2020-10-08 01:53:29,751 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 3.33 2020-10-08 01:53:29,751 — rl_replicas.vpg.vpg — INFO — Total env interactions: 168000 2020-10-08 01:53:29,751 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0085 2020-10-08 01:53:29,751 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 403.092 2020-10-08 01:53:29,751 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.678 2020-10-08 01:53:29,752 — rl_replicas.vpg.vpg — INFO — Time: 87 2020-10-08 01:53:31,655 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 9 steps on the current episode 2020-10-08 01:53:31,803 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0112 2020-10-08 01:53:31,804 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 554 2020-10-08 01:53:31,804 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.0051 2020-10-08 01:53:31,805 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -42.1 2020-10-08 01:53:31,805 — rl_replicas.vpg.vpg — INFO — Epoch: 42 2020-10-08 01:53:31,805 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 40 2020-10-08 01:53:31,806 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 23.2 2020-10-08 01:53:31,806 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 127.0 2020-10-08 01:53:31,806 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9.0 2020-10-08 01:53:31,806 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 40 2020-10-08 01:53:31,806 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 4.32 2020-10-08 01:53:31,807 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.143 2020-10-08 01:53:31,807 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 4.46 2020-10-08 01:53:31,807 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 3.01 2020-10-08 01:53:31,807 — rl_replicas.vpg.vpg — INFO — Total env interactions: 172000 2020-10-08 01:53:31,807 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00856 2020-10-08 01:53:31,808 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 406.612 2020-10-08 01:53:31,808 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.675 2020-10-08 01:53:31,808 — rl_replicas.vpg.vpg — INFO — Time: 89 2020-10-08 01:53:33,708 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 105 steps on the current episode 2020-10-08 01:53:33,855 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00942 2020-10-08 01:53:33,855 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 854 2020-10-08 01:53:33,856 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00182 2020-10-08 01:53:33,856 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 300 2020-10-08 01:53:33,856 — rl_replicas.vpg.vpg — INFO — Epoch: 43 2020-10-08 01:53:33,856 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 47.1 2020-10-08 01:53:33,856 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 32.6 2020-10-08 01:53:33,857 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 198.0 2020-10-08 01:53:33,857 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 14.0 2020-10-08 01:53:33,857 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 47.1 2020-10-08 01:53:33,857 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 4.46 2020-10-08 01:53:33,857 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.135 2020-10-08 01:53:33,858 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 4.6 2020-10-08 01:53:33,858 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 3.42 2020-10-08 01:53:33,858 — rl_replicas.vpg.vpg — INFO — Total env interactions: 176000 2020-10-08 01:53:33,858 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00858 2020-10-08 01:53:33,858 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 416.780 2020-10-08 01:53:33,859 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.673 2020-10-08 01:53:33,859 — rl_replicas.vpg.vpg — INFO — Time: 91.1 2020-10-08 01:53:35,760 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 11 steps on the current episode 2020-10-08 01:53:35,907 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0155 2020-10-08 01:53:35,907 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 669 2020-10-08 01:53:35,907 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00605 2020-10-08 01:53:35,908 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -185 2020-10-08 01:53:35,908 — rl_replicas.vpg.vpg — INFO — Epoch: 44 2020-10-08 01:53:35,908 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 47.1 2020-10-08 01:53:35,908 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 25.7 2020-10-08 01:53:35,908 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 118.0 2020-10-08 01:53:35,909 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10.0 2020-10-08 01:53:35,909 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 47.1 2020-10-08 01:53:35,909 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 4.6 2020-10-08 01:53:35,909 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.159 2020-10-08 01:53:35,910 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 4.73 2020-10-08 01:53:35,910 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 3.05 2020-10-08 01:53:35,910 — rl_replicas.vpg.vpg — INFO — Total env interactions: 180000 2020-10-08 01:53:35,910 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00873 2020-10-08 01:53:35,910 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 422.379 2020-10-08 01:53:35,911 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.672 2020-10-08 01:53:35,911 — rl_replicas.vpg.vpg — INFO — Time: 93.1 2020-10-08 01:53:37,814 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 72 steps on the current episode 2020-10-08 01:53:37,959 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0106 2020-10-08 01:53:37,959 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 632 2020-10-08 01:53:37,960 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00488 2020-10-08 01:53:37,960 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -36.4 2020-10-08 01:53:37,960 — rl_replicas.vpg.vpg — INFO — Epoch: 45 2020-10-08 01:53:37,960 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 46 2020-10-08 01:53:37,961 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 24.8 2020-10-08 01:53:37,961 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 126.0 2020-10-08 01:53:37,961 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12.0 2020-10-08 01:53:37,961 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 46 2020-10-08 01:53:37,961 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 4.73 2020-10-08 01:53:37,962 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.123 2020-10-08 01:53:37,962 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 4.87 2020-10-08 01:53:37,962 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 3.99 2020-10-08 01:53:37,962 — rl_replicas.vpg.vpg — INFO — Total env interactions: 184000 2020-10-08 01:53:37,962 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00877 2020-10-08 01:53:37,963 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 426.943 2020-10-08 01:53:37,963 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.672 2020-10-08 01:53:37,963 — rl_replicas.vpg.vpg — INFO — Time: 95.2 2020-10-08 01:53:39,873 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 9 steps on the current episode 2020-10-08 01:53:40,021 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0167 2020-10-08 01:53:40,021 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 786 2020-10-08 01:53:40,022 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00614 2020-10-08 01:53:40,022 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 153 2020-10-08 01:53:40,022 — rl_replicas.vpg.vpg — INFO — Epoch: 46 2020-10-08 01:53:40,022 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 45.5 2020-10-08 01:53:40,022 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 31.2 2020-10-08 01:53:40,023 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 165.0 2020-10-08 01:53:40,023 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 13.0 2020-10-08 01:53:40,023 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 45.5 2020-10-08 01:53:40,023 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 4.88 2020-10-08 01:53:40,023 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.129 2020-10-08 01:53:40,024 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 5.01 2020-10-08 01:53:40,024 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 3.71 2020-10-08 01:53:40,024 — rl_replicas.vpg.vpg — INFO — Total env interactions: 188000 2020-10-08 01:53:40,024 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00894 2020-10-08 01:53:40,024 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 434.578 2020-10-08 01:53:40,025 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.672 2020-10-08 01:53:40,025 — rl_replicas.vpg.vpg — INFO — Time: 97.2 2020-10-08 01:53:41,935 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 10 steps on the current episode 2020-10-08 01:53:42,082 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00932 2020-10-08 01:53:42,082 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 668 2020-10-08 01:53:42,082 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.0074 2020-10-08 01:53:42,083 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -118 2020-10-08 01:53:42,083 — rl_replicas.vpg.vpg — INFO — Epoch: 47 2020-10-08 01:53:42,084 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 46.6 2020-10-08 01:53:42,084 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 26.5 2020-10-08 01:53:42,084 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 136.0 2020-10-08 01:53:42,084 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12.0 2020-10-08 01:53:42,085 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 46.5 2020-10-08 01:53:42,085 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 5.01 2020-10-08 01:53:42,085 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.114 2020-10-08 01:53:42,085 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 5.14 2020-10-08 01:53:42,085 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 4.15 2020-10-08 01:53:42,086 — rl_replicas.vpg.vpg — INFO — Total env interactions: 192000 2020-10-08 01:53:42,086 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00895 2020-10-08 01:53:42,086 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 439.438 2020-10-08 01:53:42,086 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.672 2020-10-08 01:53:42,086 — rl_replicas.vpg.vpg — INFO — Time: 99.3 2020-10-08 01:53:43,991 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 24 steps on the current episode 2020-10-08 01:53:44,137 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.014 2020-10-08 01:53:44,137 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 662 2020-10-08 01:53:44,137 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00464 2020-10-08 01:53:44,137 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -5.58 2020-10-08 01:53:44,138 — rl_replicas.vpg.vpg — INFO — Epoch: 48 2020-10-08 01:53:44,138 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 45 2020-10-08 01:53:44,138 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 27.3 2020-10-08 01:53:44,138 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 196.0 2020-10-08 01:53:44,138 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11.0 2020-10-08 01:53:44,139 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 44.9 2020-10-08 01:53:44,139 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 5.15 2020-10-08 01:53:44,139 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.121 2020-10-08 01:53:44,139 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 5.28 2020-10-08 01:53:44,139 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 4.22 2020-10-08 01:53:44,140 — rl_replicas.vpg.vpg — INFO — Total env interactions: 196000 2020-10-08 01:53:44,140 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00905 2020-10-08 01:53:44,140 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 443.986 2020-10-08 01:53:44,140 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.672 2020-10-08 01:53:44,140 — rl_replicas.vpg.vpg — INFO — Time: 101 2020-10-08 01:53:46,022 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 17 steps on the current episode 2020-10-08 01:53:46,023 — rl_replicas.vpg.vpg — WARNING — Saving model is not implemented 2020-10-08 01:53:46,169 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0128 2020-10-08 01:53:46,170 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 823 2020-10-08 01:53:46,170 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00113 2020-10-08 01:53:46,170 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 161 2020-10-08 01:53:46,170 — rl_replicas.vpg.vpg — INFO — Epoch: 49 2020-10-08 01:53:46,171 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 51.4 2020-10-08 01:53:46,171 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 31.3 2020-10-08 01:53:46,171 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 161.0 2020-10-08 01:53:46,171 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 16.0 2020-10-08 01:53:46,172 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 51.3 2020-10-08 01:53:46,173 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 5.29 2020-10-08 01:53:46,173 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.13 2020-10-08 01:53:46,173 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 5.42 2020-10-08 01:53:46,174 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 4.03 2020-10-08 01:53:46,174 — rl_replicas.vpg.vpg — INFO — Total env interactions: 200000 2020-10-08 01:53:46,174 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00913 2020-10-08 01:53:46,174 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 451.574 2020-10-08 01:53:46,175 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.672 2020-10-08 01:53:46,175 — rl_replicas.vpg.vpg — INFO — Time: 103 73010951 function calls (70179832 primitive calls) in 104.567 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 1200150 10.264 0.000 10.264 0.000 {method 'matmul' of 'torch._C._TensorBase' objects} 3201200/400150 9.945 0.000 62.774 0.000 module.py:710(_call_impl) 200050 8.315 0.000 8.315 0.000 {method 'logsumexp' of 'torch._C._TensorBase' objects} 1200450 7.935 0.000 26.568 0.000 functional.py:1655(linear) 100 4.794 0.048 4.794 0.048 {method 'run_backward' of 'torch._C._EngineBase' objects} 1 4.559 4.559 103.376 103.376 vpg.py:60(learn) 800300 4.069 0.000 4.069 0.000 {built-in method tanh} 1200450 3.613 0.000 3.613 0.000 {method 't' of 'torch._C._TensorBase' objects} 200000 3.405 0.000 15.860 0.000 cartpole.py:102(step) 400150 2.420 0.000 46.730 0.000 container.py:115(forward) 1200450 2.280 0.000 29.503 0.000 linear.py:90(forward) ```

yamatokataoka commented 4 years ago

spinningup's stats

``` [32;1mLogging data to /tmp/experiments/1602032256/progress.txt[0m [36;1mSaving config: [0m { "ac_kwargs": {}, "actor_critic": "MLPActorCritic", "env_fn": " at 0x7fb9815869d8>", "epochs": 50, "gamma": 0.99, "lam": 0.97, "logger": { "": { "epoch_dict": {}, "exp_name": null, "first_row": true, "log_current_row": {}, "log_headers": [], "output_dir": "/tmp/experiments/1602032256", "output_file": { "<_io.TextIOWrapper name='/tmp/experiments/1602032256/progress.txt' mode='w' encoding='ANSI_X3.4-1968'>": { "mode": "w" } } } }, "logger_kwargs": {}, "max_ep_len": 1000, "pi_lr": 0.0003, "save_freq": 10, "seed": 0, "steps_per_epoch": 4000, "train_v_iters": 80, "vf_lr": 0.001 } [32;1m Number of parameters: pi: 4610, v: 4545 [0m Warning: trajectory cut off by epoch at 17 steps. --------------------------------------- | Epoch | 0 | | AverageEpRet | 21.5 | | StdEpRet | 11.3 | | MaxEpRet | 66 | | MinEpRet | 8 | | EpLen | 21.5 | | AverageVVals | -0.26 | | StdVVals | 0.0665 | | MaxVVals | -0.0211 | | MinVVals | -0.448 | | TotalEnvInteracts | 4e+03 | | LossPi | 0.00715 | | LossV | 253 | | DeltaLossPi | 0 | | DeltaLossV | -148 | | Entropy | 0.689 | | KL | 4.92e-10 | | Time | 4.09 | --------------------------------------- Warning: trajectory cut off by epoch at 12 steps. --------------------------------------- | Epoch | 1 | | AverageEpRet | 21.8 | | StdEpRet | 10.8 | | MaxEpRet | 71 | | MinEpRet | 9 | | EpLen | 21.8 | | AverageVVals | 7.82 | | StdVVals | 0.333 | | MaxVVals | 8.28 | | MinVVals | 5.26 | | TotalEnvInteracts | 8e+03 | | LossPi | 0.00567 | | LossV | 102 | | DeltaLossPi | 0 | | DeltaLossV | -31.2 | | Entropy | 0.689 | | KL | -3.87e-10 | | Time | 7.35 | --------------------------------------- --------------------------------------- | Epoch | 2 | | AverageEpRet | 22.9 | | StdEpRet | 12.9 | | MaxEpRet | 111 | | MinEpRet | 8 | | EpLen | 22.9 | | AverageVVals | 11 | | StdVVals | 2.93 | | MaxVVals | 12.8 | | MinVVals | -8.08 | | TotalEnvInteracts | 1.2e+04 | | LossPi | 0.00374 | | LossV | 102 | | DeltaLossPi | 0 | | DeltaLossV | -11.2 | | Entropy | 0.69 | | KL | 8.64e-10 | | Time | 10.4 | --------------------------------------- Warning: trajectory cut off by epoch at 3 steps. --------------------------------------- | Epoch | 3 | | AverageEpRet | 20.9 | | StdEpRet | 9.85 | | MaxEpRet | 72 | | MinEpRet | 9 | | EpLen | 20.9 | | AverageVVals | 12.7 | | StdVVals | 4.15 | | MaxVVals | 15.2 | | MinVVals | -6.07 | | TotalEnvInteracts | 1.6e+04 | | LossPi | 0.00271 | | LossV | 60.9 | | DeltaLossPi | 0 | | DeltaLossV | -1.88 | | Entropy | 0.691 | | KL | 1.34e-10 | | Time | 13.5 | --------------------------------------- Warning: trajectory cut off by epoch at 31 steps. --------------------------------------- | Epoch | 4 | | AverageEpRet | 22.2 | | StdEpRet | 12.7 | | MaxEpRet | 95 | | MinEpRet | 9 | | EpLen | 22.2 | | AverageVVals | 12.4 | | StdVVals | 3.7 | | MaxVVals | 14.9 | | MinVVals | -1.52 | | TotalEnvInteracts | 2e+04 | | LossPi | -0.000394 | | LossV | 92.3 | | DeltaLossPi | 0 | | DeltaLossV | -3.1 | | Entropy | 0.691 | | KL | -3.13e-10 | | Time | 16.6 | --------------------------------------- Warning: trajectory cut off by epoch at 15 steps. --------------------------------------- | Epoch | 5 | | AverageEpRet | 23.2 | | StdEpRet | 12.9 | | MaxEpRet | 85 | | MinEpRet | 9 | | EpLen | 23.2 | | AverageVVals | 13.2 | | StdVVals | 3.82 | | MaxVVals | 15.7 | | MinVVals | -2.77 | | TotalEnvInteracts | 2.4e+04 | | LossPi | -0.000898 | | LossV | 87.9 | | DeltaLossPi | 0 | | DeltaLossV | -1.83 | | Entropy | 0.691 | | KL | -1.34e-10 | | Time | 19.8 | --------------------------------------- Warning: trajectory cut off by epoch at 13 steps. --------------------------------------- | Epoch | 6 | | AverageEpRet | 23.2 | | StdEpRet | 11.8 | | MaxEpRet | 73 | | MinEpRet | 9 | | EpLen | 23.2 | | AverageVVals | 14 | | StdVVals | 4.41 | | MaxVVals | 16.7 | | MinVVals | -3.97 | | TotalEnvInteracts | 2.8e+04 | | LossPi | -0.00537 | | LossV | 73.9 | | DeltaLossPi | 0 | | DeltaLossV | -1.42 | | Entropy | 0.691 | | KL | 1.49e-11 | | Time | 22.9 | --------------------------------------- Warning: trajectory cut off by epoch at 40 steps. --------------------------------------- | Epoch | 7 | | AverageEpRet | 24.3 | | StdEpRet | 14.2 | | MaxEpRet | 117 | | MinEpRet | 9 | | EpLen | 24.3 | | AverageVVals | 14 | | StdVVals | 4.06 | | MaxVVals | 16.5 | | MinVVals | -3.01 | | TotalEnvInteracts | 3.2e+04 | | LossPi | -0.0054 | | LossV | 115 | | DeltaLossPi | 0 | | DeltaLossV | -7.1 | | Entropy | 0.691 | | KL | -1.04e-10 | | Time | 26 | --------------------------------------- Warning: trajectory cut off by epoch at 9 steps. --------------------------------------- | Epoch | 8 | | AverageEpRet | 24 | | StdEpRet | 13 | | MaxEpRet | 105 | | MinEpRet | 8 | | EpLen | 24 | | AverageVVals | 14.4 | | StdVVals | 4.12 | | MaxVVals | 17.4 | | MinVVals | -0.732 | | TotalEnvInteracts | 3.6e+04 | | LossPi | -0.00927 | | LossV | 93.1 | | DeltaLossPi | 0 | | DeltaLossV | -8.95 | | Entropy | 0.69 | | KL | -8.2e-10 | | Time | 29.1 | --------------------------------------- Warning: trajectory cut off by epoch at 2 steps. --------------------------------------- | Epoch | 9 | | AverageEpRet | 25.6 | | StdEpRet | 17.9 | | MaxEpRet | 128 | | MinEpRet | 9 | | EpLen | 25.6 | | AverageVVals | 14 | | StdVVals | 4.62 | | MaxVVals | 18.3 | | MinVVals | -0.92 | | TotalEnvInteracts | 4e+04 | | LossPi | -0.0116 | | LossV | 151 | | DeltaLossPi | 0 | | DeltaLossV | -14 | | Entropy | 0.69 | | KL | 1.19e-09 | | Time | 32.3 | --------------------------------------- --------------------------------------- | Epoch | 10 | | AverageEpRet | 26.1 | | StdEpRet | 15.6 | | MaxEpRet | 107 | | MinEpRet | 10 | | EpLen | 26.1 | | AverageVVals | 16 | | StdVVals | 6.07 | | MaxVVals | 21.7 | | MinVVals | -3.28 | | TotalEnvInteracts | 4.4e+04 | | LossPi | -0.0136 | | LossV | 106 | | DeltaLossPi | 0 | | DeltaLossV | -5.39 | | Entropy | 0.689 | | KL | 3.05e-10 | | Time | 35.4 | --------------------------------------- Warning: trajectory cut off by epoch at 7 steps. --------------------------------------- | Epoch | 11 | | AverageEpRet | 26.8 | | StdEpRet | 16 | | MaxEpRet | 121 | | MinEpRet | 9 | | EpLen | 26.8 | | AverageVVals | 15.6 | | StdVVals | 6.16 | | MaxVVals | 22 | | MinVVals | -0.324 | | TotalEnvInteracts | 4.8e+04 | | LossPi | -0.0145 | | LossV | 103 | | DeltaLossPi | 0 | | DeltaLossV | -5.9 | | Entropy | 0.689 | | KL | 2.46e-10 | | Time | 38.5 | --------------------------------------- Warning: trajectory cut off by epoch at 3 steps. --------------------------------------- | Epoch | 12 | | AverageEpRet | 26.5 | | StdEpRet | 15.6 | | MaxEpRet | 85 | | MinEpRet | 9 | | EpLen | 26.5 | | AverageVVals | 16.1 | | StdVVals | 6.14 | | MaxVVals | 22.5 | | MinVVals | -0.198 | | TotalEnvInteracts | 5.2e+04 | | LossPi | -0.0168 | | LossV | 92.1 | | DeltaLossPi | 0 | | DeltaLossV | -3.61 | | Entropy | 0.688 | | KL | -2.24e-10 | | Time | 41.6 | --------------------------------------- Warning: trajectory cut off by epoch at 10 steps. --------------------------------------- | Epoch | 13 | | AverageEpRet | 25.7 | | StdEpRet | 14.6 | | MaxEpRet | 96 | | MinEpRet | 9 | | EpLen | 25.7 | | AverageVVals | 16 | | StdVVals | 6.61 | | MaxVVals | 23.1 | | MinVVals | 0.173 | | TotalEnvInteracts | 5.6e+04 | | LossPi | -0.0196 | | LossV | 75.7 | | DeltaLossPi | 0 | | DeltaLossV | -2.56 | | Entropy | 0.687 | | KL | -1.13e-09 | | Time | 44.7 | --------------------------------------- Warning: trajectory cut off by epoch at 39 steps. --------------------------------------- | Epoch | 14 | | AverageEpRet | 27.1 | | StdEpRet | 16.5 | | MaxEpRet | 119 | | MinEpRet | 10 | | EpLen | 27.1 | | AverageVVals | 15.7 | | StdVVals | 6.66 | | MaxVVals | 22.8 | | MinVVals | 0.412 | | TotalEnvInteracts | 6e+04 | | LossPi | -0.0207 | | LossV | 105 | | DeltaLossPi | 0 | | DeltaLossV | -6.69 | | Entropy | 0.686 | | KL | 2.98e-11 | | Time | 47.9 | --------------------------------------- Warning: trajectory cut off by epoch at 5 steps. --------------------------------------- | Epoch | 15 | | AverageEpRet | 25.4 | | StdEpRet | 13 | | MaxEpRet | 106 | | MinEpRet | 9 | | EpLen | 25.4 | | AverageVVals | 15.9 | | StdVVals | 7.47 | | MaxVVals | 23.8 | | MinVVals | 0.7 | | TotalEnvInteracts | 6.4e+04 | | LossPi | -0.0244 | | LossV | 72.7 | | DeltaLossPi | 0 | | DeltaLossV | -9.23 | | Entropy | 0.685 | | KL | -1.56e-10 | | Time | 51 | --------------------------------------- Warning: trajectory cut off by epoch at 2 steps. --------------------------------------- | Epoch | 16 | | AverageEpRet | 28.4 | | StdEpRet | 17.1 | | MaxEpRet | 106 | | MinEpRet | 10 | | EpLen | 28.4 | | AverageVVals | 14.8 | | StdVVals | 6.43 | | MaxVVals | 22.2 | | MinVVals | 1.48 | | TotalEnvInteracts | 6.8e+04 | | LossPi | -0.0229 | | LossV | 117 | | DeltaLossPi | 0 | | DeltaLossV | -9.85 | | Entropy | 0.684 | | KL | -3.73e-10 | | Time | 54.1 | --------------------------------------- Warning: trajectory cut off by epoch at 17 steps. --------------------------------------- | Epoch | 17 | | AverageEpRet | 28.9 | | StdEpRet | 15.7 | | MaxEpRet | 84 | | MinEpRet | 9 | | EpLen | 28.9 | | AverageVVals | 17.1 | | StdVVals | 6.82 | | MaxVVals | 23.8 | | MinVVals | 0.706 | | TotalEnvInteracts | 7.2e+04 | | LossPi | -0.0279 | | LossV | 78.3 | | DeltaLossPi | 0 | | DeltaLossV | -3.62 | | Entropy | 0.683 | | KL | 1.05e-09 | | Time | 57.2 | --------------------------------------- Warning: trajectory cut off by epoch at 26 steps. --------------------------------------- | Epoch | 18 | | AverageEpRet | 29.7 | | StdEpRet | 15.7 | | MaxEpRet | 86 | | MinEpRet | 9 | | EpLen | 29.7 | | AverageVVals | 16.8 | | StdVVals | 7.39 | | MaxVVals | 24 | | MinVVals | 0.922 | | TotalEnvInteracts | 7.6e+04 | | LossPi | -0.0258 | | LossV | 78.2 | | DeltaLossPi | 0 | | DeltaLossV | -2.41 | | Entropy | 0.683 | | KL | 4.4e-10 | | Time | 60.3 | --------------------------------------- Warning: trajectory cut off by epoch at 11 steps. --------------------------------------- | Epoch | 19 | | AverageEpRet | 27.9 | | StdEpRet | 13.7 | | MaxEpRet | 77 | | MinEpRet | 10 | | EpLen | 27.9 | | AverageVVals | 16.1 | | StdVVals | 7.72 | | MaxVVals | 24.3 | | MinVVals | 0.243 | | TotalEnvInteracts | 8e+04 | | LossPi | -0.0311 | | LossV | 65 | | DeltaLossPi | 0 | | DeltaLossV | -3.08 | | Entropy | 0.681 | | KL | 8.05e-10 | | Time | 63.4 | --------------------------------------- Warning: trajectory cut off by epoch at 15 steps. --------------------------------------- | Epoch | 20 | | AverageEpRet | 30.4 | | StdEpRet | 18.9 | | MaxEpRet | 122 | | MinEpRet | 11 | | EpLen | 30.4 | | AverageVVals | 15.8 | | StdVVals | 6.88 | | MaxVVals | 23.7 | | MinVVals | 0.291 | | TotalEnvInteracts | 8.4e+04 | | LossPi | -0.0289 | | LossV | 126 | | DeltaLossPi | 0 | | DeltaLossV | -10.7 | | Entropy | 0.68 | | KL | -3.13e-10 | | Time | 66.6 | --------------------------------------- Warning: trajectory cut off by epoch at 6 steps. --------------------------------------- | Epoch | 21 | | AverageEpRet | 32.5 | | StdEpRet | 18.1 | | MaxEpRet | 99 | | MinEpRet | 11 | | EpLen | 32.5 | | AverageVVals | 18.2 | | StdVVals | 7.82 | | MaxVVals | 25.8 | | MinVVals | 0.99 | | TotalEnvInteracts | 8.8e+04 | | LossPi | -0.03 | | LossV | 111 | | DeltaLossPi | 0 | | DeltaLossV | -2.78 | | Entropy | 0.68 | | KL | -7.15e-10 | | Time | 69.7 | --------------------------------------- Warning: trajectory cut off by epoch at 3 steps. --------------------------------------- | Epoch | 22 | | AverageEpRet | 31 | | StdEpRet | 20.7 | | MaxEpRet | 133 | | MinEpRet | 10 | | EpLen | 31 | | AverageVVals | 18.6 | | StdVVals | 7.58 | | MaxVVals | 25.9 | | MinVVals | 0.63 | | TotalEnvInteracts | 9.2e+04 | | LossPi | -0.032 | | LossV | 157 | | DeltaLossPi | 0 | | DeltaLossV | -4.81 | | Entropy | 0.677 | | KL | 4.92e-10 | | Time | 72.8 | --------------------------------------- Warning: trajectory cut off by epoch at 20 steps. --------------------------------------- | Epoch | 23 | | AverageEpRet | 32.9 | | StdEpRet | 20.2 | | MaxEpRet | 109 | | MinEpRet | 9 | | EpLen | 32.9 | | AverageVVals | 19 | | StdVVals | 7.68 | | MaxVVals | 26.4 | | MinVVals | 0.413 | | TotalEnvInteracts | 9.6e+04 | | LossPi | -0.035 | | LossV | 128 | | DeltaLossPi | 0 | | DeltaLossV | -5.26 | | Entropy | 0.676 | | KL | -3.05e-10 | | Time | 75.9 | --------------------------------------- Warning: trajectory cut off by epoch at 1 steps. --------------------------------------- | Epoch | 24 | | AverageEpRet | 34.8 | | StdEpRet | 19 | | MaxEpRet | 106 | | MinEpRet | 10 | | EpLen | 34.8 | | AverageVVals | 19.5 | | StdVVals | 8.08 | | MaxVVals | 27.5 | | MinVVals | 1.19 | | TotalEnvInteracts | 1e+05 | | LossPi | -0.0353 | | LossV | 108 | | DeltaLossPi | 0 | | DeltaLossV | -2.43 | | Entropy | 0.676 | | KL | -2.91e-10 | | Time | 79 | --------------------------------------- Warning: trajectory cut off by epoch at 15 steps. --------------------------------------- | Epoch | 25 | | AverageEpRet | 34.1 | | StdEpRet | 19.8 | | MaxEpRet | 172 | | MinEpRet | 11 | | EpLen | 34.1 | | AverageVVals | 19.4 | | StdVVals | 8.28 | | MaxVVals | 27.7 | | MinVVals | 0.966 | | TotalEnvInteracts | 1.04e+05 | | LossPi | -0.0323 | | LossV | 144 | | DeltaLossPi | 0 | | DeltaLossV | -9.21 | | Entropy | 0.675 | | KL | 5.74e-10 | | Time | 82.2 | --------------------------------------- Warning: trajectory cut off by epoch at 19 steps. --------------------------------------- | Epoch | 26 | | AverageEpRet | 31.8 | | StdEpRet | 17.6 | | MaxEpRet | 120 | | MinEpRet | 12 | | EpLen | 31.8 | | AverageVVals | 18.9 | | StdVVals | 8.14 | | MaxVVals | 27.1 | | MinVVals | 0.946 | | TotalEnvInteracts | 1.08e+05 | | LossPi | -0.0387 | | LossV | 111 | | DeltaLossPi | 0 | | DeltaLossV | -6.38 | | Entropy | 0.672 | | KL | 2.01e-10 | | Time | 85.2 | --------------------------------------- Warning: trajectory cut off by epoch at 12 steps. --------------------------------------- | Epoch | 27 | | AverageEpRet | 34.7 | | StdEpRet | 21.8 | | MaxEpRet | 134 | | MinEpRet | 10 | | EpLen | 34.7 | | AverageVVals | 18.9 | | StdVVals | 7.78 | | MaxVVals | 27.4 | | MinVVals | 0.705 | | TotalEnvInteracts | 1.12e+05 | | LossPi | -0.036 | | LossV | 145 | | DeltaLossPi | 0 | | DeltaLossV | -7.16 | | Entropy | 0.671 | | KL | -2.09e-09 | | Time | 88.3 | --------------------------------------- Warning: trajectory cut off by epoch at 29 steps. --------------------------------------- | Epoch | 28 | | AverageEpRet | 33.7 | | StdEpRet | 17.1 | | MaxEpRet | 100 | | MinEpRet | 10 | | EpLen | 33.7 | | AverageVVals | 18.7 | | StdVVals | 8.79 | | MaxVVals | 29.1 | | MinVVals | 0.367 | | TotalEnvInteracts | 1.16e+05 | | LossPi | -0.0408 | | LossV | 96.7 | | DeltaLossPi | 0 | | DeltaLossV | -9.85 | | Entropy | 0.671 | | KL | -2.91e-10 | | Time | 91.5 | --------------------------------------- Warning: trajectory cut off by epoch at 6 steps. --------------------------------------- | Epoch | 29 | | AverageEpRet | 38.4 | | StdEpRet | 27.2 | | MaxEpRet | 155 | | MinEpRet | 10 | | EpLen | 38.4 | | AverageVVals | 19.2 | | StdVVals | 7.58 | | MaxVVals | 27.3 | | MinVVals | 1.3 | | TotalEnvInteracts | 1.2e+05 | | LossPi | -0.0358 | | LossV | 222 | | DeltaLossPi | 0 | | DeltaLossV | -26.5 | | Entropy | 0.669 | | KL | -7.75e-10 | | Time | 94.6 | --------------------------------------- Warning: trajectory cut off by epoch at 12 steps. --------------------------------------- | Epoch | 30 | | AverageEpRet | 35.6 | | StdEpRet | 19 | | MaxEpRet | 101 | | MinEpRet | 11 | | EpLen | 35.6 | | AverageVVals | 22.9 | | StdVVals | 9.87 | | MaxVVals | 32.9 | | MinVVals | -0.156 | | TotalEnvInteracts | 1.24e+05 | | LossPi | -0.04 | | LossV | 125 | | DeltaLossPi | 0 | | DeltaLossV | -17.6 | | Entropy | 0.668 | | KL | 1.79e-10 | | Time | 97.7 | --------------------------------------- Warning: trajectory cut off by epoch at 27 steps. --------------------------------------- | Epoch | 31 | | AverageEpRet | 34.9 | | StdEpRet | 16.6 | | MaxEpRet | 102 | | MinEpRet | 12 | | EpLen | 34.9 | | AverageVVals | 19.1 | | StdVVals | 8.56 | | MaxVVals | 29.5 | | MinVVals | 1.59 | | TotalEnvInteracts | 1.28e+05 | | LossPi | -0.0396 | | LossV | 90.3 | | DeltaLossPi | 0 | | DeltaLossV | -2.3 | | Entropy | 0.669 | | KL | 4.25e-10 | | Time | 101 | --------------------------------------- Warning: trajectory cut off by epoch at 9 steps. --------------------------------------- | Epoch | 32 | | AverageEpRet | 35.6 | | StdEpRet | 19.9 | | MaxEpRet | 103 | | MinEpRet | 12 | | EpLen | 35.6 | | AverageVVals | 19.3 | | StdVVals | 8.06 | | MaxVVals | 28.5 | | MinVVals | 0.544 | | TotalEnvInteracts | 1.32e+05 | | LossPi | -0.0424 | | LossV | 116 | | DeltaLossPi | 0 | | DeltaLossV | -2.03 | | Entropy | 0.664 | | KL | -1.33e-09 | | Time | 104 | --------------------------------------- Warning: trajectory cut off by epoch at 59 steps. --------------------------------------- | Epoch | 33 | | AverageEpRet | 36.8 | | StdEpRet | 23.4 | | MaxEpRet | 126 | | MinEpRet | 10 | | EpLen | 36.8 | | AverageVVals | 19.8 | | StdVVals | 8.29 | | MaxVVals | 28.9 | | MinVVals | 0.913 | | TotalEnvInteracts | 1.36e+05 | | LossPi | -0.044 | | LossV | 155 | | DeltaLossPi | 0 | | DeltaLossV | -8.15 | | Entropy | 0.664 | | KL | -3.13e-10 | | Time | 107 | --------------------------------------- Warning: trajectory cut off by epoch at 56 steps. --------------------------------------- | Epoch | 34 | | AverageEpRet | 38.7 | | StdEpRet | 23 | | MaxEpRet | 128 | | MinEpRet | 11 | | EpLen | 38.7 | | AverageVVals | 21.7 | | StdVVals | 8.98 | | MaxVVals | 32.1 | | MinVVals | 0.141 | | TotalEnvInteracts | 1.4e+05 | | LossPi | -0.0454 | | LossV | 148 | | DeltaLossPi | 0 | | DeltaLossV | -4.9 | | Entropy | 0.664 | | KL | -4.02e-10 | | Time | 110 | --------------------------------------- Warning: trajectory cut off by epoch at 17 steps. --------------------------------------- | Epoch | 35 | | AverageEpRet | 42.8 | | StdEpRet | 23.7 | | MaxEpRet | 141 | | MinEpRet | 11 | | EpLen | 42.8 | | AverageVVals | 22.7 | | StdVVals | 9.59 | | MaxVVals | 32.1 | | MinVVals | 0.15 | | TotalEnvInteracts | 1.44e+05 | | LossPi | -0.0388 | | LossV | 139 | | DeltaLossPi | 0 | | DeltaLossV | -4.08 | | Entropy | 0.664 | | KL | 6.71e-11 | | Time | 113 | --------------------------------------- Warning: trajectory cut off by epoch at 3 steps. --------------------------------------- | Epoch | 36 | | AverageEpRet | 36.7 | | StdEpRet | 17.7 | | MaxEpRet | 89 | | MinEpRet | 10 | | EpLen | 36.7 | | AverageVVals | 22 | | StdVVals | 9.99 | | MaxVVals | 33.8 | | MinVVals | 0.767 | | TotalEnvInteracts | 1.48e+05 | | LossPi | -0.0451 | | LossV | 91 | | DeltaLossPi | 0 | | DeltaLossV | -8.08 | | Entropy | 0.662 | | KL | -5.96e-11 | | Time | 116 | --------------------------------------- Warning: trajectory cut off by epoch at 5 steps. --------------------------------------- | Epoch | 37 | | AverageEpRet | 38.4 | | StdEpRet | 19.6 | | MaxEpRet | 96 | | MinEpRet | 12 | | EpLen | 38.4 | | AverageVVals | 19.7 | | StdVVals | 9 | | MaxVVals | 31.6 | | MinVVals | 0.745 | | TotalEnvInteracts | 1.52e+05 | | LossPi | -0.0429 | | LossV | 109 | | DeltaLossPi | 0 | | DeltaLossV | -4.91 | | Entropy | 0.659 | | KL | -2.68e-10 | | Time | 119 | --------------------------------------- Warning: trajectory cut off by epoch at 40 steps. --------------------------------------- | Epoch | 38 | | AverageEpRet | 43.5 | | StdEpRet | 25.9 | | MaxEpRet | 169 | | MinEpRet | 9 | | EpLen | 43.5 | | AverageVVals | 20.8 | | StdVVals | 8.95 | | MaxVVals | 32.1 | | MinVVals | -0.266 | | TotalEnvInteracts | 1.56e+05 | | LossPi | -0.0421 | | LossV | 176 | | DeltaLossPi | 0 | | DeltaLossV | -21 | | Entropy | 0.658 | | KL | -1.21e-09 | | Time | 122 | --------------------------------------- Warning: trajectory cut off by epoch at 5 steps. --------------------------------------- | Epoch | 39 | | AverageEpRet | 43.9 | | StdEpRet | 28.4 | | MaxEpRet | 187 | | MinEpRet | 12 | | EpLen | 43.9 | | AverageVVals | 24 | | StdVVals | 9.99 | | MaxVVals | 35.2 | | MinVVals | 0.99 | | TotalEnvInteracts | 1.6e+05 | | LossPi | -0.0425 | | LossV | 203 | | DeltaLossPi | 0 | | DeltaLossV | -13.5 | | Entropy | 0.658 | | KL | -3.2e-10 | | Time | 126 | --------------------------------------- Warning: trajectory cut off by epoch at 28 steps. --------------------------------------- | Epoch | 40 | | AverageEpRet | 44.6 | | StdEpRet | 25.8 | | MaxEpRet | 119 | | MinEpRet | 13 | | EpLen | 44.6 | | AverageVVals | 25.4 | | StdVVals | 10.7 | | MaxVVals | 35.4 | | MinVVals | 0.72 | | TotalEnvInteracts | 1.64e+05 | | LossPi | -0.0358 | | LossV | 167 | | DeltaLossPi | 0 | | DeltaLossV | -11 | | Entropy | 0.656 | | KL | 6.18e-10 | | Time | 129 | --------------------------------------- Warning: trajectory cut off by epoch at 20 steps. --------------------------------------- | Epoch | 41 | | AverageEpRet | 41.5 | | StdEpRet | 25.9 | | MaxEpRet | 154 | | MinEpRet | 14 | | EpLen | 41.5 | | AverageVVals | 23.4 | | StdVVals | 10.2 | | MaxVVals | 35.3 | | MinVVals | 0.565 | | TotalEnvInteracts | 1.68e+05 | | LossPi | -0.0433 | | LossV | 179 | | DeltaLossPi | 0 | | DeltaLossV | -5.17 | | Entropy | 0.653 | | KL | -2.76e-10 | | Time | 132 | --------------------------------------- Warning: trajectory cut off by epoch at 21 steps. --------------------------------------- | Epoch | 42 | | AverageEpRet | 41.9 | | StdEpRet | 22.5 | | MaxEpRet | 115 | | MinEpRet | 13 | | EpLen | 41.9 | | AverageVVals | 22.5 | | StdVVals | 10.3 | | MaxVVals | 33.8 | | MinVVals | 0.308 | | TotalEnvInteracts | 1.72e+05 | | LossPi | -0.0418 | | LossV | 144 | | DeltaLossPi | 0 | | DeltaLossV | -5.07 | | Entropy | 0.654 | | KL | 1.01e-09 | | Time | 135 | --------------------------------------- Warning: trajectory cut off by epoch at 23 steps. --------------------------------------- | Epoch | 43 | | AverageEpRet | 42.3 | | StdEpRet | 25.4 | | MaxEpRet | 200 | | MinEpRet | 12 | | EpLen | 42.3 | | AverageVVals | 22.7 | | StdVVals | 9.04 | | MaxVVals | 32.7 | | MinVVals | 0.33 | | TotalEnvInteracts | 1.76e+05 | | LossPi | -0.0511 | | LossV | 164 | | DeltaLossPi | 0 | | DeltaLossV | -7.32 | | Entropy | 0.653 | | KL | -6.85e-10 | | Time | 138 | --------------------------------------- Warning: trajectory cut off by epoch at 30 steps. --------------------------------------- | Epoch | 44 | | AverageEpRet | 40.1 | | StdEpRet | 20.8 | | MaxEpRet | 113 | | MinEpRet | 11 | | EpLen | 40.1 | | AverageVVals | 23.4 | | StdVVals | 10.2 | | MaxVVals | 34.2 | | MinVVals | 0.692 | | TotalEnvInteracts | 1.8e+05 | | LossPi | -0.0534 | | LossV | 119 | | DeltaLossPi | 0 | | DeltaLossV | -9.65 | | Entropy | 0.648 | | KL | -5.89e-10 | | Time | 141 | --------------------------------------- Warning: trajectory cut off by epoch at 49 steps. --------------------------------------- | Epoch | 45 | | AverageEpRet | 41.6 | | StdEpRet | 24.1 | | MaxEpRet | 146 | | MinEpRet | 12 | | EpLen | 41.6 | | AverageVVals | 21.3 | | StdVVals | 9.82 | | MaxVVals | 32.3 | | MinVVals | 0.703 | | TotalEnvInteracts | 1.84e+05 | | LossPi | -0.0527 | | LossV | 148 | | DeltaLossPi | 0 | | DeltaLossV | -6.75 | | Entropy | 0.648 | | KL | -9.83e-10 | | Time | 144 | --------------------------------------- Warning: trajectory cut off by epoch at 20 steps. --------------------------------------- | Epoch | 46 | | AverageEpRet | 44.7 | | StdEpRet | 23.1 | | MaxEpRet | 108 | | MinEpRet | 11 | | EpLen | 44.7 | | AverageVVals | 22.2 | | StdVVals | 9.82 | | MaxVVals | 34.4 | | MinVVals | 0.685 | | TotalEnvInteracts | 1.88e+05 | | LossPi | -0.0471 | | LossV | 127 | | DeltaLossPi | 0 | | DeltaLossV | -7.4 | | Entropy | 0.651 | | KL | -3.43e-10 | | Time | 147 | --------------------------------------- Warning: trajectory cut off by epoch at 78 steps. --------------------------------------- | Epoch | 47 | | AverageEpRet | 45.1 | | StdEpRet | 21.9 | | MaxEpRet | 110 | | MinEpRet | 10 | | EpLen | 45.1 | | AverageVVals | 24.2 | | StdVVals | 10 | | MaxVVals | 36.4 | | MinVVals | 0.665 | | TotalEnvInteracts | 1.92e+05 | | LossPi | -0.0414 | | LossV | 122 | | DeltaLossPi | 0 | | DeltaLossV | -5.13 | | Entropy | 0.647 | | KL | -6.93e-10 | | Time | 150 | --------------------------------------- Warning: trajectory cut off by epoch at 62 steps. --------------------------------------- | Epoch | 48 | | AverageEpRet | 45.8 | | StdEpRet | 25.4 | | MaxEpRet | 153 | | MinEpRet | 14 | | EpLen | 45.8 | | AverageVVals | 22.7 | | StdVVals | 9.83 | | MaxVVals | 36.9 | | MinVVals | 0.653 | | TotalEnvInteracts | 1.96e+05 | | LossPi | -0.0416 | | LossV | 172 | | DeltaLossPi | 0 | | DeltaLossV | -20.5 | | Entropy | 0.65 | | KL | 4.4e-10 | | Time | 153 | --------------------------------------- Warning: trajectory cut off by epoch at 33 steps. --------------------------------------- | Epoch | 49 | | AverageEpRet | 45.1 | | StdEpRet | 26.4 | | MaxEpRet | 159 | | MinEpRet | 12 | | EpLen | 45.1 | | AverageVVals | 24.2 | | StdVVals | 11 | | MaxVVals | 37.5 | | MinVVals | 0.224 | | TotalEnvInteracts | 2e+05 | | LossPi | -0.0439 | | LossV | 171 | | DeltaLossPi | 0 | | DeltaLossV | -8.56 | | Entropy | 0.647 | | KL | 1.07e-09 | | Time | 157 | --------------------------------------- 43216335 function calls (40435708 primitive calls) in 169.783 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 1200300 18.082 0.000 18.082 0.000 {method 'matmul' of 'torch._C._TensorBase' objects} 4050 14.348 0.004 14.348 0.004 {method 'run_backward' of 'torch._C._EngineBase' objects} 808500 13.275 0.000 13.275 0.000 {built-in method tanh} 200150 12.876 0.000 12.876 0.000 {method 'logsumexp' of 'torch._C._TensorBase' objects} 1212750 9.589 0.000 38.420 0.000 functional.py:1355(linear) 3033950/404250 9.217 0.000 74.797 0.000 module.py:531(__call__) 2830 6.419 0.002 6.419 0.002 {method 'read' of '_io.FileIO' objects} 12450 5.643 0.000 5.643 0.000 {built-in method addmm} 200300 4.866 0.000 4.866 0.000 {built-in method as_tensor} 1212750 4.790 0.000 4.790 0.000 {method 't' of 'torch._C._TensorBase' objects} 200000 4.332 0.000 7.135 0.000 cartpole.py:91(step) 200150 3.459 0.000 17.169 0.000 categorical.py:44(__init__) 404250 3.391 0.000 69.865 0.000 container.py:90(forward) 1212750 3.138 0.000 42.537 0.000 linear.py:86(forward) 200050 2.890 0.000 104.369 0.001 core.py:126(step) 1 2.766 2.766 157.027 157.027 vpg.py:89(vpg) ```

yamatokataoka commented 4 years ago

code: https://github.com/yamatokataoka/reinforcement-learning-replications/commit/09dd98db30039b38d2b853fce642afc43a1571d7

the avarage value loss was increasing from 165.594 to 213.139.

value loss and peroformance of the multiple updates of value function.

``` 2020-10-10 07:58:41,292 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 9 steps on the current episode 2020-10-10 07:59:03,663 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00184 2020-10-10 07:59:03,663 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 166 2020-10-10 07:59:03,664 — rl_replicas.vpg.vpg — INFO — Epoch: 0 2020-10-10 07:59:03,665 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 24.4 2020-10-10 07:59:03,665 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 14.2 2020-10-10 07:59:03,666 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 91 2020-10-10 07:59:03,666 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 8 2020-10-10 07:59:03,667 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 24.4 2020-10-10 07:59:03,667 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 8.14 2020-10-10 07:59:03,667 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.492 2020-10-10 07:59:03,668 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 8.65 2020-10-10 07:59:03,668 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 4.33 2020-10-10 07:59:03,669 — rl_replicas.vpg.vpg — INFO — Total env interactions: 4000 2020-10-10 07:59:03,669 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00184 2020-10-10 07:59:03,670 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 165.594 2020-10-10 07:59:03,670 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.68 2020-10-10 07:59:03,670 — rl_replicas.vpg.vpg — INFO — Time: 24.9 2020-10-10 07:59:06,084 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 15 steps on the current episode 2020-10-10 07:59:28,165 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00324 2020-10-10 07:59:28,165 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 132 2020-10-10 07:59:28,166 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00139 2020-10-10 07:59:28,166 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -33.9 2020-10-10 07:59:28,166 — rl_replicas.vpg.vpg — INFO — Epoch: 1 2020-10-10 07:59:28,167 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 24.9 2020-10-10 07:59:28,167 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 14.7 2020-10-10 07:59:28,168 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 72 2020-10-10 07:59:28,168 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 8 2020-10-10 07:59:28,168 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 24.8 2020-10-10 07:59:28,169 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 12.9 2020-10-10 07:59:28,170 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.153 2020-10-10 07:59:28,170 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 13 2020-10-10 07:59:28,170 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 11.4 2020-10-10 07:59:28,171 — rl_replicas.vpg.vpg — INFO — Total env interactions: 8000 2020-10-10 07:59:28,171 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00254 2020-10-10 07:59:28,172 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 148.622 2020-10-10 07:59:28,173 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-10 07:59:28,173 — rl_replicas.vpg.vpg — INFO — Time: 49.4 2020-10-10 07:59:30,678 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 9 steps on the current episode 2020-10-10 07:59:52,627 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00193 2020-10-10 07:59:52,627 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 104 2020-10-10 07:59:52,628 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00131 2020-10-10 07:59:52,628 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -27.5 2020-10-10 07:59:52,628 — rl_replicas.vpg.vpg — INFO — Epoch: 2 2020-10-10 07:59:52,629 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 26.2 2020-10-10 07:59:52,629 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 12.7 2020-10-10 07:59:52,629 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 73 2020-10-10 07:59:52,630 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 07:59:52,630 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 26.1 2020-10-10 07:59:52,630 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 14.4 2020-10-10 07:59:52,631 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.106 2020-10-10 07:59:52,631 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 14.5 2020-10-10 07:59:52,631 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 13.5 2020-10-10 07:59:52,631 — rl_replicas.vpg.vpg — INFO — Total env interactions: 12000 2020-10-10 07:59:52,632 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00234 2020-10-10 07:59:52,632 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 133.806 2020-10-10 07:59:52,632 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.688 2020-10-10 07:59:52,632 — rl_replicas.vpg.vpg — INFO — Time: 73.9 2020-10-10 07:59:55,068 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 39 steps on the current episode 2020-10-10 08:00:19,326 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00506 2020-10-10 08:00:19,326 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 129 2020-10-10 08:00:19,327 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00313 2020-10-10 08:00:19,327 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 24.9 2020-10-10 08:00:19,327 — rl_replicas.vpg.vpg — INFO — Epoch: 3 2020-10-10 08:00:19,328 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 23.3 2020-10-10 08:00:19,328 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 14.1 2020-10-10 08:00:19,329 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 104 2020-10-10 08:00:19,329 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:00:19,330 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 23.3 2020-10-10 08:00:19,331 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 14.4 2020-10-10 08:00:19,331 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.092 2020-10-10 08:00:19,332 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 14.5 2020-10-10 08:00:19,332 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 13.5 2020-10-10 08:00:19,333 — rl_replicas.vpg.vpg — INFO — Total env interactions: 16000 2020-10-10 08:00:19,333 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00302 2020-10-10 08:00:19,334 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 132.618 2020-10-10 08:00:19,334 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-10 08:00:19,335 — rl_replicas.vpg.vpg — INFO — Time: 101 2020-10-10 08:00:22,087 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 9 steps on the current episode 2020-10-10 08:00:44,971 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00574 2020-10-10 08:00:44,972 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 164 2020-10-10 08:00:44,973 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000681 2020-10-10 08:00:44,973 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 35.2 2020-10-10 08:00:44,973 — rl_replicas.vpg.vpg — INFO — Epoch: 4 2020-10-10 08:00:44,974 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 26.4 2020-10-10 08:00:44,974 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 16.7 2020-10-10 08:00:44,975 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 131 2020-10-10 08:00:44,975 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:00:44,975 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 26.3 2020-10-10 08:00:44,976 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 15.9 2020-10-10 08:00:44,976 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0813 2020-10-10 08:00:44,976 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 15.9 2020-10-10 08:00:44,977 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 15.3 2020-10-10 08:00:44,977 — rl_replicas.vpg.vpg — INFO — Total env interactions: 20000 2020-10-10 08:00:44,977 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00356 2020-10-10 08:00:44,978 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 138.939 2020-10-10 08:00:44,978 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.688 2020-10-10 08:00:44,978 — rl_replicas.vpg.vpg — INFO — Time: 126 2020-10-10 08:01:09,505 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0103 2020-10-10 08:01:09,506 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 137 2020-10-10 08:01:09,506 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00454 2020-10-10 08:01:09,506 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -27.5 2020-10-10 08:01:09,507 — rl_replicas.vpg.vpg — INFO — Epoch: 5 2020-10-10 08:01:09,507 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 27.1 2020-10-10 08:01:09,507 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 15.3 2020-10-10 08:01:09,508 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 104 2020-10-10 08:01:09,508 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:01:09,508 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 27 2020-10-10 08:01:09,509 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 16 2020-10-10 08:01:09,509 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.056 2020-10-10 08:01:09,509 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 16 2020-10-10 08:01:09,509 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 15.5 2020-10-10 08:01:09,510 — rl_replicas.vpg.vpg — INFO — Total env interactions: 24000 2020-10-10 08:01:09,510 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00468 2020-10-10 08:01:09,510 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 138.571 2020-10-10 08:01:09,510 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.687 2020-10-10 08:01:09,511 — rl_replicas.vpg.vpg — INFO — Time: 151 2020-10-10 08:01:11,966 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 20 steps on the current episode 2020-10-10 08:01:33,994 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00805 2020-10-10 08:01:33,995 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 186 2020-10-10 08:01:33,995 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00223 2020-10-10 08:01:33,995 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 49.3 2020-10-10 08:01:33,995 — rl_replicas.vpg.vpg — INFO — Epoch: 6 2020-10-10 08:01:33,996 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 30 2020-10-10 08:01:33,996 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 19 2020-10-10 08:01:33,997 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 113 2020-10-10 08:01:33,997 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 8 2020-10-10 08:01:33,997 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 29.9 2020-10-10 08:01:33,997 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 17.9 2020-10-10 08:01:33,998 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0563 2020-10-10 08:01:33,998 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 18 2020-10-10 08:01:33,999 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 17.5 2020-10-10 08:01:33,999 — rl_replicas.vpg.vpg — INFO — Total env interactions: 28000 2020-10-10 08:01:33,999 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00517 2020-10-10 08:01:34,000 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 145.344 2020-10-10 08:01:34,001 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.688 2020-10-10 08:01:34,001 — rl_replicas.vpg.vpg — INFO — Time: 175 2020-10-10 08:01:36,435 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 9 steps on the current episode 2020-10-10 08:01:58,689 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0118 2020-10-10 08:01:58,690 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 133 2020-10-10 08:01:58,691 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00378 2020-10-10 08:01:58,691 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -52.8 2020-10-10 08:01:58,691 — rl_replicas.vpg.vpg — INFO — Epoch: 7 2020-10-10 08:01:58,692 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 28.5 2020-10-10 08:01:58,692 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 15.2 2020-10-10 08:01:58,693 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 91 2020-10-10 08:01:58,693 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:01:58,693 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 28.4 2020-10-10 08:01:58,694 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 16.5 2020-10-10 08:01:58,694 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0497 2020-10-10 08:01:58,695 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 16.5 2020-10-10 08:01:58,695 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 15.9 2020-10-10 08:01:58,695 — rl_replicas.vpg.vpg — INFO — Total env interactions: 32000 2020-10-10 08:01:58,696 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.006 2020-10-10 08:01:58,696 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 143.826 2020-10-10 08:01:58,696 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-10 08:01:58,697 — rl_replicas.vpg.vpg — INFO — Time: 200 2020-10-10 08:02:01,238 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 15 steps on the current episode 2020-10-10 08:02:23,314 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0141 2020-10-10 08:02:23,314 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 119 2020-10-10 08:02:23,315 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00229 2020-10-10 08:02:23,316 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -14.4 2020-10-10 08:02:23,316 — rl_replicas.vpg.vpg — INFO — Epoch: 8 2020-10-10 08:02:23,316 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 24.8 2020-10-10 08:02:23,317 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 14.2 2020-10-10 08:02:23,317 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 78 2020-10-10 08:02:23,317 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:02:23,317 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 24.7 2020-10-10 08:02:23,318 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 15 2020-10-10 08:02:23,318 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0436 2020-10-10 08:02:23,319 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 15 2020-10-10 08:02:23,319 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 14.5 2020-10-10 08:02:23,320 — rl_replicas.vpg.vpg — INFO — Total env interactions: 36000 2020-10-10 08:02:23,320 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0069 2020-10-10 08:02:23,320 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 141.044 2020-10-10 08:02:23,321 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-10 08:02:23,321 — rl_replicas.vpg.vpg — INFO — Time: 225 2020-10-10 08:02:25,763 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 43 steps on the current episode 2020-10-10 08:02:48,178 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0134 2020-10-10 08:02:48,179 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 133 2020-10-10 08:02:48,180 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.000763 2020-10-10 08:02:48,180 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 14 2020-10-10 08:02:48,181 — rl_replicas.vpg.vpg — INFO — Epoch: 9 2020-10-10 08:02:48,181 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 28.1 2020-10-10 08:02:48,182 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 15.4 2020-10-10 08:02:48,182 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 85 2020-10-10 08:02:48,183 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:02:48,183 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 28 2020-10-10 08:02:48,184 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 16.3 2020-10-10 08:02:48,185 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0424 2020-10-10 08:02:48,185 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 16.3 2020-10-10 08:02:48,186 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 15.7 2020-10-10 08:02:48,186 — rl_replicas.vpg.vpg — INFO — Total env interactions: 40000 2020-10-10 08:02:48,186 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00755 2020-10-10 08:02:48,186 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 140.222 2020-10-10 08:02:48,187 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-10 08:02:48,187 — rl_replicas.vpg.vpg — INFO — Time: 249 2020-10-10 08:02:50,819 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 14 steps on the current episode 2020-10-10 08:03:13,519 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0147 2020-10-10 08:03:13,520 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 151 2020-10-10 08:03:13,521 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00138 2020-10-10 08:03:13,521 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 18.4 2020-10-10 08:03:13,522 — rl_replicas.vpg.vpg — INFO — Epoch: 10 2020-10-10 08:03:13,522 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 30.2 2020-10-10 08:03:13,522 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 16.3 2020-10-10 08:03:13,523 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 113 2020-10-10 08:03:13,523 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:03:13,524 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 30.1 2020-10-10 08:03:13,524 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 17.2 2020-10-10 08:03:13,524 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0352 2020-10-10 08:03:13,525 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 17.2 2020-10-10 08:03:13,525 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 16.9 2020-10-10 08:03:13,525 — rl_replicas.vpg.vpg — INFO — Total env interactions: 44000 2020-10-10 08:03:13,526 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0082 2020-10-10 08:03:13,526 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 141.227 2020-10-10 08:03:13,526 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.687 2020-10-10 08:03:13,526 — rl_replicas.vpg.vpg — INFO — Time: 275 2020-10-10 08:03:16,069 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 7 steps on the current episode 2020-10-10 08:03:38,826 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0117 2020-10-10 08:03:38,827 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 146 2020-10-10 08:03:38,827 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00303 2020-10-10 08:03:38,828 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -5.51 2020-10-10 08:03:38,828 — rl_replicas.vpg.vpg — INFO — Epoch: 11 2020-10-10 08:03:38,828 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 31.1 2020-10-10 08:03:38,829 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 16.4 2020-10-10 08:03:38,829 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 85 2020-10-10 08:03:38,830 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:03:38,830 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 31 2020-10-10 08:03:38,831 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 17.6 2020-10-10 08:03:38,832 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0383 2020-10-10 08:03:38,832 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 17.6 2020-10-10 08:03:38,832 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 17.2 2020-10-10 08:03:38,833 — rl_replicas.vpg.vpg — INFO — Total env interactions: 48000 2020-10-10 08:03:38,833 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00849 2020-10-10 08:03:38,833 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 141.606 2020-10-10 08:03:38,833 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.687 2020-10-10 08:03:38,834 — rl_replicas.vpg.vpg — INFO — Time: 300 2020-10-10 08:03:41,339 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 9 steps on the current episode 2020-10-10 08:04:03,763 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0103 2020-10-10 08:04:03,763 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 123 2020-10-10 08:04:03,763 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00145 2020-10-10 08:04:03,764 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -23.2 2020-10-10 08:04:03,764 — rl_replicas.vpg.vpg — INFO — Epoch: 12 2020-10-10 08:04:03,764 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 28.9 2020-10-10 08:04:03,765 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 14.4 2020-10-10 08:04:03,765 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 86 2020-10-10 08:04:03,765 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:04:03,766 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 28.8 2020-10-10 08:04:03,767 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 16.2 2020-10-10 08:04:03,767 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0319 2020-10-10 08:04:03,768 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 16.2 2020-10-10 08:04:03,768 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 15.9 2020-10-10 08:04:03,768 — rl_replicas.vpg.vpg — INFO — Total env interactions: 52000 2020-10-10 08:04:03,769 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00863 2020-10-10 08:04:03,769 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 140.146 2020-10-10 08:04:03,769 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.688 2020-10-10 08:04:03,770 — rl_replicas.vpg.vpg — INFO — Time: 325 2020-10-10 08:04:06,189 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 36 steps on the current episode 2020-10-10 08:04:28,249 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0141 2020-10-10 08:04:28,250 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 125 2020-10-10 08:04:28,250 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00387 2020-10-10 08:04:28,251 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 2.84 2020-10-10 08:04:28,251 — rl_replicas.vpg.vpg — INFO — Epoch: 13 2020-10-10 08:04:28,252 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 31.9 2020-10-10 08:04:28,252 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 14.3 2020-10-10 08:04:28,253 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 91 2020-10-10 08:04:28,253 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:04:28,254 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 31.7 2020-10-10 08:04:28,254 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 17.2 2020-10-10 08:04:28,255 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.03 2020-10-10 08:04:28,255 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 17.2 2020-10-10 08:04:28,255 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 16.8 2020-10-10 08:04:28,255 — rl_replicas.vpg.vpg — INFO — Total env interactions: 56000 2020-10-10 08:04:28,256 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00902 2020-10-10 08:04:28,256 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 139.097 2020-10-10 08:04:28,256 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.687 2020-10-10 08:04:28,257 — rl_replicas.vpg.vpg — INFO — Time: 349 2020-10-10 08:04:30,715 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 74 steps on the current episode 2020-10-10 08:04:53,028 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0174 2020-10-10 08:04:53,029 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 170 2020-10-10 08:04:53,030 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00326 2020-10-10 08:04:53,030 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 44.6 2020-10-10 08:04:53,031 — rl_replicas.vpg.vpg — INFO — Epoch: 14 2020-10-10 08:04:53,031 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 31.1 2020-10-10 08:04:53,031 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 18.1 2020-10-10 08:04:53,032 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 92 2020-10-10 08:04:53,033 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:04:53,033 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 31 2020-10-10 08:04:53,034 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 18.2 2020-10-10 08:04:53,034 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0295 2020-10-10 08:04:53,034 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 18.2 2020-10-10 08:04:53,035 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 18 2020-10-10 08:04:53,035 — rl_replicas.vpg.vpg — INFO — Total env interactions: 60000 2020-10-10 08:04:53,035 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00958 2020-10-10 08:04:53,035 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 141.163 2020-10-10 08:04:53,036 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.687 2020-10-10 08:04:53,036 — rl_replicas.vpg.vpg — INFO — Time: 374 2020-10-10 08:04:55,510 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 48 steps on the current episode 2020-10-10 08:05:18,945 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0157 2020-10-10 08:05:18,946 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 198 2020-10-10 08:05:18,947 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.0017 2020-10-10 08:05:18,947 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 27.9 2020-10-10 08:05:18,948 — rl_replicas.vpg.vpg — INFO — Epoch: 15 2020-10-10 08:05:18,948 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 32.7 2020-10-10 08:05:18,949 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 20.2 2020-10-10 08:05:18,949 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 122 2020-10-10 08:05:18,950 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:05:18,950 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 32.5 2020-10-10 08:05:18,951 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 19.4 2020-10-10 08:05:18,951 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0204 2020-10-10 08:05:18,952 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 19.4 2020-10-10 08:05:18,952 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 19.1 2020-10-10 08:05:18,952 — rl_replicas.vpg.vpg — INFO — Total env interactions: 64000 2020-10-10 08:05:18,953 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00996 2020-10-10 08:05:18,954 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 144.717 2020-10-10 08:05:18,954 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-10 08:05:18,955 — rl_replicas.vpg.vpg — INFO — Time: 400 2020-10-10 08:05:21,431 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 23 steps on the current episode 2020-10-10 08:05:45,777 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0136 2020-10-10 08:05:45,778 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 225 2020-10-10 08:05:45,780 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00207 2020-10-10 08:05:45,780 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 27.4 2020-10-10 08:05:45,781 — rl_replicas.vpg.vpg — INFO — Epoch: 16 2020-10-10 08:05:45,781 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 35 2020-10-10 08:05:45,783 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 22.6 2020-10-10 08:05:45,786 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 133 2020-10-10 08:05:45,787 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:05:45,788 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 34.8 2020-10-10 08:05:45,789 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 20.8 2020-10-10 08:05:45,789 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.021 2020-10-10 08:05:45,790 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 20.8 2020-10-10 08:05:45,790 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 20.6 2020-10-10 08:05:45,790 — rl_replicas.vpg.vpg — INFO — Total env interactions: 68000 2020-10-10 08:05:45,791 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0102 2020-10-10 08:05:45,792 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 149.463 2020-10-10 08:05:45,793 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-10 08:05:45,793 — rl_replicas.vpg.vpg — INFO — Time: 427 2020-10-10 08:05:49,155 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 20 steps on the current episode 2020-10-10 08:06:12,165 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0156 2020-10-10 08:06:12,165 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 187 2020-10-10 08:06:12,166 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00197 2020-10-10 08:06:12,166 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -38.2 2020-10-10 08:06:12,166 — rl_replicas.vpg.vpg — INFO — Epoch: 17 2020-10-10 08:06:12,166 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 33.2 2020-10-10 08:06:12,167 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 19.3 2020-10-10 08:06:12,167 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 131 2020-10-10 08:06:12,167 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:06:12,168 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 33.1 2020-10-10 08:06:12,168 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 19.1 2020-10-10 08:06:12,168 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0518 2020-10-10 08:06:12,169 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 19.1 2020-10-10 08:06:12,169 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 18.4 2020-10-10 08:06:12,169 — rl_replicas.vpg.vpg — INFO — Total env interactions: 72000 2020-10-10 08:06:12,170 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0105 2020-10-10 08:06:12,170 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 151.562 2020-10-10 08:06:12,170 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-10 08:06:12,171 — rl_replicas.vpg.vpg — INFO — Time: 453 2020-10-10 08:06:14,570 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 21 steps on the current episode 2020-10-10 08:06:36,469 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0162 2020-10-10 08:06:36,470 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 178 2020-10-10 08:06:36,470 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00063 2020-10-10 08:06:36,471 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -9.44 2020-10-10 08:06:36,471 — rl_replicas.vpg.vpg — INFO — Epoch: 18 2020-10-10 08:06:36,471 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 37.9 2020-10-10 08:06:36,472 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 19.2 2020-10-10 08:06:36,472 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 94 2020-10-10 08:06:36,473 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:06:36,473 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 37.7 2020-10-10 08:06:36,474 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 20.5 2020-10-10 08:06:36,474 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0169 2020-10-10 08:06:36,475 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 20.5 2020-10-10 08:06:36,475 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 20.4 2020-10-10 08:06:36,475 — rl_replicas.vpg.vpg — INFO — Total env interactions: 76000 2020-10-10 08:06:36,475 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0108 2020-10-10 08:06:36,476 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 152.942 2020-10-10 08:06:36,476 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.686 2020-10-10 08:06:36,476 — rl_replicas.vpg.vpg — INFO — Time: 478 2020-10-10 08:06:39,016 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 42 steps on the current episode 2020-10-10 08:07:08,287 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0153 2020-10-10 08:07:08,288 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 165 2020-10-10 08:07:08,289 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.000909 2020-10-10 08:07:08,289 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -12.8 2020-10-10 08:07:08,290 — rl_replicas.vpg.vpg — INFO — Epoch: 19 2020-10-10 08:07:08,291 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 35.3 2020-10-10 08:07:08,292 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 17.8 2020-10-10 08:07:08,292 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 103 2020-10-10 08:07:08,293 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:07:08,294 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 35.1 2020-10-10 08:07:08,295 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 19.3 2020-10-10 08:07:08,296 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0294 2020-10-10 08:07:08,296 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 19.3 2020-10-10 08:07:08,297 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 18.8 2020-10-10 08:07:08,298 — rl_replicas.vpg.vpg — INFO — Total env interactions: 80000 2020-10-10 08:07:08,298 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.011 2020-10-10 08:07:08,299 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 153.543 2020-10-10 08:07:08,300 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.684 2020-10-10 08:07:08,300 — rl_replicas.vpg.vpg — INFO — Time: 510 2020-10-10 08:07:13,103 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 13 steps on the current episode 2020-10-10 08:07:35,836 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.018 2020-10-10 08:07:35,837 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 162 2020-10-10 08:07:35,837 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00269 2020-10-10 08:07:35,838 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -2.66 2020-10-10 08:07:35,838 — rl_replicas.vpg.vpg — INFO — Epoch: 20 2020-10-10 08:07:35,839 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 31.9 2020-10-10 08:07:35,839 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 17.3 2020-10-10 08:07:35,840 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 102 2020-10-10 08:07:35,840 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:07:35,841 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 31.7 2020-10-10 08:07:35,841 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 18 2020-10-10 08:07:35,841 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0208 2020-10-10 08:07:35,842 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 18.1 2020-10-10 08:07:35,842 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 17.8 2020-10-10 08:07:35,842 — rl_replicas.vpg.vpg — INFO — Total env interactions: 84000 2020-10-10 08:07:35,843 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0113 2020-10-10 08:07:35,843 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 153.960 2020-10-10 08:07:35,843 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.683 2020-10-10 08:07:35,844 — rl_replicas.vpg.vpg — INFO — Time: 537 2020-10-10 08:07:38,323 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 24 steps on the current episode 2020-10-10 08:08:00,405 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0177 2020-10-10 08:08:00,406 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 204 2020-10-10 08:08:00,406 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.000273 2020-10-10 08:08:00,408 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 42.2 2020-10-10 08:08:00,408 — rl_replicas.vpg.vpg — INFO — Epoch: 21 2020-10-10 08:08:00,409 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 36.2 2020-10-10 08:08:00,409 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 21.1 2020-10-10 08:08:00,410 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 102 2020-10-10 08:08:00,411 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:08:00,411 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 36 2020-10-10 08:08:00,412 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 20.6 2020-10-10 08:08:00,413 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0181 2020-10-10 08:08:00,414 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 20.7 2020-10-10 08:08:00,414 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 20.4 2020-10-10 08:08:00,415 — rl_replicas.vpg.vpg — INFO — Total env interactions: 88000 2020-10-10 08:08:00,416 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0116 2020-10-10 08:08:00,416 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 156.256 2020-10-10 08:08:00,416 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.682 2020-10-10 08:08:00,417 — rl_replicas.vpg.vpg — INFO — Time: 562 2020-10-10 08:08:02,869 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 17 steps on the current episode 2020-10-10 08:08:24,734 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0183 2020-10-10 08:08:24,735 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 194 2020-10-10 08:08:24,735 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000594 2020-10-10 08:08:24,735 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -10.7 2020-10-10 08:08:24,735 — rl_replicas.vpg.vpg — INFO — Epoch: 22 2020-10-10 08:08:24,736 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 35.6 2020-10-10 08:08:24,736 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 20 2020-10-10 08:08:24,737 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 128 2020-10-10 08:08:24,737 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:08:24,737 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 35.4 2020-10-10 08:08:24,738 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 20 2020-10-10 08:08:24,738 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0202 2020-10-10 08:08:24,738 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 20 2020-10-10 08:08:24,739 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 19.8 2020-10-10 08:08:24,739 — rl_replicas.vpg.vpg — INFO — Total env interactions: 92000 2020-10-10 08:08:24,739 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0119 2020-10-10 08:08:24,739 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 157.888 2020-10-10 08:08:24,740 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.682 2020-10-10 08:08:24,740 — rl_replicas.vpg.vpg — INFO — Time: 586 2020-10-10 08:08:27,134 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 11 steps on the current episode 2020-10-10 08:08:49,197 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0158 2020-10-10 08:08:49,198 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 215 2020-10-10 08:08:49,198 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00248 2020-10-10 08:08:49,198 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 20.8 2020-10-10 08:08:49,199 — rl_replicas.vpg.vpg — INFO — Epoch: 23 2020-10-10 08:08:49,199 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 34.7 2020-10-10 08:08:49,199 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 21.5 2020-10-10 08:08:49,200 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 120 2020-10-10 08:08:49,200 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:08:49,200 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 34.5 2020-10-10 08:08:49,200 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 20.3 2020-10-10 08:08:49,201 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0172 2020-10-10 08:08:49,201 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 20.3 2020-10-10 08:08:49,201 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 20.1 2020-10-10 08:08:49,202 — rl_replicas.vpg.vpg — INFO — Total env interactions: 96000 2020-10-10 08:08:49,202 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0121 2020-10-10 08:08:49,203 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 160.248 2020-10-10 08:08:49,203 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.682 2020-10-10 08:08:49,204 — rl_replicas.vpg.vpg — INFO — Time: 610 2020-10-10 08:08:51,614 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 28 steps on the current episode 2020-10-10 08:09:13,463 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0108 2020-10-10 08:09:13,464 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 162 2020-10-10 08:09:13,464 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00504 2020-10-10 08:09:13,464 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -52.8 2020-10-10 08:09:13,465 — rl_replicas.vpg.vpg — INFO — Epoch: 24 2020-10-10 08:09:13,465 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 35.9 2020-10-10 08:09:13,465 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 17.2 2020-10-10 08:09:13,466 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 101 2020-10-10 08:09:13,466 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:09:13,466 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 35.7 2020-10-10 08:09:13,466 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 19.2 2020-10-10 08:09:13,467 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0154 2020-10-10 08:09:13,467 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 19.2 2020-10-10 08:09:13,467 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 19 2020-10-10 08:09:13,468 — rl_replicas.vpg.vpg — INFO — Total env interactions: 100000 2020-10-10 08:09:13,468 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.012 2020-10-10 08:09:13,468 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 160.308 2020-10-10 08:09:13,468 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.683 2020-10-10 08:09:13,469 — rl_replicas.vpg.vpg — INFO — Time: 635 2020-10-10 08:09:15,864 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 21 steps on the current episode 2020-10-10 08:09:37,769 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0185 2020-10-10 08:09:37,769 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 179 2020-10-10 08:09:37,770 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.0077 2020-10-10 08:09:37,770 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 17.1 2020-10-10 08:09:37,770 — rl_replicas.vpg.vpg — INFO — Epoch: 25 2020-10-10 08:09:37,770 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 37.2 2020-10-10 08:09:37,771 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 19.1 2020-10-10 08:09:37,771 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 105 2020-10-10 08:09:37,771 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:09:37,772 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 37 2020-10-10 08:09:37,772 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 20.3 2020-10-10 08:09:37,772 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0163 2020-10-10 08:09:37,772 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 20.3 2020-10-10 08:09:37,773 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 20.1 2020-10-10 08:09:37,773 — rl_replicas.vpg.vpg — INFO — Total env interactions: 104000 2020-10-10 08:09:37,773 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0123 2020-10-10 08:09:37,773 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 161.019 2020-10-10 08:09:37,774 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.683 2020-10-10 08:09:37,774 — rl_replicas.vpg.vpg — INFO — Time: 659 2020-10-10 08:09:40,279 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 34 steps on the current episode 2020-10-10 08:10:02,173 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0191 2020-10-10 08:10:02,173 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 175 2020-10-10 08:10:02,174 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000599 2020-10-10 08:10:02,175 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -3.4 2020-10-10 08:10:02,175 — rl_replicas.vpg.vpg — INFO — Epoch: 26 2020-10-10 08:10:02,175 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 36.2 2020-10-10 08:10:02,175 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 18.9 2020-10-10 08:10:02,176 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 104 2020-10-10 08:10:02,176 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:10:02,176 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 36 2020-10-10 08:10:02,177 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 19.9 2020-10-10 08:10:02,177 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0152 2020-10-10 08:10:02,178 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 19.9 2020-10-10 08:10:02,178 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 19.8 2020-10-10 08:10:02,178 — rl_replicas.vpg.vpg — INFO — Total env interactions: 108000 2020-10-10 08:10:02,179 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0125 2020-10-10 08:10:02,179 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 161.552 2020-10-10 08:10:02,179 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-10 08:10:02,179 — rl_replicas.vpg.vpg — INFO — Time: 683 2020-10-10 08:10:04,624 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 2 steps on the current episode 2020-10-10 08:10:27,506 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0117 2020-10-10 08:10:27,507 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 167 2020-10-10 08:10:27,507 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00741 2020-10-10 08:10:27,508 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -8.58 2020-10-10 08:10:27,508 — rl_replicas.vpg.vpg — INFO — Epoch: 27 2020-10-10 08:10:27,508 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 35 2020-10-10 08:10:27,509 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 17.8 2020-10-10 08:10:27,509 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 95 2020-10-10 08:10:27,509 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:10:27,509 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 34.8 2020-10-10 08:10:27,510 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 19.1 2020-10-10 08:10:27,510 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.018 2020-10-10 08:10:27,510 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 19.1 2020-10-10 08:10:27,511 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 19 2020-10-10 08:10:27,511 — rl_replicas.vpg.vpg — INFO — Total env interactions: 112000 2020-10-10 08:10:27,511 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0125 2020-10-10 08:10:27,511 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 161.740 2020-10-10 08:10:27,511 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-10 08:10:27,512 — rl_replicas.vpg.vpg — INFO — Time: 709 2020-10-10 08:10:29,996 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 16 steps on the current episode 2020-10-10 08:10:52,103 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.015 2020-10-10 08:10:52,104 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 240 2020-10-10 08:10:52,104 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00334 2020-10-10 08:10:52,104 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 73.6 2020-10-10 08:10:52,105 — rl_replicas.vpg.vpg — INFO — Epoch: 28 2020-10-10 08:10:52,105 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 39.8 2020-10-10 08:10:52,105 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 23.7 2020-10-10 08:10:52,106 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 152 2020-10-10 08:10:52,106 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:10:52,106 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 39.6 2020-10-10 08:10:52,106 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 22.3 2020-10-10 08:10:52,107 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0113 2020-10-10 08:10:52,107 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 22.3 2020-10-10 08:10:52,108 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 22.2 2020-10-10 08:10:52,108 — rl_replicas.vpg.vpg — INFO — Total env interactions: 116000 2020-10-10 08:10:52,108 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0126 2020-10-10 08:10:52,108 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 164.455 2020-10-10 08:10:52,109 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-10 08:10:52,109 — rl_replicas.vpg.vpg — INFO — Time: 733 2020-10-10 08:10:54,507 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 47 steps on the current episode 2020-10-10 08:11:19,724 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0169 2020-10-10 08:11:19,724 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 253 2020-10-10 08:11:19,725 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00193 2020-10-10 08:11:19,725 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 12.7 2020-10-10 08:11:19,726 — rl_replicas.vpg.vpg — INFO — Epoch: 29 2020-10-10 08:11:19,726 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 42.3 2020-10-10 08:11:19,727 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 25.2 2020-10-10 08:11:19,727 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 120 2020-10-10 08:11:19,728 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:11:19,729 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 42.1 2020-10-10 08:11:19,729 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 23.6 2020-10-10 08:11:19,730 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0134 2020-10-10 08:11:19,730 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 23.6 2020-10-10 08:11:19,731 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 23.5 2020-10-10 08:11:19,731 — rl_replicas.vpg.vpg — INFO — Total env interactions: 120000 2020-10-10 08:11:19,731 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0127 2020-10-10 08:11:19,732 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 167.413 2020-10-10 08:11:19,733 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-10 08:11:19,733 — rl_replicas.vpg.vpg — INFO — Time: 761 2020-10-10 08:11:22,490 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 30 steps on the current episode 2020-10-10 08:11:44,576 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0126 2020-10-10 08:11:44,577 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 273 2020-10-10 08:11:44,577 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00434 2020-10-10 08:11:44,578 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 19.6 2020-10-10 08:11:44,579 — rl_replicas.vpg.vpg — INFO — Epoch: 30 2020-10-10 08:11:44,579 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 46.8 2020-10-10 08:11:44,580 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 26.9 2020-10-10 08:11:44,580 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 136 2020-10-10 08:11:44,580 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 13 2020-10-10 08:11:44,581 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 46.5 2020-10-10 08:11:44,581 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 25.2 2020-10-10 08:11:44,582 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0113 2020-10-10 08:11:44,582 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 25.2 2020-10-10 08:11:44,582 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 25.1 2020-10-10 08:11:44,583 — rl_replicas.vpg.vpg — INFO — Total env interactions: 124000 2020-10-10 08:11:44,583 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0127 2020-10-10 08:11:44,583 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 170.813 2020-10-10 08:11:44,583 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.678 2020-10-10 08:11:44,584 — rl_replicas.vpg.vpg — INFO — Time: 786 2020-10-10 08:11:47,072 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 33 steps on the current episode 2020-10-10 08:12:09,222 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00712 2020-10-10 08:12:09,222 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 192 2020-10-10 08:12:09,223 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00549 2020-10-10 08:12:09,223 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -80.7 2020-10-10 08:12:09,223 — rl_replicas.vpg.vpg — INFO — Epoch: 31 2020-10-10 08:12:09,224 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 41.5 2020-10-10 08:12:09,225 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 19.9 2020-10-10 08:12:09,225 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 105 2020-10-10 08:12:09,225 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:12:09,226 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 41.2 2020-10-10 08:12:09,226 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 21.7 2020-10-10 08:12:09,226 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0129 2020-10-10 08:12:09,227 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 21.7 2020-10-10 08:12:09,227 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 21.6 2020-10-10 08:12:09,227 — rl_replicas.vpg.vpg — INFO — Total env interactions: 128000 2020-10-10 08:12:09,228 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0126 2020-10-10 08:12:09,228 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 171.478 2020-10-10 08:12:09,228 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.678 2020-10-10 08:12:09,229 — rl_replicas.vpg.vpg — INFO — Time: 810 2020-10-10 08:12:11,628 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 27 steps on the current episode 2020-10-10 08:12:33,626 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00831 2020-10-10 08:12:33,626 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 222 2020-10-10 08:12:33,627 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00119 2020-10-10 08:12:33,627 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 29.5 2020-10-10 08:12:33,627 — rl_replicas.vpg.vpg — INFO — Epoch: 32 2020-10-10 08:12:33,628 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 42.3 2020-10-10 08:12:33,628 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 22.1 2020-10-10 08:12:33,628 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 109 2020-10-10 08:12:33,629 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 13 2020-10-10 08:12:33,629 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 42.1 2020-10-10 08:12:33,630 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 22.6 2020-10-10 08:12:33,630 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0135 2020-10-10 08:12:33,631 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 22.6 2020-10-10 08:12:33,631 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 22.4 2020-10-10 08:12:33,632 — rl_replicas.vpg.vpg — INFO — Total env interactions: 132000 2020-10-10 08:12:33,632 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0124 2020-10-10 08:12:33,632 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 172.998 2020-10-10 08:12:33,633 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.677 2020-10-10 08:12:33,633 — rl_replicas.vpg.vpg — INFO — Time: 835 2020-10-10 08:12:36,109 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 16 steps on the current episode 2020-10-10 08:12:58,417 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00995 2020-10-10 08:12:58,418 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 243 2020-10-10 08:12:58,418 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00164 2020-10-10 08:12:58,419 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 21.5 2020-10-10 08:12:58,420 — rl_replicas.vpg.vpg — INFO — Epoch: 33 2020-10-10 08:12:58,420 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 41.9 2020-10-10 08:12:58,421 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 23.4 2020-10-10 08:12:58,421 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 148 2020-10-10 08:12:58,422 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 14 2020-10-10 08:12:58,423 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 41.7 2020-10-10 08:12:58,424 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 22.7 2020-10-10 08:12:58,425 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0114 2020-10-10 08:12:58,425 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 22.7 2020-10-10 08:12:58,426 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 22.6 2020-10-10 08:12:58,426 — rl_replicas.vpg.vpg — INFO — Total env interactions: 136000 2020-10-10 08:12:58,427 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0124 2020-10-10 08:12:58,428 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 175.060 2020-10-10 08:12:58,428 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.677 2020-10-10 08:12:58,429 — rl_replicas.vpg.vpg — INFO — Time: 860 2020-10-10 08:13:22,845 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0105 2020-10-10 08:13:22,846 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 310 2020-10-10 08:13:22,846 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000502 2020-10-10 08:13:22,847 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 66.7 2020-10-10 08:13:22,847 — rl_replicas.vpg.vpg — INFO — Epoch: 34 2020-10-10 08:13:22,847 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 45.2 2020-10-10 08:13:22,848 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 29 2020-10-10 08:13:22,848 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 171 2020-10-10 08:13:22,848 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:13:22,849 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 44.9 2020-10-10 08:13:22,849 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 25.3 2020-10-10 08:13:22,850 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0143 2020-10-10 08:13:22,850 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 25.3 2020-10-10 08:13:22,850 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 25.2 2020-10-10 08:13:22,850 — rl_replicas.vpg.vpg — INFO — Total env interactions: 140000 2020-10-10 08:13:22,851 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0123 2020-10-10 08:13:22,851 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 178.911 2020-10-10 08:13:22,851 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.675 2020-10-10 08:13:22,852 — rl_replicas.vpg.vpg — INFO — Time: 884 2020-10-10 08:13:25,234 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 29 steps on the current episode 2020-10-10 08:13:47,274 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0135 2020-10-10 08:13:47,275 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 214 2020-10-10 08:13:47,275 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.0031 2020-10-10 08:13:47,275 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -95.4 2020-10-10 08:13:47,276 — rl_replicas.vpg.vpg — INFO — Epoch: 35 2020-10-10 08:13:47,276 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 44.7 2020-10-10 08:13:47,276 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 21.5 2020-10-10 08:13:47,276 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 115 2020-10-10 08:13:47,277 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:13:47,277 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 44.4 2020-10-10 08:13:47,277 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 23.1 2020-10-10 08:13:47,278 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0103 2020-10-10 08:13:47,278 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 23.1 2020-10-10 08:13:47,278 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 23 2020-10-10 08:13:47,278 — rl_replicas.vpg.vpg — INFO — Total env interactions: 144000 2020-10-10 08:13:47,279 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0123 2020-10-10 08:13:47,279 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 179.899 2020-10-10 08:13:47,279 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.673 2020-10-10 08:13:47,279 — rl_replicas.vpg.vpg — INFO — Time: 909 2020-10-10 08:13:49,691 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 11 steps on the current episode 2020-10-10 08:14:11,677 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0139 2020-10-10 08:14:11,677 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 306 2020-10-10 08:14:11,678 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000354 2020-10-10 08:14:11,678 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 91.6 2020-10-10 08:14:11,679 — rl_replicas.vpg.vpg — INFO — Epoch: 36 2020-10-10 08:14:11,679 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 44.2 2020-10-10 08:14:11,679 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 28 2020-10-10 08:14:11,680 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 200 2020-10-10 08:14:11,680 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:14:11,680 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 44 2020-10-10 08:14:11,680 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 24.4 2020-10-10 08:14:11,681 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0102 2020-10-10 08:14:11,681 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 24.4 2020-10-10 08:14:11,681 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 24.4 2020-10-10 08:14:11,681 — rl_replicas.vpg.vpg — INFO — Total env interactions: 148000 2020-10-10 08:14:11,682 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0124 2020-10-10 08:14:11,682 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 183.308 2020-10-10 08:14:11,682 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.673 2020-10-10 08:14:11,682 — rl_replicas.vpg.vpg — INFO — Time: 933 2020-10-10 08:14:14,114 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 35 steps on the current episode 2020-10-10 08:14:36,035 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0094 2020-10-10 08:14:36,036 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 234 2020-10-10 08:14:36,036 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.0045 2020-10-10 08:14:36,037 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -71.7 2020-10-10 08:14:36,037 — rl_replicas.vpg.vpg — INFO — Epoch: 37 2020-10-10 08:14:36,038 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 42.8 2020-10-10 08:14:36,038 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 23.1 2020-10-10 08:14:36,038 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 120 2020-10-10 08:14:36,039 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 13 2020-10-10 08:14:36,039 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 42.6 2020-10-10 08:14:36,039 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 23 2020-10-10 08:14:36,040 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0105 2020-10-10 08:14:36,040 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 23 2020-10-10 08:14:36,040 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 22.9 2020-10-10 08:14:36,041 — rl_replicas.vpg.vpg — INFO — Total env interactions: 152000 2020-10-10 08:14:36,041 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0123 2020-10-10 08:14:36,041 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 184.651 2020-10-10 08:14:36,042 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.672 2020-10-10 08:14:36,042 — rl_replicas.vpg.vpg — INFO — Time: 957 2020-10-10 08:14:38,468 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 60 steps on the current episode 2020-10-10 08:15:00,530 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00907 2020-10-10 08:15:00,530 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 266 2020-10-10 08:15:00,531 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.000331 2020-10-10 08:15:00,531 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 32 2020-10-10 08:15:00,532 — rl_replicas.vpg.vpg — INFO — Epoch: 38 2020-10-10 08:15:00,532 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 47.9 2020-10-10 08:15:00,533 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 25.4 2020-10-10 08:15:00,533 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 151 2020-10-10 08:15:00,533 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 15 2020-10-10 08:15:00,534 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 47.6 2020-10-10 08:15:00,534 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 24.9 2020-10-10 08:15:00,535 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.00764 2020-10-10 08:15:00,535 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 24.9 2020-10-10 08:15:00,535 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 24.8 2020-10-10 08:15:00,536 — rl_replicas.vpg.vpg — INFO — Total env interactions: 156000 2020-10-10 08:15:00,536 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0122 2020-10-10 08:15:00,536 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 186.746 2020-10-10 08:15:00,536 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.672 2020-10-10 08:15:00,537 — rl_replicas.vpg.vpg — INFO — Time: 982 2020-10-10 08:15:02,995 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 54 steps on the current episode 2020-10-10 08:15:25,020 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0114 2020-10-10 08:15:25,021 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 304 2020-10-10 08:15:25,022 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00232 2020-10-10 08:15:25,022 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 38 2020-10-10 08:15:25,023 — rl_replicas.vpg.vpg — INFO — Epoch: 39 2020-10-10 08:15:25,023 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 51.6 2020-10-10 08:15:25,023 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 28.4 2020-10-10 08:15:25,024 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 164 2020-10-10 08:15:25,024 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 17 2020-10-10 08:15:25,025 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 51.3 2020-10-10 08:15:25,025 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 26.6 2020-10-10 08:15:25,026 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0106 2020-10-10 08:15:25,027 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 26.6 2020-10-10 08:15:25,027 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 26.5 2020-10-10 08:15:25,027 — rl_replicas.vpg.vpg — INFO — Total env interactions: 160000 2020-10-10 08:15:25,027 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0122 2020-10-10 08:15:25,028 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 189.687 2020-10-10 08:15:25,028 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.668 2020-10-10 08:15:25,029 — rl_replicas.vpg.vpg — INFO — Time: 1.01e+03 2020-10-10 08:15:27,380 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 20 steps on the current episode 2020-10-10 08:15:50,580 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00895 2020-10-10 08:15:50,581 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 359 2020-10-10 08:15:50,582 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00244 2020-10-10 08:15:50,582 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 54.5 2020-10-10 08:15:50,582 — rl_replicas.vpg.vpg — INFO — Epoch: 40 2020-10-10 08:15:50,583 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 52.3 2020-10-10 08:15:50,583 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 32.9 2020-10-10 08:15:50,584 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 200 2020-10-10 08:15:50,584 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:15:50,585 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 51.9 2020-10-10 08:15:50,585 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 27.9 2020-10-10 08:15:50,586 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.00749 2020-10-10 08:15:50,586 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 27.9 2020-10-10 08:15:50,587 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 27.8 2020-10-10 08:15:50,587 — rl_replicas.vpg.vpg — INFO — Total env interactions: 164000 2020-10-10 08:15:50,587 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0121 2020-10-10 08:15:50,588 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 193.813 2020-10-10 08:15:50,588 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.667 2020-10-10 08:15:50,588 — rl_replicas.vpg.vpg — INFO — Time: 1.03e+03 2020-10-10 08:15:52,987 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 16 steps on the current episode 2020-10-10 08:16:14,919 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0131 2020-10-10 08:16:14,920 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 272 2020-10-10 08:16:14,920 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.0042 2020-10-10 08:16:14,921 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -86.8 2020-10-10 08:16:14,921 — rl_replicas.vpg.vpg — INFO — Epoch: 41 2020-10-10 08:16:14,921 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 43.8 2020-10-10 08:16:14,921 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 25.7 2020-10-10 08:16:14,922 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 167 2020-10-10 08:16:14,922 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 14 2020-10-10 08:16:14,923 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 43.5 2020-10-10 08:16:14,923 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 23.9 2020-10-10 08:16:14,924 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0149 2020-10-10 08:16:14,924 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 23.9 2020-10-10 08:16:14,924 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 23.8 2020-10-10 08:16:14,924 — rl_replicas.vpg.vpg — INFO — Total env interactions: 168000 2020-10-10 08:16:14,925 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0121 2020-10-10 08:16:14,925 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 195.677 2020-10-10 08:16:14,926 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.667 2020-10-10 08:16:14,926 — rl_replicas.vpg.vpg — INFO — Time: 1.06e+03 2020-10-10 08:16:17,318 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 63 steps on the current episode 2020-10-10 08:16:41,448 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00868 2020-10-10 08:16:41,449 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 268 2020-10-10 08:16:41,450 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00447 2020-10-10 08:16:41,450 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -3.76 2020-10-10 08:16:41,451 — rl_replicas.vpg.vpg — INFO — Epoch: 42 2020-10-10 08:16:41,452 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 47.3 2020-10-10 08:16:41,452 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 26.2 2020-10-10 08:16:41,453 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 128 2020-10-10 08:16:41,454 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:16:41,455 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 47.1 2020-10-10 08:16:41,456 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 25.1 2020-10-10 08:16:41,457 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0113 2020-10-10 08:16:41,458 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 25.1 2020-10-10 08:16:41,459 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 25 2020-10-10 08:16:41,459 — rl_replicas.vpg.vpg — INFO — Total env interactions: 172000 2020-10-10 08:16:41,460 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0121 2020-10-10 08:16:41,460 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 197.366 2020-10-10 08:16:41,461 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.663 2020-10-10 08:16:41,462 — rl_replicas.vpg.vpg — INFO — Time: 1.08e+03 2020-10-10 08:16:44,872 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 39 steps on the current episode 2020-10-10 08:17:07,448 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0105 2020-10-10 08:17:07,449 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 273 2020-10-10 08:17:07,449 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00181 2020-10-10 08:17:07,450 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 5.17 2020-10-10 08:17:07,450 — rl_replicas.vpg.vpg — INFO — Epoch: 43 2020-10-10 08:17:07,450 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 49.1 2020-10-10 08:17:07,451 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 26.3 2020-10-10 08:17:07,451 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 174 2020-10-10 08:17:07,451 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:17:07,451 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 48.8 2020-10-10 08:17:07,452 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 25.4 2020-10-10 08:17:07,452 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.00992 2020-10-10 08:17:07,452 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 25.5 2020-10-10 08:17:07,453 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 25.4 2020-10-10 08:17:07,453 — rl_replicas.vpg.vpg — INFO — Total env interactions: 176000 2020-10-10 08:17:07,454 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.012 2020-10-10 08:17:07,454 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 199.096 2020-10-10 08:17:07,454 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.662 2020-10-10 08:17:07,454 — rl_replicas.vpg.vpg — INFO — Time: 1.11e+03 2020-10-10 08:17:09,900 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 73 steps on the current episode 2020-10-10 08:17:32,023 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0121 2020-10-10 08:17:32,024 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 290 2020-10-10 08:17:32,024 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00163 2020-10-10 08:17:32,025 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 16.1 2020-10-10 08:17:32,025 — rl_replicas.vpg.vpg — INFO — Epoch: 44 2020-10-10 08:17:32,025 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 51.6 2020-10-10 08:17:32,026 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 28.1 2020-10-10 08:17:32,026 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 113 2020-10-10 08:17:32,027 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 16 2020-10-10 08:17:32,027 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 51.3 2020-10-10 08:17:32,027 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 26.8 2020-10-10 08:17:32,028 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.00927 2020-10-10 08:17:32,028 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 26.8 2020-10-10 08:17:32,029 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 26.7 2020-10-10 08:17:32,029 — rl_replicas.vpg.vpg — INFO — Total env interactions: 180000 2020-10-10 08:17:32,029 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.012 2020-10-10 08:17:32,029 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 201.106 2020-10-10 08:17:32,030 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.663 2020-10-10 08:17:32,030 — rl_replicas.vpg.vpg — INFO — Time: 1.13e+03 2020-10-10 08:17:34,514 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 9 steps on the current episode 2020-10-10 08:17:56,734 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0128 2020-10-10 08:17:56,735 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 216 2020-10-10 08:17:56,735 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000679 2020-10-10 08:17:56,736 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -73.9 2020-10-10 08:17:56,736 — rl_replicas.vpg.vpg — INFO — Epoch: 45 2020-10-10 08:17:56,737 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 45.2 2020-10-10 08:17:56,737 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 22.2 2020-10-10 08:17:56,738 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 97 2020-10-10 08:17:56,738 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:17:56,738 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 44.9 2020-10-10 08:17:56,739 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 23.5 2020-10-10 08:17:56,739 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.0159 2020-10-10 08:17:56,740 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 23.6 2020-10-10 08:17:56,740 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 23.3 2020-10-10 08:17:56,740 — rl_replicas.vpg.vpg — INFO — Total env interactions: 184000 2020-10-10 08:17:56,741 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.012 2020-10-10 08:17:56,741 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 201.422 2020-10-10 08:17:56,741 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.664 2020-10-10 08:17:56,741 — rl_replicas.vpg.vpg — INFO — Time: 1.16e+03 2020-10-10 08:17:59,187 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 49 steps on the current episode 2020-10-10 08:18:21,147 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0139 2020-10-10 08:18:21,148 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 288 2020-10-10 08:18:21,148 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00106 2020-10-10 08:18:21,149 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 72.5 2020-10-10 08:18:21,149 — rl_replicas.vpg.vpg — INFO — Epoch: 46 2020-10-10 08:18:21,149 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 47.9 2020-10-10 08:18:21,150 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 26.9 2020-10-10 08:18:21,150 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 200 2020-10-10 08:18:21,150 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 14 2020-10-10 08:18:21,151 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 47.6 2020-10-10 08:18:21,151 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 25 2020-10-10 08:18:21,152 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.00948 2020-10-10 08:18:21,152 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 25.1 2020-10-10 08:18:21,152 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 24.9 2020-10-10 08:18:21,153 — rl_replicas.vpg.vpg — INFO — Total env interactions: 188000 2020-10-10 08:18:21,153 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0121 2020-10-10 08:18:21,153 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 203.267 2020-10-10 08:18:21,154 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.663 2020-10-10 08:18:21,154 — rl_replicas.vpg.vpg — INFO — Time: 1.18e+03 2020-10-10 08:18:23,591 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 19 steps on the current episode 2020-10-10 08:18:45,695 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0116 2020-10-10 08:18:45,695 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 336 2020-10-10 08:18:45,696 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00225 2020-10-10 08:18:45,696 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 48.1 2020-10-10 08:18:45,696 — rl_replicas.vpg.vpg — INFO — Epoch: 47 2020-10-10 08:18:45,697 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 50.9 2020-10-10 08:18:45,697 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 31.1 2020-10-10 08:18:45,698 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 200 2020-10-10 08:18:45,698 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:18:45,698 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 50.6 2020-10-10 08:18:45,699 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 27.1 2020-10-10 08:18:45,699 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.00738 2020-10-10 08:18:45,708 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 27.1 2020-10-10 08:18:45,708 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 27 2020-10-10 08:18:45,709 — rl_replicas.vpg.vpg — INFO — Total env interactions: 192000 2020-10-10 08:18:45,709 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0121 2020-10-10 08:18:45,709 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 206.036 2020-10-10 08:18:45,710 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.663 2020-10-10 08:18:45,710 — rl_replicas.vpg.vpg — INFO — Time: 1.21e+03 2020-10-10 08:18:48,085 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 25 steps on the current episode 2020-10-10 08:19:09,987 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0148 2020-10-10 08:19:09,988 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 411 2020-10-10 08:19:09,988 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00316 2020-10-10 08:19:09,989 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 74.6 2020-10-10 08:19:09,989 — rl_replicas.vpg.vpg — INFO — Epoch: 48 2020-10-10 08:19:09,989 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 53.7 2020-10-10 08:19:09,990 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 36.7 2020-10-10 08:19:09,990 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 200 2020-10-10 08:19:09,990 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 16 2020-10-10 08:19:09,990 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 53.3 2020-10-10 08:19:09,991 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 29.2 2020-10-10 08:19:09,991 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.00786 2020-10-10 08:19:09,991 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 29.2 2020-10-10 08:19:09,992 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 29.1 2020-10-10 08:19:09,992 — rl_replicas.vpg.vpg — INFO — Total env interactions: 196000 2020-10-10 08:19:09,992 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0121 2020-10-10 08:19:09,993 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 210.215 2020-10-10 08:19:09,993 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.659 2020-10-10 08:19:09,993 — rl_replicas.vpg.vpg — INFO — Time: 1.23e+03 2020-10-10 08:19:12,421 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 45 steps on the current episode 2020-10-10 08:19:12,422 — rl_replicas.vpg.vpg — WARNING — Saving model is not implemented 2020-10-10 08:19:34,501 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00773 2020-10-10 08:19:34,502 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 356 2020-10-10 08:19:34,502 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00705 2020-10-10 08:19:34,502 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -54.4 2020-10-10 08:19:34,503 — rl_replicas.vpg.vpg — INFO — Epoch: 49 2020-10-10 08:19:34,503 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 58.4 2020-10-10 08:19:34,504 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 32.7 2020-10-10 08:19:34,504 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 194 2020-10-10 08:19:34,505 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 13 2020-10-10 08:19:34,505 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 58 2020-10-10 08:19:34,506 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 29.3 2020-10-10 08:19:34,506 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.00535 2020-10-10 08:19:34,507 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 29.3 2020-10-10 08:19:34,507 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 29.2 2020-10-10 08:19:34,507 — rl_replicas.vpg.vpg — INFO — Total env interactions: 200000 2020-10-10 08:19:34,507 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.012 2020-10-10 08:19:34,508 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 213.139 2020-10-10 08:19:34,508 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.659 2020-10-10 08:19:34,508 — rl_replicas.vpg.vpg — INFO — Time: 1.26e+03 39871027 function calls (37032485 primitive calls) in 1257.151 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 4050 769.895 0.190 769.895 0.190 {method 'run_backward' of 'torch._C._EngineBase' objects} 4000 309.807 0.077 338.082 0.085 vpg.py:249(_compute_value_function_loss) 4150 28.285 0.007 28.285 0.007 {method 'mean' of 'torch._C._TensorBase' objects} 1200150 20.617 0.000 20.617 0.000 {method 'matmul' of 'torch._C._TensorBase' objects} 200050 14.523 0.000 14.523 0.000 {method 'logsumexp' of 'torch._C._TensorBase' objects} 808200 14.112 0.000 14.112 0.000 {built-in method tanh} 1212300 10.985 0.000 43.704 0.000 functional.py:1355(linear) 3232800/404100 10.660 0.000 103.849 0.000 module.py:531(__call__) 12150 6.166 0.001 6.166 0.001 {built-in method addmm} 1 6.123 6.123 1255.733 1255.733 vpg.py:59(learn) 1212300 5.586 0.000 5.586 0.000 {method 't' of 'torch._C._TensorBase' objects} 200000 4.961 0.000 8.136 0.000 cartpole.py:91(step) 200050 3.877 0.000 19.332 0.000 categorical.py:44(__init__) 404100 3.849 0.000 77.768 0.000 container.py:90(forward) 1212300 3.501 0.000 48.359 0.000 linear.py:86(forward) ```

yamatokataoka commented 4 years ago

the avarage value loss was increasing from 97.665 to 129.709.

parameters

epochs (number of updates policy): 50
steps_per_epoch: 4000
n_value_gradients: 80
gamma: 0.99
gae_lambda: 0.97
value function learning_rate: 1e-3
policy learning_rate: 3e-4

value loss and peroformance of the multiple updates of value function with squeezing tensors

``` 2020-10-10 08:33:17,256 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 10 steps on the current episode 2020-10-10 08:33:18,401 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00479 2020-10-10 08:33:18,401 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 118 2020-10-10 08:33:18,402 — rl_replicas.vpg.vpg — INFO — Epoch: 0 2020-10-10 08:33:18,402 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 23.4 2020-10-10 08:33:18,403 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 11.8 2020-10-10 08:33:18,403 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 66 2020-10-10 08:33:18,403 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:33:18,404 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 23.4 2020-10-10 08:33:18,404 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 8.33 2020-10-10 08:33:18,405 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 0.161 2020-10-10 08:33:18,405 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 8.66 2020-10-10 08:33:18,405 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 7.36 2020-10-10 08:33:18,406 — rl_replicas.vpg.vpg — INFO — Total env interactions: 4000 2020-10-10 08:33:18,406 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00479 2020-10-10 08:33:18,407 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 117.805 2020-10-10 08:33:18,407 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.693 2020-10-10 08:33:18,408 — rl_replicas.vpg.vpg — INFO — Time: 3.71 2020-10-10 08:33:20,956 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 9 steps on the current episode 2020-10-10 08:33:21,695 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00716 2020-10-10 08:33:21,696 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 77.5 2020-10-10 08:33:21,696 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00237 2020-10-10 08:33:21,696 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -40.3 2020-10-10 08:33:21,697 — rl_replicas.vpg.vpg — INFO — Epoch: 1 2020-10-10 08:33:21,697 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 23.4 2020-10-10 08:33:21,697 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 11.2 2020-10-10 08:33:21,698 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 62 2020-10-10 08:33:21,698 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:33:21,698 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 23.4 2020-10-10 08:33:21,699 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 11.7 2020-10-10 08:33:21,699 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 2.52 2020-10-10 08:33:21,700 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 13.3 2020-10-10 08:33:21,700 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -6.2 2020-10-10 08:33:21,700 — rl_replicas.vpg.vpg — INFO — Total env interactions: 8000 2020-10-10 08:33:21,700 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00598 2020-10-10 08:33:21,701 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 97.665 2020-10-10 08:33:21,701 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.692 2020-10-10 08:33:21,701 — rl_replicas.vpg.vpg — INFO — Time: 7 2020-10-10 08:33:24,145 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 19 steps on the current episode 2020-10-10 08:33:24,850 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.00592 2020-10-10 08:33:24,850 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 112 2020-10-10 08:33:24,851 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00124 2020-10-10 08:33:24,851 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 34.3 2020-10-10 08:33:24,851 — rl_replicas.vpg.vpg — INFO — Epoch: 2 2020-10-10 08:33:24,852 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 26.2 2020-10-10 08:33:24,852 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 14.7 2020-10-10 08:33:24,852 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 93 2020-10-10 08:33:24,853 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 8 2020-10-10 08:33:24,853 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 26.1 2020-10-10 08:33:24,853 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 14.4 2020-10-10 08:33:24,854 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 3.72 2020-10-10 08:33:24,854 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 16.4 2020-10-10 08:33:24,854 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -10.4 2020-10-10 08:33:24,854 — rl_replicas.vpg.vpg — INFO — Total env interactions: 12000 2020-10-10 08:33:24,855 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00596 2020-10-10 08:33:24,855 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 102.389 2020-10-10 08:33:24,855 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.69 2020-10-10 08:33:24,855 — rl_replicas.vpg.vpg — INFO — Time: 10.2 2020-10-10 08:33:27,307 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 42 steps on the current episode 2020-10-10 08:33:27,986 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0114 2020-10-10 08:33:27,986 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 77.6 2020-10-10 08:33:27,987 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00545 2020-10-10 08:33:27,987 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -34.2 2020-10-10 08:33:27,988 — rl_replicas.vpg.vpg — INFO — Epoch: 3 2020-10-10 08:33:27,988 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 24.4 2020-10-10 08:33:27,989 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 11.6 2020-10-10 08:33:27,989 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 78 2020-10-10 08:33:27,989 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:33:27,990 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 24.4 2020-10-10 08:33:27,990 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 13.9 2020-10-10 08:33:27,991 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 3.63 2020-10-10 08:33:27,991 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 15.9 2020-10-10 08:33:27,992 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -6.72 2020-10-10 08:33:27,993 — rl_replicas.vpg.vpg — INFO — Total env interactions: 16000 2020-10-10 08:33:27,993 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00731 2020-10-10 08:33:27,994 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 96.204 2020-10-10 08:33:27,995 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.689 2020-10-10 08:33:27,995 — rl_replicas.vpg.vpg — INFO — Time: 13.3 2020-10-10 08:33:30,506 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 7 steps on the current episode 2020-10-10 08:33:31,248 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.012 2020-10-10 08:33:31,248 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 124 2020-10-10 08:33:31,249 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000626 2020-10-10 08:33:31,249 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 46.6 2020-10-10 08:33:31,249 — rl_replicas.vpg.vpg — INFO — Epoch: 4 2020-10-10 08:33:31,249 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 26.6 2020-10-10 08:33:31,250 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 15.9 2020-10-10 08:33:31,251 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 105 2020-10-10 08:33:31,251 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 8 2020-10-10 08:33:31,252 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 26.5 2020-10-10 08:33:31,252 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 15.6 2020-10-10 08:33:31,252 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 4.38 2020-10-10 08:33:31,253 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 17.9 2020-10-10 08:33:31,253 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -6.6 2020-10-10 08:33:31,253 — rl_replicas.vpg.vpg — INFO — Total env interactions: 20000 2020-10-10 08:33:31,253 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00825 2020-10-10 08:33:31,254 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 101.806 2020-10-10 08:33:31,254 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.69 2020-10-10 08:33:31,254 — rl_replicas.vpg.vpg — INFO — Time: 16.6 2020-10-10 08:33:33,736 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 8 steps on the current episode 2020-10-10 08:33:34,474 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0129 2020-10-10 08:33:34,474 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 88.9 2020-10-10 08:33:34,475 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000882 2020-10-10 08:33:34,475 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -35.3 2020-10-10 08:33:34,476 — rl_replicas.vpg.vpg — INFO — Epoch: 5 2020-10-10 08:33:34,476 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 24.9 2020-10-10 08:33:34,476 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 12.8 2020-10-10 08:33:34,477 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 85 2020-10-10 08:33:34,477 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:33:34,477 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 24.8 2020-10-10 08:33:34,478 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 14.6 2020-10-10 08:33:34,478 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 3.84 2020-10-10 08:33:34,478 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 16.7 2020-10-10 08:33:34,479 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -3.41 2020-10-10 08:33:34,479 — rl_replicas.vpg.vpg — INFO — Total env interactions: 24000 2020-10-10 08:33:34,479 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.00902 2020-10-10 08:33:34,480 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 99.652 2020-10-10 08:33:34,480 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.69 2020-10-10 08:33:34,480 — rl_replicas.vpg.vpg — INFO — Time: 19.8 2020-10-10 08:33:37,790 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0131 2020-10-10 08:33:37,791 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 108 2020-10-10 08:33:37,791 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000239 2020-10-10 08:33:37,792 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 19 2020-10-10 08:33:37,792 — rl_replicas.vpg.vpg — INFO — Epoch: 6 2020-10-10 08:33:37,792 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 28.3 2020-10-10 08:33:37,793 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 15 2020-10-10 08:33:37,793 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 94 2020-10-10 08:33:37,794 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:33:37,794 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 28.2 2020-10-10 08:33:37,794 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 16 2020-10-10 08:33:37,795 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 4.39 2020-10-10 08:33:37,795 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 18.3 2020-10-10 08:33:37,795 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -6.43 2020-10-10 08:33:37,796 — rl_replicas.vpg.vpg — INFO — Total env interactions: 28000 2020-10-10 08:33:37,796 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0096 2020-10-10 08:33:37,796 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 100.828 2020-10-10 08:33:37,797 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.689 2020-10-10 08:33:37,798 — rl_replicas.vpg.vpg — INFO — Time: 23.1 2020-10-10 08:33:40,391 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 12 steps on the current episode 2020-10-10 08:33:41,097 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0182 2020-10-10 08:33:41,097 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 89.5 2020-10-10 08:33:41,098 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00505 2020-10-10 08:33:41,098 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -18.4 2020-10-10 08:33:41,098 — rl_replicas.vpg.vpg — INFO — Epoch: 7 2020-10-10 08:33:41,099 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 25.2 2020-10-10 08:33:41,099 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 12.9 2020-10-10 08:33:41,099 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 75 2020-10-10 08:33:41,100 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:33:41,101 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 25.2 2020-10-10 08:33:41,101 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 14.7 2020-10-10 08:33:41,102 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 4.14 2020-10-10 08:33:41,102 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 16.9 2020-10-10 08:33:41,102 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -2.69 2020-10-10 08:33:41,103 — rl_replicas.vpg.vpg — INFO — Total env interactions: 32000 2020-10-10 08:33:41,103 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0107 2020-10-10 08:33:41,104 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 99.412 2020-10-10 08:33:41,104 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.688 2020-10-10 08:33:41,104 — rl_replicas.vpg.vpg — INFO — Time: 26.4 2020-10-10 08:33:43,613 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 8 steps on the current episode 2020-10-10 08:33:44,333 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0193 2020-10-10 08:33:44,334 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 104 2020-10-10 08:33:44,334 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00118 2020-10-10 08:33:44,335 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 14.6 2020-10-10 08:33:44,335 — rl_replicas.vpg.vpg — INFO — Epoch: 8 2020-10-10 08:33:44,335 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 27.1 2020-10-10 08:33:44,336 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 14.5 2020-10-10 08:33:44,336 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 90 2020-10-10 08:33:44,336 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:33:44,336 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 27 2020-10-10 08:33:44,337 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 15.6 2020-10-10 08:33:44,337 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 4.32 2020-10-10 08:33:44,338 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 17.8 2020-10-10 08:33:44,338 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -5.58 2020-10-10 08:33:44,338 — rl_replicas.vpg.vpg — INFO — Total env interactions: 36000 2020-10-10 08:33:44,338 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0116 2020-10-10 08:33:44,339 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 99.939 2020-10-10 08:33:44,339 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.688 2020-10-10 08:33:44,339 — rl_replicas.vpg.vpg — INFO — Time: 29.6 2020-10-10 08:33:46,846 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 8 steps on the current episode 2020-10-10 08:33:47,533 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0202 2020-10-10 08:33:47,533 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 126 2020-10-10 08:33:47,534 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000832 2020-10-10 08:33:47,534 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 21.4 2020-10-10 08:33:47,534 — rl_replicas.vpg.vpg — INFO — Epoch: 9 2020-10-10 08:33:47,534 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 29.1 2020-10-10 08:33:47,535 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 16.4 2020-10-10 08:33:47,535 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 86 2020-10-10 08:33:47,535 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:33:47,535 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 29 2020-10-10 08:33:47,536 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 16.7 2020-10-10 08:33:47,536 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 4.47 2020-10-10 08:33:47,536 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 19.2 2020-10-10 08:33:47,537 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -5.26 2020-10-10 08:33:47,537 — rl_replicas.vpg.vpg — INFO — Total env interactions: 40000 2020-10-10 08:33:47,537 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0125 2020-10-10 08:33:47,537 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 102.499 2020-10-10 08:33:47,538 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.688 2020-10-10 08:33:47,538 — rl_replicas.vpg.vpg — INFO — Time: 32.8 2020-10-10 08:33:50,086 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 9 steps on the current episode 2020-10-10 08:33:50,765 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.017 2020-10-10 08:33:50,766 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 121 2020-10-10 08:33:50,766 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00318 2020-10-10 08:33:50,767 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -5.01 2020-10-10 08:33:50,767 — rl_replicas.vpg.vpg — INFO — Epoch: 10 2020-10-10 08:33:50,767 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 30 2020-10-10 08:33:50,768 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 16.7 2020-10-10 08:33:50,768 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 89 2020-10-10 08:33:50,768 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:33:50,769 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 29.9 2020-10-10 08:33:50,769 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 17 2020-10-10 08:33:50,769 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 4.53 2020-10-10 08:33:50,770 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 20.5 2020-10-10 08:33:50,770 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -0.938 2020-10-10 08:33:50,770 — rl_replicas.vpg.vpg — INFO — Total env interactions: 44000 2020-10-10 08:33:50,770 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0129 2020-10-10 08:33:50,771 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 104.139 2020-10-10 08:33:50,771 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.689 2020-10-10 08:33:50,771 — rl_replicas.vpg.vpg — INFO — Time: 36.1 2020-10-10 08:33:53,275 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 54 steps on the current episode 2020-10-10 08:33:53,953 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0222 2020-10-10 08:33:53,954 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 114 2020-10-10 08:33:53,955 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00518 2020-10-10 08:33:53,955 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -6.85 2020-10-10 08:33:53,956 — rl_replicas.vpg.vpg — INFO — Epoch: 11 2020-10-10 08:33:53,956 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 30.7 2020-10-10 08:33:53,956 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 17.2 2020-10-10 08:33:53,957 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 94 2020-10-10 08:33:53,957 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:33:53,957 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 30.5 2020-10-10 08:33:53,958 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 17.5 2020-10-10 08:33:53,958 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 5.52 2020-10-10 08:33:53,958 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 22.5 2020-10-10 08:33:53,959 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -0.979 2020-10-10 08:33:53,959 — rl_replicas.vpg.vpg — INFO — Total env interactions: 48000 2020-10-10 08:33:53,959 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0137 2020-10-10 08:33:53,959 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 104.935 2020-10-10 08:33:53,960 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.688 2020-10-10 08:33:53,960 — rl_replicas.vpg.vpg — INFO — Time: 39.3 2020-10-10 08:33:56,432 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 8 steps on the current episode 2020-10-10 08:33:57,093 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0221 2020-10-10 08:33:57,094 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 113 2020-10-10 08:33:57,095 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00012 2020-10-10 08:33:57,096 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -1.08 2020-10-10 08:33:57,096 — rl_replicas.vpg.vpg — INFO — Epoch: 12 2020-10-10 08:33:57,096 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 32.7 2020-10-10 08:33:57,097 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 17.8 2020-10-10 08:33:57,097 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 99 2020-10-10 08:33:57,098 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:33:57,098 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 32.5 2020-10-10 08:33:57,098 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 18.3 2020-10-10 08:33:57,099 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 6.04 2020-10-10 08:33:57,099 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 24.2 2020-10-10 08:33:57,100 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.0476 2020-10-10 08:33:57,100 — rl_replicas.vpg.vpg — INFO — Total env interactions: 52000 2020-10-10 08:33:57,101 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0143 2020-10-10 08:33:57,101 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 105.524 2020-10-10 08:33:57,101 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.688 2020-10-10 08:33:57,102 — rl_replicas.vpg.vpg — INFO — Time: 42.4 2020-10-10 08:33:59,631 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 20 steps on the current episode 2020-10-10 08:34:00,349 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.027 2020-10-10 08:34:00,349 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 149 2020-10-10 08:34:00,350 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00496 2020-10-10 08:34:00,350 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 36.9 2020-10-10 08:34:00,350 — rl_replicas.vpg.vpg — INFO — Epoch: 13 2020-10-10 08:34:00,351 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 31.1 2020-10-10 08:34:00,351 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 21.8 2020-10-10 08:34:00,351 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 111 2020-10-10 08:34:00,352 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:34:00,352 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 31 2020-10-10 08:34:00,352 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 19.5 2020-10-10 08:34:00,353 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 7.52 2020-10-10 08:34:00,353 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 26.6 2020-10-10 08:34:00,353 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -0.027 2020-10-10 08:34:00,353 — rl_replicas.vpg.vpg — INFO — Total env interactions: 56000 2020-10-10 08:34:00,354 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0152 2020-10-10 08:34:00,355 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 108.664 2020-10-10 08:34:00,355 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.689 2020-10-10 08:34:00,355 — rl_replicas.vpg.vpg — INFO — Time: 45.7 2020-10-10 08:34:02,957 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 34 steps on the current episode 2020-10-10 08:34:03,680 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0359 2020-10-10 08:34:03,681 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 87.8 2020-10-10 08:34:03,681 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00888 2020-10-10 08:34:03,681 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -61.7 2020-10-10 08:34:03,682 — rl_replicas.vpg.vpg — INFO — Epoch: 14 2020-10-10 08:34:03,682 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 31.1 2020-10-10 08:34:03,682 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 16.2 2020-10-10 08:34:03,683 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 99 2020-10-10 08:34:03,683 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:34:03,683 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 31 2020-10-10 08:34:03,684 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 17.7 2020-10-10 08:34:03,684 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 6.87 2020-10-10 08:34:03,685 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 25.4 2020-10-10 08:34:03,685 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.241 2020-10-10 08:34:03,685 — rl_replicas.vpg.vpg — INFO — Total env interactions: 60000 2020-10-10 08:34:03,685 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0166 2020-10-10 08:34:03,686 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 107.270 2020-10-10 08:34:03,686 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.689 2020-10-10 08:34:03,686 — rl_replicas.vpg.vpg — INFO — Time: 49 2020-10-10 08:34:06,236 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 17 steps on the current episode 2020-10-10 08:34:06,916 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0288 2020-10-10 08:34:06,917 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 150 2020-10-10 08:34:06,917 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.0071 2020-10-10 08:34:06,918 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 62.4 2020-10-10 08:34:06,918 — rl_replicas.vpg.vpg — INFO — Epoch: 15 2020-10-10 08:34:06,918 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 34 2020-10-10 08:34:06,919 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 22 2020-10-10 08:34:06,919 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 114 2020-10-10 08:34:06,919 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:34:06,920 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 33.9 2020-10-10 08:34:06,920 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 20.4 2020-10-10 08:34:06,920 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 7.6 2020-10-10 08:34:06,921 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 27 2020-10-10 08:34:06,921 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -1.71 2020-10-10 08:34:06,922 — rl_replicas.vpg.vpg — INFO — Total env interactions: 64000 2020-10-10 08:34:06,923 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0174 2020-10-10 08:34:06,924 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 109.954 2020-10-10 08:34:06,924 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.687 2020-10-10 08:34:06,924 — rl_replicas.vpg.vpg — INFO — Time: 52.2 2020-10-10 08:34:09,555 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 20 steps on the current episode 2020-10-10 08:34:10,231 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0355 2020-10-10 08:34:10,231 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 96.9 2020-10-10 08:34:10,232 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00672 2020-10-10 08:34:10,232 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -53.4 2020-10-10 08:34:10,232 — rl_replicas.vpg.vpg — INFO — Epoch: 16 2020-10-10 08:34:10,232 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 31.1 2020-10-10 08:34:10,233 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 17.7 2020-10-10 08:34:10,233 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 100 2020-10-10 08:34:10,233 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:34:10,234 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 31 2020-10-10 08:34:10,234 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 18 2020-10-10 08:34:10,235 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 7.8 2020-10-10 08:34:10,235 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 26.4 2020-10-10 08:34:10,236 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.925 2020-10-10 08:34:10,236 — rl_replicas.vpg.vpg — INFO — Total env interactions: 68000 2020-10-10 08:34:10,236 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0184 2020-10-10 08:34:10,236 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 109.183 2020-10-10 08:34:10,237 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.687 2020-10-10 08:34:10,237 — rl_replicas.vpg.vpg — INFO — Time: 55.5 2020-10-10 08:34:12,692 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 38 steps on the current episode 2020-10-10 08:34:13,433 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0354 2020-10-10 08:34:13,434 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 85.3 2020-10-10 08:34:13,434 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 7.53e-05 2020-10-10 08:34:13,435 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -11.5 2020-10-10 08:34:13,436 — rl_replicas.vpg.vpg — INFO — Epoch: 17 2020-10-10 08:34:13,436 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 32.3 2020-10-10 08:34:13,437 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 17.1 2020-10-10 08:34:13,437 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 84 2020-10-10 08:34:13,438 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:34:13,439 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 32.3 2020-10-10 08:34:13,439 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 18.2 2020-10-10 08:34:13,440 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 7.59 2020-10-10 08:34:13,441 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 26.5 2020-10-10 08:34:13,441 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -0.216 2020-10-10 08:34:13,441 — rl_replicas.vpg.vpg — INFO — Total env interactions: 72000 2020-10-10 08:34:13,442 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0194 2020-10-10 08:34:13,442 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 107.859 2020-10-10 08:34:13,443 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.687 2020-10-10 08:34:13,443 — rl_replicas.vpg.vpg — INFO — Time: 58.7 2020-10-10 08:34:15,915 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 9 steps on the current episode 2020-10-10 08:34:16,593 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0325 2020-10-10 08:34:16,594 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 135 2020-10-10 08:34:16,595 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00299 2020-10-10 08:34:16,596 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 49.2 2020-10-10 08:34:16,597 — rl_replicas.vpg.vpg — INFO — Epoch: 18 2020-10-10 08:34:16,597 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 32.7 2020-10-10 08:34:16,598 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 19.6 2020-10-10 08:34:16,598 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 133 2020-10-10 08:34:16,598 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:34:16,599 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 32.5 2020-10-10 08:34:16,599 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 19 2020-10-10 08:34:16,600 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 7.86 2020-10-10 08:34:16,600 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 27 2020-10-10 08:34:16,600 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -0.174 2020-10-10 08:34:16,600 — rl_replicas.vpg.vpg — INFO — Total env interactions: 76000 2020-10-10 08:34:16,601 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0201 2020-10-10 08:34:16,601 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 109.263 2020-10-10 08:34:16,601 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.688 2020-10-10 08:34:16,602 — rl_replicas.vpg.vpg — INFO — Time: 61.9 2020-10-10 08:34:19,780 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0387 2020-10-10 08:34:19,781 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 115 2020-10-10 08:34:19,782 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00628 2020-10-10 08:34:19,782 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -19.8 2020-10-10 08:34:19,783 — rl_replicas.vpg.vpg — INFO — Epoch: 19 2020-10-10 08:34:19,783 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 36.4 2020-10-10 08:34:19,783 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 19.4 2020-10-10 08:34:19,784 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 108 2020-10-10 08:34:19,784 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:34:19,784 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 36.4 2020-10-10 08:34:19,785 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 20.1 2020-10-10 08:34:19,785 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 7.85 2020-10-10 08:34:19,786 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 27.5 2020-10-10 08:34:19,786 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.962 2020-10-10 08:34:19,786 — rl_replicas.vpg.vpg — INFO — Total env interactions: 80000 2020-10-10 08:34:19,787 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.021 2020-10-10 08:34:19,787 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 109.535 2020-10-10 08:34:19,787 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.684 2020-10-10 08:34:19,787 — rl_replicas.vpg.vpg — INFO — Time: 65.1 2020-10-10 08:34:22,279 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 43 steps on the current episode 2020-10-10 08:34:22,937 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.039 2020-10-10 08:34:22,938 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 134 2020-10-10 08:34:22,939 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00023 2020-10-10 08:34:22,939 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 19.6 2020-10-10 08:34:22,940 — rl_replicas.vpg.vpg — INFO — Epoch: 20 2020-10-10 08:34:22,940 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 34.1 2020-10-10 08:34:22,941 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 21.1 2020-10-10 08:34:22,941 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 125 2020-10-10 08:34:22,942 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:34:22,942 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 33.9 2020-10-10 08:34:22,942 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 20 2020-10-10 08:34:22,943 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 8.26 2020-10-10 08:34:22,943 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 28.1 2020-10-10 08:34:22,943 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.383 2020-10-10 08:34:22,944 — rl_replicas.vpg.vpg — INFO — Total env interactions: 84000 2020-10-10 08:34:22,944 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0219 2020-10-10 08:34:22,944 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 110.715 2020-10-10 08:34:22,944 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.683 2020-10-10 08:34:22,945 — rl_replicas.vpg.vpg — INFO — Time: 68.2 2020-10-10 08:34:25,397 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 31 steps on the current episode 2020-10-10 08:34:26,058 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0355 2020-10-10 08:34:26,058 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 124 2020-10-10 08:34:26,059 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00347 2020-10-10 08:34:26,059 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -10.8 2020-10-10 08:34:26,059 — rl_replicas.vpg.vpg — INFO — Epoch: 21 2020-10-10 08:34:26,059 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 35 2020-10-10 08:34:26,060 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 20 2020-10-10 08:34:26,060 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 101 2020-10-10 08:34:26,060 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:34:26,061 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 34.8 2020-10-10 08:34:26,061 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 19.8 2020-10-10 08:34:26,061 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 7.82 2020-10-10 08:34:26,062 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 28.7 2020-10-10 08:34:26,062 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.00319 2020-10-10 08:34:26,062 — rl_replicas.vpg.vpg — INFO — Total env interactions: 88000 2020-10-10 08:34:26,063 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0225 2020-10-10 08:34:26,063 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 111.298 2020-10-10 08:34:26,064 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.682 2020-10-10 08:34:26,064 — rl_replicas.vpg.vpg — INFO — Time: 71.4 2020-10-10 08:34:28,525 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 5 steps on the current episode 2020-10-10 08:34:29,206 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0377 2020-10-10 08:34:29,207 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 152 2020-10-10 08:34:29,208 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00223 2020-10-10 08:34:29,208 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 28.6 2020-10-10 08:34:29,208 — rl_replicas.vpg.vpg — INFO — Epoch: 22 2020-10-10 08:34:29,209 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 36.2 2020-10-10 08:34:29,209 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 23.4 2020-10-10 08:34:29,210 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 146 2020-10-10 08:34:29,210 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:34:29,211 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 36 2020-10-10 08:34:29,211 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 21.2 2020-10-10 08:34:29,212 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 9.18 2020-10-10 08:34:29,213 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 30.3 2020-10-10 08:34:29,213 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -1.14 2020-10-10 08:34:29,213 — rl_replicas.vpg.vpg — INFO — Total env interactions: 92000 2020-10-10 08:34:29,214 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0231 2020-10-10 08:34:29,214 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 113.071 2020-10-10 08:34:29,215 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-10 08:34:29,215 — rl_replicas.vpg.vpg — INFO — Time: 74.5 2020-10-10 08:34:31,661 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 19 steps on the current episode 2020-10-10 08:34:32,342 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0392 2020-10-10 08:34:32,343 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 149 2020-10-10 08:34:32,343 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00151 2020-10-10 08:34:32,343 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -2.84 2020-10-10 08:34:32,344 — rl_replicas.vpg.vpg — INFO — Epoch: 23 2020-10-10 08:34:32,344 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 37.7 2020-10-10 08:34:32,345 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 23 2020-10-10 08:34:32,345 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 134 2020-10-10 08:34:32,345 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:34:32,346 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 37.4 2020-10-10 08:34:32,346 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 21.5 2020-10-10 08:34:32,347 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 8.99 2020-10-10 08:34:32,347 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 30.8 2020-10-10 08:34:32,348 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: -1 2020-10-10 08:34:32,348 — rl_replicas.vpg.vpg — INFO — Total env interactions: 96000 2020-10-10 08:34:32,348 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0238 2020-10-10 08:34:32,349 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 114.579 2020-10-10 08:34:32,349 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-10 08:34:32,349 — rl_replicas.vpg.vpg — INFO — Time: 77.6 2020-10-10 08:34:34,821 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 20 steps on the current episode 2020-10-10 08:34:35,556 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.047 2020-10-10 08:34:35,556 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 106 2020-10-10 08:34:35,557 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00777 2020-10-10 08:34:35,557 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -42.9 2020-10-10 08:34:35,557 — rl_replicas.vpg.vpg — INFO — Epoch: 24 2020-10-10 08:34:35,558 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 35.6 2020-10-10 08:34:35,558 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 20 2020-10-10 08:34:35,559 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 111 2020-10-10 08:34:35,559 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:34:35,559 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 35.4 2020-10-10 08:34:35,560 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 20 2020-10-10 08:34:35,560 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 9.13 2020-10-10 08:34:35,560 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 30.5 2020-10-10 08:34:35,561 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.135 2020-10-10 08:34:35,561 — rl_replicas.vpg.vpg — INFO — Total env interactions: 100000 2020-10-10 08:34:35,561 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0247 2020-10-10 08:34:35,562 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 114.250 2020-10-10 08:34:35,562 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-10 08:34:35,562 — rl_replicas.vpg.vpg — INFO — Time: 80.9 2020-10-10 08:34:38,000 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 16 steps on the current episode 2020-10-10 08:34:38,692 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0452 2020-10-10 08:34:38,693 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 101 2020-10-10 08:34:38,693 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00182 2020-10-10 08:34:38,693 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -5.81 2020-10-10 08:34:38,694 — rl_replicas.vpg.vpg — INFO — Epoch: 25 2020-10-10 08:34:38,694 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 35.3 2020-10-10 08:34:38,694 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 19 2020-10-10 08:34:38,695 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 102 2020-10-10 08:34:38,695 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:34:38,695 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 35.1 2020-10-10 08:34:38,696 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 19.7 2020-10-10 08:34:38,696 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 8.67 2020-10-10 08:34:38,697 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 28.9 2020-10-10 08:34:38,697 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.92 2020-10-10 08:34:38,698 — rl_replicas.vpg.vpg — INFO — Total env interactions: 104000 2020-10-10 08:34:38,698 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0255 2020-10-10 08:34:38,699 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 113.723 2020-10-10 08:34:38,699 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.682 2020-10-10 08:34:38,699 — rl_replicas.vpg.vpg — INFO — Time: 84 2020-10-10 08:34:41,210 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 33 steps on the current episode 2020-10-10 08:34:41,897 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0364 2020-10-10 08:34:41,898 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 160 2020-10-10 08:34:41,899 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00877 2020-10-10 08:34:41,899 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 59.1 2020-10-10 08:34:41,899 — rl_replicas.vpg.vpg — INFO — Epoch: 26 2020-10-10 08:34:41,899 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 37.6 2020-10-10 08:34:41,900 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 23.6 2020-10-10 08:34:41,900 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 155 2020-10-10 08:34:41,901 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:34:41,901 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 37.4 2020-10-10 08:34:41,901 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 21.6 2020-10-10 08:34:41,902 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 8.93 2020-10-10 08:34:41,902 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 31 2020-10-10 08:34:41,902 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.622 2020-10-10 08:34:41,902 — rl_replicas.vpg.vpg — INFO — Total env interactions: 108000 2020-10-10 08:34:41,903 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0259 2020-10-10 08:34:41,903 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 115.425 2020-10-10 08:34:41,903 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-10 08:34:41,904 — rl_replicas.vpg.vpg — INFO — Time: 87.2 2020-10-10 08:34:44,399 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 32 steps on the current episode 2020-10-10 08:34:45,063 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0391 2020-10-10 08:34:45,064 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 135 2020-10-10 08:34:45,064 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00265 2020-10-10 08:34:45,064 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -24.4 2020-10-10 08:34:45,065 — rl_replicas.vpg.vpg — INFO — Epoch: 27 2020-10-10 08:34:45,065 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 40.5 2020-10-10 08:34:45,066 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 22 2020-10-10 08:34:45,066 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 109 2020-10-10 08:34:45,067 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 9 2020-10-10 08:34:45,067 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 40.4 2020-10-10 08:34:45,067 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 22.1 2020-10-10 08:34:45,068 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 8.71 2020-10-10 08:34:45,068 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 31.1 2020-10-10 08:34:45,068 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.886 2020-10-10 08:34:45,069 — rl_replicas.vpg.vpg — INFO — Total env interactions: 112000 2020-10-10 08:34:45,069 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0264 2020-10-10 08:34:45,069 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 116.135 2020-10-10 08:34:45,070 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-10 08:34:45,070 — rl_replicas.vpg.vpg — INFO — Time: 90.4 2020-10-10 08:34:47,532 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 43 steps on the current episode 2020-10-10 08:34:48,185 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0393 2020-10-10 08:34:48,185 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 177 2020-10-10 08:34:48,185 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000232 2020-10-10 08:34:48,186 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 41.7 2020-10-10 08:34:48,186 — rl_replicas.vpg.vpg — INFO — Epoch: 28 2020-10-10 08:34:48,186 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 43.5 2020-10-10 08:34:48,187 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 25.5 2020-10-10 08:34:48,187 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 131 2020-10-10 08:34:48,187 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:34:48,188 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 43.5 2020-10-10 08:34:48,188 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 23.7 2020-10-10 08:34:48,189 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 9.15 2020-10-10 08:34:48,189 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 32.4 2020-10-10 08:34:48,190 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.997 2020-10-10 08:34:48,190 — rl_replicas.vpg.vpg — INFO — Total env interactions: 116000 2020-10-10 08:34:48,190 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0268 2020-10-10 08:34:48,191 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 118.235 2020-10-10 08:34:48,191 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.681 2020-10-10 08:34:48,191 — rl_replicas.vpg.vpg — INFO — Time: 93.5 2020-10-10 08:34:50,676 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 15 steps on the current episode 2020-10-10 08:34:51,352 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0435 2020-10-10 08:34:51,353 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 135 2020-10-10 08:34:51,353 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00416 2020-10-10 08:34:51,354 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -41.8 2020-10-10 08:34:51,354 — rl_replicas.vpg.vpg — INFO — Epoch: 29 2020-10-10 08:34:51,354 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 38.4 2020-10-10 08:34:51,355 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 22.7 2020-10-10 08:34:51,355 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 152 2020-10-10 08:34:51,355 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:34:51,356 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 38.1 2020-10-10 08:34:51,356 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 21.6 2020-10-10 08:34:51,356 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 9.52 2020-10-10 08:34:51,357 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 31.8 2020-10-10 08:34:51,357 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.575 2020-10-10 08:34:51,357 — rl_replicas.vpg.vpg — INFO — Total env interactions: 120000 2020-10-10 08:34:51,357 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0274 2020-10-10 08:34:51,358 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 118.801 2020-10-10 08:34:51,358 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.68 2020-10-10 08:34:51,358 — rl_replicas.vpg.vpg — INFO — Time: 96.7 2020-10-10 08:34:53,778 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 27 steps on the current episode 2020-10-10 08:34:54,443 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0412 2020-10-10 08:34:54,443 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 151 2020-10-10 08:34:54,444 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00222 2020-10-10 08:34:54,444 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 15.7 2020-10-10 08:34:54,444 — rl_replicas.vpg.vpg — INFO — Epoch: 30 2020-10-10 08:34:54,445 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 43.8 2020-10-10 08:34:54,445 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 26.1 2020-10-10 08:34:54,445 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 144 2020-10-10 08:34:54,445 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 10 2020-10-10 08:34:54,446 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 43.5 2020-10-10 08:34:54,446 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 24.1 2020-10-10 08:34:54,446 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 10.4 2020-10-10 08:34:54,447 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 34.3 2020-10-10 08:34:54,447 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.228 2020-10-10 08:34:54,447 — rl_replicas.vpg.vpg — INFO — Total env interactions: 124000 2020-10-10 08:34:54,447 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0278 2020-10-10 08:34:54,448 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 119.837 2020-10-10 08:34:54,448 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.677 2020-10-10 08:34:54,448 — rl_replicas.vpg.vpg — INFO — Time: 99.7 2020-10-10 08:34:56,895 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 17 steps on the current episode 2020-10-10 08:34:57,580 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.043 2020-10-10 08:34:57,580 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 155 2020-10-10 08:34:57,581 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00175 2020-10-10 08:34:57,581 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 4.03 2020-10-10 08:34:57,582 — rl_replicas.vpg.vpg — INFO — Epoch: 31 2020-10-10 08:34:57,582 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 41.2 2020-10-10 08:34:57,582 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 24.7 2020-10-10 08:34:57,582 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 119 2020-10-10 08:34:57,583 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:34:57,583 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 40.8 2020-10-10 08:34:57,583 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 23.1 2020-10-10 08:34:57,584 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 9.9 2020-10-10 08:34:57,584 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 34.1 2020-10-10 08:34:57,584 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.752 2020-10-10 08:34:57,584 — rl_replicas.vpg.vpg — INFO — Total env interactions: 128000 2020-10-10 08:34:57,585 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0283 2020-10-10 08:34:57,585 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 120.933 2020-10-10 08:34:57,585 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.678 2020-10-10 08:34:57,586 — rl_replicas.vpg.vpg — INFO — Time: 103 2020-10-10 08:35:00,047 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 25 steps on the current episode 2020-10-10 08:35:00,705 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0455 2020-10-10 08:35:00,705 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 171 2020-10-10 08:35:00,706 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00255 2020-10-10 08:35:00,706 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 15.6 2020-10-10 08:35:00,706 — rl_replicas.vpg.vpg — INFO — Epoch: 32 2020-10-10 08:35:00,707 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 42.4 2020-10-10 08:35:00,707 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 27.1 2020-10-10 08:35:00,708 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 200 2020-10-10 08:35:00,708 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:35:00,708 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 42.1 2020-10-10 08:35:00,708 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 23.6 2020-10-10 08:35:00,709 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 10.6 2020-10-10 08:35:00,709 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 35.9 2020-10-10 08:35:00,710 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.05 2020-10-10 08:35:00,710 — rl_replicas.vpg.vpg — INFO — Total env interactions: 132000 2020-10-10 08:35:00,710 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0288 2020-10-10 08:35:00,711 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 122.436 2020-10-10 08:35:00,711 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.678 2020-10-10 08:35:00,711 — rl_replicas.vpg.vpg — INFO — Time: 106 2020-10-10 08:35:03,150 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 24 steps on the current episode 2020-10-10 08:35:04,390 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0462 2020-10-10 08:35:04,390 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 127 2020-10-10 08:35:04,391 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000674 2020-10-10 08:35:04,391 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -43.3 2020-10-10 08:35:04,391 — rl_replicas.vpg.vpg — INFO — Epoch: 33 2020-10-10 08:35:04,392 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 42.5 2020-10-10 08:35:04,392 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 23.5 2020-10-10 08:35:04,393 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 145 2020-10-10 08:35:04,393 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 13 2020-10-10 08:35:04,393 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 42.1 2020-10-10 08:35:04,394 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 23 2020-10-10 08:35:04,394 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 10.3 2020-10-10 08:35:04,394 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 35.4 2020-10-10 08:35:04,394 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.08 2020-10-10 08:35:04,395 — rl_replicas.vpg.vpg — INFO — Total env interactions: 136000 2020-10-10 08:35:04,395 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0293 2020-10-10 08:35:04,395 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 122.576 2020-10-10 08:35:04,395 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.678 2020-10-10 08:35:04,395 — rl_replicas.vpg.vpg — INFO — Time: 110 2020-10-10 08:35:08,330 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 4 steps on the current episode 2020-10-10 08:35:09,333 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0354 2020-10-10 08:35:09,333 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 144 2020-10-10 08:35:09,333 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.0108 2020-10-10 08:35:09,334 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 17.3 2020-10-10 08:35:09,334 — rl_replicas.vpg.vpg — INFO — Epoch: 34 2020-10-10 08:35:09,334 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 43.8 2020-10-10 08:35:09,335 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 23 2020-10-10 08:35:09,335 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 131 2020-10-10 08:35:09,335 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:35:09,335 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 43.5 2020-10-10 08:35:09,336 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 23.2 2020-10-10 08:35:09,336 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 9.27 2020-10-10 08:35:09,336 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 34.7 2020-10-10 08:35:09,337 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.56 2020-10-10 08:35:09,337 — rl_replicas.vpg.vpg — INFO — Total env interactions: 140000 2020-10-10 08:35:09,337 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0295 2020-10-10 08:35:09,337 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 123.202 2020-10-10 08:35:09,337 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.678 2020-10-10 08:35:09,338 — rl_replicas.vpg.vpg — INFO — Time: 115 2020-10-10 08:35:12,944 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 38 steps on the current episode 2020-10-10 08:35:14,108 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.048 2020-10-10 08:35:14,108 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 121 2020-10-10 08:35:14,109 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.0126 2020-10-10 08:35:14,109 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -23.9 2020-10-10 08:35:14,110 — rl_replicas.vpg.vpg — INFO — Epoch: 35 2020-10-10 08:35:14,111 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 44.1 2020-10-10 08:35:14,111 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 22.2 2020-10-10 08:35:14,111 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 128 2020-10-10 08:35:14,112 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 14 2020-10-10 08:35:14,112 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 44 2020-10-10 08:35:14,112 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 22.9 2020-10-10 08:35:14,112 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 10.1 2020-10-10 08:35:14,113 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 35.1 2020-10-10 08:35:14,113 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.47 2020-10-10 08:35:14,114 — rl_replicas.vpg.vpg — INFO — Total env interactions: 144000 2020-10-10 08:35:14,114 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.03 2020-10-10 08:35:14,114 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 123.130 2020-10-10 08:35:14,115 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.678 2020-10-10 08:35:14,115 — rl_replicas.vpg.vpg — INFO — Time: 119 2020-10-10 08:35:17,539 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 1 steps on the current episode 2020-10-10 08:35:18,202 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0423 2020-10-10 08:35:18,203 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 153 2020-10-10 08:35:18,204 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00568 2020-10-10 08:35:18,204 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 32.5 2020-10-10 08:35:18,205 — rl_replicas.vpg.vpg — INFO — Epoch: 36 2020-10-10 08:35:18,205 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 48 2020-10-10 08:35:18,206 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 26.2 2020-10-10 08:35:18,206 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 129 2020-10-10 08:35:18,206 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 14 2020-10-10 08:35:18,207 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 47.6 2020-10-10 08:35:18,207 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 25.3 2020-10-10 08:35:18,207 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 10.6 2020-10-10 08:35:18,208 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 36.1 2020-10-10 08:35:18,208 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.5 2020-10-10 08:35:18,208 — rl_replicas.vpg.vpg — INFO — Total env interactions: 148000 2020-10-10 08:35:18,209 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0304 2020-10-10 08:35:18,209 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 123.939 2020-10-10 08:35:18,209 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.678 2020-10-10 08:35:18,210 — rl_replicas.vpg.vpg — INFO — Time: 124 2020-10-10 08:35:20,661 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 35 steps on the current episode 2020-10-10 08:35:21,347 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0384 2020-10-10 08:35:21,348 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 160 2020-10-10 08:35:21,348 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00385 2020-10-10 08:35:21,349 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 6.97 2020-10-10 08:35:21,349 — rl_replicas.vpg.vpg — INFO — Epoch: 37 2020-10-10 08:35:21,349 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 46 2020-10-10 08:35:21,350 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 26.3 2020-10-10 08:35:21,350 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 137 2020-10-10 08:35:21,350 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 13 2020-10-10 08:35:21,351 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 46 2020-10-10 08:35:21,351 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 24.6 2020-10-10 08:35:21,352 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 10.6 2020-10-10 08:35:21,352 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 36.7 2020-10-10 08:35:21,352 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.46 2020-10-10 08:35:21,353 — rl_replicas.vpg.vpg — INFO — Total env interactions: 152000 2020-10-10 08:35:21,353 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0306 2020-10-10 08:35:21,353 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 124.889 2020-10-10 08:35:21,353 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.679 2020-10-10 08:35:21,354 — rl_replicas.vpg.vpg — INFO — Time: 127 2020-10-10 08:35:23,929 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 13 steps on the current episode 2020-10-10 08:35:24,601 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0405 2020-10-10 08:35:24,602 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 124 2020-10-10 08:35:24,603 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00205 2020-10-10 08:35:24,603 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -36.4 2020-10-10 08:35:24,604 — rl_replicas.vpg.vpg — INFO — Epoch: 38 2020-10-10 08:35:24,604 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 46.8 2020-10-10 08:35:24,605 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 22.1 2020-10-10 08:35:24,605 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 127 2020-10-10 08:35:24,605 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 13 2020-10-10 08:35:24,606 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 46.5 2020-10-10 08:35:24,606 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 23.7 2020-10-10 08:35:24,607 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 9.98 2020-10-10 08:35:24,607 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 35.1 2020-10-10 08:35:24,607 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.81 2020-10-10 08:35:24,608 — rl_replicas.vpg.vpg — INFO — Total env interactions: 156000 2020-10-10 08:35:24,608 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0308 2020-10-10 08:35:24,609 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 124.856 2020-10-10 08:35:24,609 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.679 2020-10-10 08:35:24,609 — rl_replicas.vpg.vpg — INFO — Time: 130 2020-10-10 08:35:26,977 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 11 steps on the current episode 2020-10-10 08:35:27,634 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0447 2020-10-10 08:35:27,635 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 150 2020-10-10 08:35:27,635 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00421 2020-10-10 08:35:27,636 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 26 2020-10-10 08:35:27,636 — rl_replicas.vpg.vpg — INFO — Epoch: 39 2020-10-10 08:35:27,636 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 46.8 2020-10-10 08:35:27,637 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 25.3 2020-10-10 08:35:27,637 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 159 2020-10-10 08:35:27,637 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:35:27,638 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 46.5 2020-10-10 08:35:27,638 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 24.6 2020-10-10 08:35:27,638 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 10.4 2020-10-10 08:35:27,639 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 36 2020-10-10 08:35:27,639 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.64 2020-10-10 08:35:27,639 — rl_replicas.vpg.vpg — INFO — Total env interactions: 160000 2020-10-10 08:35:27,640 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0312 2020-10-10 08:35:27,640 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 125.475 2020-10-10 08:35:27,641 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.678 2020-10-10 08:35:27,641 — rl_replicas.vpg.vpg — INFO — Time: 133 2020-10-10 08:35:30,117 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 24 steps on the current episode 2020-10-10 08:35:30,800 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0466 2020-10-10 08:35:30,800 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 160 2020-10-10 08:35:30,800 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00196 2020-10-10 08:35:30,801 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 10.5 2020-10-10 08:35:30,801 — rl_replicas.vpg.vpg — INFO — Epoch: 40 2020-10-10 08:35:30,801 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 45.7 2020-10-10 08:35:30,801 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 27.2 2020-10-10 08:35:30,802 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 136 2020-10-10 08:35:30,802 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 14 2020-10-10 08:35:30,802 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 45.5 2020-10-10 08:35:30,803 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 24.9 2020-10-10 08:35:30,804 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 11.2 2020-10-10 08:35:30,804 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 36.6 2020-10-10 08:35:30,804 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.68 2020-10-10 08:35:30,805 — rl_replicas.vpg.vpg — INFO — Total env interactions: 164000 2020-10-10 08:35:30,805 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0316 2020-10-10 08:35:30,806 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 126.321 2020-10-10 08:35:30,806 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.677 2020-10-10 08:35:30,806 — rl_replicas.vpg.vpg — INFO — Time: 136 2020-10-10 08:35:33,243 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 95 steps on the current episode 2020-10-10 08:35:33,929 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0461 2020-10-10 08:35:33,930 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 138 2020-10-10 08:35:33,930 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.000572 2020-10-10 08:35:33,930 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -22 2020-10-10 08:35:33,931 — rl_replicas.vpg.vpg — INFO — Epoch: 41 2020-10-10 08:35:33,932 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 43.4 2020-10-10 08:35:33,932 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 23.7 2020-10-10 08:35:33,933 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 127 2020-10-10 08:35:33,933 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 14 2020-10-10 08:35:33,933 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 43 2020-10-10 08:35:33,934 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 23.3 2020-10-10 08:35:33,934 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 10.1 2020-10-10 08:35:33,934 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 36.1 2020-10-10 08:35:33,935 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.73 2020-10-10 08:35:33,935 — rl_replicas.vpg.vpg — INFO — Total env interactions: 168000 2020-10-10 08:35:33,935 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0319 2020-10-10 08:35:33,936 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 126.603 2020-10-10 08:35:33,936 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.676 2020-10-10 08:35:33,936 — rl_replicas.vpg.vpg — INFO — Time: 139 2020-10-10 08:35:36,412 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 55 steps on the current episode 2020-10-10 08:35:37,090 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0439 2020-10-10 08:35:37,091 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 163 2020-10-10 08:35:37,091 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00216 2020-10-10 08:35:37,091 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 25.2 2020-10-10 08:35:37,092 — rl_replicas.vpg.vpg — INFO — Epoch: 42 2020-10-10 08:35:37,092 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 49.8 2020-10-10 08:35:37,092 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 27.5 2020-10-10 08:35:37,092 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 139 2020-10-10 08:35:37,093 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 11 2020-10-10 08:35:37,093 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 49.4 2020-10-10 08:35:37,093 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 26.1 2020-10-10 08:35:37,094 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 11.1 2020-10-10 08:35:37,095 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 38.3 2020-10-10 08:35:37,095 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.52 2020-10-10 08:35:37,096 — rl_replicas.vpg.vpg — INFO — Total env interactions: 172000 2020-10-10 08:35:37,096 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0322 2020-10-10 08:35:37,097 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 127.458 2020-10-10 08:35:37,097 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.672 2020-10-10 08:35:37,097 — rl_replicas.vpg.vpg — INFO — Time: 142 2020-10-10 08:35:39,493 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 19 steps on the current episode 2020-10-10 08:35:40,172 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0483 2020-10-10 08:35:40,172 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 127 2020-10-10 08:35:40,173 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.00436 2020-10-10 08:35:40,174 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -36.1 2020-10-10 08:35:40,174 — rl_replicas.vpg.vpg — INFO — Epoch: 43 2020-10-10 08:35:40,174 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 49.1 2020-10-10 08:35:40,175 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 23.9 2020-10-10 08:35:40,176 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 116 2020-10-10 08:35:40,176 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:35:40,177 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 48.8 2020-10-10 08:35:40,177 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 25 2020-10-10 08:35:40,178 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 10.5 2020-10-10 08:35:40,178 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 36.8 2020-10-10 08:35:40,179 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.07 2020-10-10 08:35:40,179 — rl_replicas.vpg.vpg — INFO — Total env interactions: 176000 2020-10-10 08:35:40,180 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0325 2020-10-10 08:35:40,180 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 127.455 2020-10-10 08:35:40,180 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.669 2020-10-10 08:35:40,181 — rl_replicas.vpg.vpg — INFO — Time: 145 2020-10-10 08:35:42,586 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 38 steps on the current episode 2020-10-10 08:35:43,268 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.044 2020-10-10 08:35:43,269 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 149 2020-10-10 08:35:43,269 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00425 2020-10-10 08:35:43,269 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 21.5 2020-10-10 08:35:43,269 — rl_replicas.vpg.vpg — INFO — Epoch: 44 2020-10-10 08:35:43,270 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 49.2 2020-10-10 08:35:43,270 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 25.9 2020-10-10 08:35:43,270 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 128 2020-10-10 08:35:43,270 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 13 2020-10-10 08:35:43,271 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 48.8 2020-10-10 08:35:43,271 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 25.5 2020-10-10 08:35:43,271 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 10.8 2020-10-10 08:35:43,272 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 38.5 2020-10-10 08:35:43,272 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.937 2020-10-10 08:35:43,272 — rl_replicas.vpg.vpg — INFO — Total env interactions: 180000 2020-10-10 08:35:43,272 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0328 2020-10-10 08:35:43,273 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 127.930 2020-10-10 08:35:43,273 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.669 2020-10-10 08:35:43,273 — rl_replicas.vpg.vpg — INFO — Time: 149 2020-10-10 08:35:45,713 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 7 steps on the current episode 2020-10-10 08:35:46,389 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0436 2020-10-10 08:35:46,389 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 113 2020-10-10 08:35:46,389 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.000472 2020-10-10 08:35:46,390 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -35.4 2020-10-10 08:35:46,390 — rl_replicas.vpg.vpg — INFO — Epoch: 45 2020-10-10 08:35:46,390 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 46.9 2020-10-10 08:35:46,391 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 22.1 2020-10-10 08:35:46,391 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 126 2020-10-10 08:35:46,391 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 12 2020-10-10 08:35:46,392 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 46.5 2020-10-10 08:35:46,392 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 23.9 2020-10-10 08:35:46,393 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 10.4 2020-10-10 08:35:46,393 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 37.6 2020-10-10 08:35:46,393 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.452 2020-10-10 08:35:46,394 — rl_replicas.vpg.vpg — INFO — Total env interactions: 184000 2020-10-10 08:35:46,394 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.033 2020-10-10 08:35:46,394 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 127.614 2020-10-10 08:35:46,395 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.67 2020-10-10 08:35:46,395 — rl_replicas.vpg.vpg — INFO — Time: 152 2020-10-10 08:35:48,842 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 38 steps on the current episode 2020-10-10 08:35:49,540 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0419 2020-10-10 08:35:49,540 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 152 2020-10-10 08:35:49,540 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00168 2020-10-10 08:35:49,541 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 38.6 2020-10-10 08:35:49,541 — rl_replicas.vpg.vpg — INFO — Epoch: 46 2020-10-10 08:35:49,541 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 52.2 2020-10-10 08:35:49,542 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 27.7 2020-10-10 08:35:49,542 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 142 2020-10-10 08:35:49,542 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 16 2020-10-10 08:35:49,542 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 51.9 2020-10-10 08:35:49,543 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 26.7 2020-10-10 08:35:49,543 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 11.4 2020-10-10 08:35:49,543 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 39.7 2020-10-10 08:35:49,544 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.295 2020-10-10 08:35:49,544 — rl_replicas.vpg.vpg — INFO — Total env interactions: 188000 2020-10-10 08:35:49,544 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0332 2020-10-10 08:35:49,544 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 128.133 2020-10-10 08:35:49,545 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.669 2020-10-10 08:35:49,545 — rl_replicas.vpg.vpg — INFO — Time: 155 2020-10-10 08:35:52,011 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 51 steps on the current episode 2020-10-10 08:35:52,669 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0421 2020-10-10 08:35:52,670 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 158 2020-10-10 08:35:52,670 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.000217 2020-10-10 08:35:52,670 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 6.07 2020-10-10 08:35:52,670 — rl_replicas.vpg.vpg — INFO — Epoch: 47 2020-10-10 08:35:52,671 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 53.1 2020-10-10 08:35:52,671 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 26.7 2020-10-10 08:35:52,671 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 170 2020-10-10 08:35:52,672 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 18 2020-10-10 08:35:52,672 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 52.6 2020-10-10 08:35:52,673 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 26.6 2020-10-10 08:35:52,673 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 11.2 2020-10-10 08:35:52,674 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 39.3 2020-10-10 08:35:52,674 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 1.21 2020-10-10 08:35:52,674 — rl_replicas.vpg.vpg — INFO — Total env interactions: 192000 2020-10-10 08:35:52,674 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0334 2020-10-10 08:35:52,675 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 128.757 2020-10-10 08:35:52,675 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.669 2020-10-10 08:35:52,675 — rl_replicas.vpg.vpg — INFO — Time: 158 2020-10-10 08:35:55,040 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 22 steps on the current episode 2020-10-10 08:35:55,690 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0495 2020-10-10 08:35:55,691 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 140 2020-10-10 08:35:55,691 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: -0.0074 2020-10-10 08:35:55,691 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: -17.7 2020-10-10 08:35:55,691 — rl_replicas.vpg.vpg — INFO — Epoch: 48 2020-10-10 08:35:55,692 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 49.8 2020-10-10 08:35:55,692 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 25.5 2020-10-10 08:35:55,692 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 187 2020-10-10 08:35:55,693 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 20 2020-10-10 08:35:55,693 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 49.4 2020-10-10 08:35:55,693 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 25.1 2020-10-10 08:35:55,693 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 11.3 2020-10-10 08:35:55,694 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 39.7 2020-10-10 08:35:55,694 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.599 2020-10-10 08:35:55,694 — rl_replicas.vpg.vpg — INFO — Total env interactions: 196000 2020-10-10 08:35:55,694 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0337 2020-10-10 08:35:55,695 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 128.994 2020-10-10 08:35:55,695 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.668 2020-10-10 08:35:55,695 — rl_replicas.vpg.vpg — INFO — Time: 161 2020-10-10 08:35:58,100 — rl_replicas.vpg.vpg — WARNING — The trajectory cut off at 15 steps on the current episode 2020-10-10 08:35:58,102 — rl_replicas.vpg.vpg — WARNING — Saving model is not implemented 2020-10-10 08:35:58,785 — rl_replicas.vpg.vpg — INFO — Loss of the current policy: -0.0443 2020-10-10 08:35:58,786 — rl_replicas.vpg.vpg — INFO — Loss of the current value function: 165 2020-10-10 08:35:58,786 — rl_replicas.vpg.vpg — INFO — Difference of the previous policy loss: 0.00519 2020-10-10 08:35:58,786 — rl_replicas.vpg.vpg — INFO — Difference of the previous value function loss: 24.3 2020-10-10 08:35:58,786 — rl_replicas.vpg.vpg — INFO — Epoch: 49 2020-10-10 08:35:58,787 — rl_replicas.vpg.vpg — INFO — Average Episode Return: 52.4 2020-10-10 08:35:58,787 — rl_replicas.vpg.vpg — INFO — Std Episode Return: 30.8 2020-10-10 08:35:58,788 — rl_replicas.vpg.vpg — INFO — Maximum Episode Return: 151 2020-10-10 08:35:58,788 — rl_replicas.vpg.vpg — INFO — Minimum Episode Return: 15 2020-10-10 08:35:58,788 — rl_replicas.vpg.vpg — INFO — Average Episode Length: 51.9 2020-10-10 08:35:58,789 — rl_replicas.vpg.vpg — INFO — Average Episode Value: 27.4 2020-10-10 08:35:58,789 — rl_replicas.vpg.vpg — INFO — Std Episode Value: 12.5 2020-10-10 08:35:58,790 — rl_replicas.vpg.vpg — INFO — Maximum Episode Value: 43.4 2020-10-10 08:35:58,790 — rl_replicas.vpg.vpg — INFO — Minimum Episode Value: 0.322 2020-10-10 08:35:58,790 — rl_replicas.vpg.vpg — INFO — Total env interactions: 200000 2020-10-10 08:35:58,791 — rl_replicas.vpg.vpg — INFO — Avarage Policy Loss: -0.0339 2020-10-10 08:35:58,791 — rl_replicas.vpg.vpg — INFO — Avarage Value function Loss: 129.709 2020-10-10 08:35:58,792 — rl_replicas.vpg.vpg — INFO — Avarage Entropy: 0.668 2020-10-10 08:35:58,792 — rl_replicas.vpg.vpg — INFO — Time: 164 39876567 function calls (37038025 primitive calls) in 165.460 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 1200150 20.928 0.000 20.928 0.000 {method 'matmul' of 'torch._C._TensorBase' objects} 4050 15.983 0.004 15.983 0.004 {method 'run_backward' of 'torch._C._EngineBase' objects} 200050 14.572 0.000 14.572 0.000 {method 'logsumexp' of 'torch._C._TensorBase' objects} 808200 14.204 0.000 14.204 0.000 {built-in method tanh} 3232800/404100 10.749 0.000 104.046 0.000 module.py:531(__call__) 1212300 10.586 0.000 43.588 0.000 functional.py:1355(linear) 12150 6.139 0.001 6.139 0.001 {built-in method addmm} 1 6.130 6.130 164.092 164.092 vpg.py:59(learn) 1212300 5.595 0.000 5.595 0.000 {method 't' of 'torch._C._TensorBase' objects} 200000 4.829 0.000 7.966 0.000 cartpole.py:91(step) 200050 4.007 0.000 19.496 0.000 categorical.py:44(__init__) 404100 3.690 0.000 77.790 0.000 container.py:90(forward) 1212300 3.579 0.000 48.290 0.000 linear.py:86(forward) ```

yamatokataoka commented 4 years ago

I found that it is not a loss function in the typical sense from supervised learning.

The data distribution depends on the most recent policy.
It doesn’t measure performance.

As you see here, the average return is increasing so let's only care about average return.

https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html

yamatokataoka / reinforcement-learning-replications

Value loss is Increasing #6