NaN in policy loss - Githubissues

$ kubectl logs job1-optimizer-master-0
2019-02-03 08:08:55,542 INFO     main(rmq_host=job1-rmq.default.svc.cluster.local, rmq_port=5672, epochs=4 seq_per_epoch=32, batch_size=8, seq_len=256 learning_rate=1e-05, pretrained_model=None, mq_prefetch_count=4, entropy_coef=0.0001)
2019-02-03 08:08:55,542 INFO     init_distribution
2019-02-03 08:08:55,543 WARNING  skipping distribution: world size too small (1)
2019-02-03 08:08:55,555 INFO     Checkpointing to: exp3/job1
2019-02-03 08:08:55,820 INFO     Found a latest model in pretrained dir: exp3/job1/model_000000018.pt
2019-02-03 08:08:55,821 INFO     Downloading: exp3/job1/model_000000018.pt
2019-02-03 08:08:55,967 INFO     Connected to RMQ
2019-02-03 08:08:56,118 INFO     iteration 19/10000
2019-02-03 08:09:04,984 INFO      epoch 1/4
2019-02-03 08:09:09,354 INFO      epoch 2/4
2019-02-03 08:09:13,669 INFO      epoch 3/4
2019-02-03 08:09:17,948 INFO      epoch 4/4
2019-02-03 08:09:22,256 INFO     steps_per_s=321.15, avg_weight_age=2.8, reward_per_sec=-0.0045, loss=0.0894, entropy=7.509, advantage=-0.047
2019-02-03 08:09:22,535 INFO     iteration 20/10000
2019-02-03 08:09:36,020 INFO      epoch 1/4
2019-02-03 08:09:40,070 INFO      epoch 2/4
2019-02-03 08:09:44,142 INFO      epoch 3/4
2019-02-03 08:09:48,175 INFO      epoch 4/4
2019-02-03 08:09:52,276 INFO     steps_per_s=273.46, avg_weight_age=2.8, reward_per_sec=-0.0024, loss=0.3443, entropy=7.537, advantage=-0.098
2019-02-03 08:09:52,509 INFO     iteration 21/10000
2019-02-03 08:10:05,592 INFO      epoch 1/4
2019-02-03 08:10:09,640 INFO      epoch 2/4
2019-02-03 08:10:13,671 INFO      epoch 3/4
2019-02-03 08:10:17,794 INFO      epoch 4/4
2019-02-03 08:10:22,531 INFO     steps_per_s=287.60, avg_weight_age=2.8, reward_per_sec=-0.0050, loss=0.3568, entropy=7.509, advantage=0.088
2019-02-03 08:10:22,784 INFO     iteration 22/10000
2019-02-03 08:10:33,289 INFO      epoch 1/4
2019-02-03 08:10:37,302 INFO      epoch 2/4
2019-02-03 08:10:41,340 INFO      epoch 3/4
2019-02-03 08:10:45,373 INFO      epoch 4/4
2019-02-03 08:10:49,536 INFO     steps_per_s=322.51, avg_weight_age=13.8, reward_per_sec=-0.0008, loss=0.2533, entropy=7.513, advantage=0.070
2019-02-03 08:10:49,779 INFO     iteration 23/10000
2019-02-03 08:11:01,499 INFO      epoch 1/4
2019-02-03 08:11:05,246 INFO      epoch 2/4
2019-02-03 08:11:09,120 INFO      epoch 3/4
2019-02-03 08:11:12,983 INFO      epoch 4/4
2019-02-03 08:11:16,889 INFO     steps_per_s=299.38, avg_weight_age=12.0, reward_per_sec=-0.0036, loss=0.3263, entropy=7.541, advantage=0.130
2019-02-03 08:11:17,135 INFO     iteration 24/10000
tzaman@Tims-Mac-Pro dotaclient (master) $ kubectl logs job1-optimizer-master-0 -p
2019-02-03 07:58:09,186 INFO     main(rmq_host=job1-rmq.default.svc.cluster.local, rmq_port=5672, epochs=4 seq_per_epoch=32, batch_size=8, seq_len=256 learning_rate=1e-05, pretrained_model=None, mq_prefetch_count=4, entropy_coef=0.0001)
2019-02-03 07:58:09,186 INFO     init_distribution
2019-02-03 07:58:09,186 WARNING  skipping distribution: world size too small (1)
2019-02-03 07:58:09,198 INFO     Checkpointing to: exp3/job1
2019-02-03 07:58:09,442 INFO     Connected to RMQ
2019-02-03 07:58:09,649 INFO     iteration 1/10000
2019-02-03 08:00:13,929 INFO      epoch 1/4
2019-02-03 08:00:18,560 INFO      epoch 2/4
2019-02-03 08:00:22,967 INFO      epoch 3/4
2019-02-03 08:00:27,461 INFO      epoch 4/4
2019-02-03 08:00:32,009 INFO     steps_per_s=59.25, avg_weight_age=0.0, reward_per_sec=-0.0082, loss=0.3138, entropy=7.515, advantage=0.227
2019-02-03 08:00:32,601 INFO     iteration 2/10000
2019-02-03 08:00:34,976 INFO      epoch 1/4
2019-02-03 08:00:39,304 INFO      epoch 2/4
2019-02-03 08:00:43,549 INFO      epoch 3/4
2019-02-03 08:00:47,804 INFO      epoch 4/4
2019-02-03 08:00:52,170 INFO     steps_per_s=407.36, avg_weight_age=1.0, reward_per_sec=-0.0057, loss=0.1834, entropy=7.549, advantage=0.153
2019-02-03 08:00:52,454 INFO     iteration 3/10000
2019-02-03 08:00:54,928 INFO      epoch 1/4
2019-02-03 08:00:58,679 INFO      epoch 2/4
2019-02-03 08:01:02,412 INFO      epoch 3/4
2019-02-03 08:01:06,143 INFO      epoch 4/4
2019-02-03 08:01:10,061 INFO     steps_per_s=457.79, avg_weight_age=2.0, reward_per_sec=-0.0045, loss=0.1381, entropy=7.549, advantage=0.109
2019-02-03 08:01:10,330 INFO     iteration 4/10000
2019-02-03 08:01:36,341 INFO      epoch 1/4
2019-02-03 08:01:40,406 INFO      epoch 2/4
2019-02-03 08:01:44,472 INFO      epoch 3/4
2019-02-03 08:01:48,500 INFO      epoch 4/4
2019-02-03 08:01:52,630 INFO     steps_per_s=204.48, avg_weight_age=3.0, reward_per_sec=-0.0052, loss=0.2404, entropy=7.518, advantage=0.120
2019-02-03 08:01:52,858 INFO     iteration 5/10000
2019-02-03 08:01:55,481 INFO      epoch 1/4
2019-02-03 08:01:59,453 INFO      epoch 2/4
2019-02-03 08:02:03,407 INFO      epoch 3/4
2019-02-03 08:02:07,419 INFO      epoch 4/4
2019-02-03 08:02:11,792 INFO     steps_per_s=441.39, avg_weight_age=4.0, reward_per_sec=-0.0048, loss=0.2777, entropy=7.482, advantage=0.130
2019-02-03 08:02:12,060 INFO     iteration 6/10000
2019-02-03 08:02:15,000 INFO      epoch 1/4
2019-02-03 08:02:19,372 INFO      epoch 2/4
2019-02-03 08:02:23,516 INFO      epoch 3/4
2019-02-03 08:02:27,732 INFO      epoch 4/4
2019-02-03 08:02:32,221 INFO     steps_per_s=425.83, avg_weight_age=5.0, reward_per_sec=-0.0029, loss=0.2681, entropy=7.498, advantage=-0.018
2019-02-03 08:02:32,489 INFO     iteration 7/10000
2019-02-03 08:03:00,428 INFO      epoch 1/4
2019-02-03 08:03:04,404 INFO      epoch 2/4
2019-02-03 08:03:08,522 INFO      epoch 3/4
2019-02-03 08:03:12,645 INFO      epoch 4/4
2019-02-03 08:03:16,870 INFO     steps_per_s=189.22, avg_weight_age=4.0, reward_per_sec=-0.0069, loss=0.2963, entropy=7.481, advantage=0.130
2019-02-03 08:03:17,154 INFO     iteration 8/10000
2019-02-03 08:03:19,849 INFO      epoch 1/4
2019-02-03 08:03:23,899 INFO      epoch 2/4
2019-02-03 08:03:28,062 INFO      epoch 3/4
2019-02-03 08:03:32,274 INFO      epoch 4/4
2019-02-03 08:03:36,499 INFO     steps_per_s=443.39, avg_weight_age=4.9, reward_per_sec=-0.0046, loss=0.2225, entropy=7.487, advantage=0.121
2019-02-03 08:03:36,767 INFO     iteration 9/10000
2019-02-03 08:03:40,356 INFO      epoch 1/4
2019-02-03 08:03:44,524 INFO      epoch 2/4
2019-02-03 08:03:48,800 INFO      epoch 3/4
2019-02-03 08:03:53,079 INFO      epoch 4/4
2019-02-03 08:03:57,397 INFO     steps_per_s=428.69, avg_weight_age=5.8, reward_per_sec=-0.0031, loss=0.1637, entropy=7.529, advantage=-0.027
2019-02-03 08:03:57,624 INFO     iteration 10/10000
2019-02-03 08:04:09,730 INFO      epoch 1/4
2019-02-03 08:04:13,922 INFO      epoch 2/4
2019-02-03 08:04:18,159 INFO      epoch 3/4
2019-02-03 08:04:22,414 INFO      epoch 4/4
2019-02-03 08:04:26,668 INFO     steps_per_s=297.42, avg_weight_age=4.4, reward_per_sec=-0.0052, loss=0.2684, entropy=7.551, advantage=0.137
2019-02-03 08:04:26,937 INFO     iteration 11/10000
2019-02-03 08:04:40,910 INFO      epoch 1/4
2019-02-03 08:04:44,869 INFO      epoch 2/4
2019-02-03 08:04:48,852 INFO      epoch 3/4
2019-02-03 08:04:52,930 INFO      epoch 4/4
2019-02-03 08:04:57,053 INFO     steps_per_s=277.99, avg_weight_age=5.0, reward_per_sec=-0.0055, loss=0.3511, entropy=7.519, advantage=0.157
2019-02-03 08:04:57,328 INFO     iteration 12/10000
2019-02-03 08:05:08,032 INFO      epoch 1/4
2019-02-03 08:05:12,169 INFO      epoch 2/4
2019-02-03 08:05:16,640 INFO      epoch 3/4
2019-02-03 08:05:20,895 INFO      epoch 4/4
2019-02-03 08:05:25,137 INFO     steps_per_s=309.99, avg_weight_age=5.4, reward_per_sec=-0.0043, loss=0.1677, entropy=7.552, advantage=0.002
2019-02-03 08:05:25,377 INFO     iteration 13/10000
2019-02-03 08:05:32,969 INFO      epoch 1/4
2019-02-03 08:05:36,927 INFO      epoch 2/4
2019-02-03 08:05:40,960 INFO      epoch 3/4
2019-02-03 08:05:45,058 INFO      epoch 4/4
2019-02-03 08:05:49,178 INFO     steps_per_s=351.36, avg_weight_age=4.3, reward_per_sec=-0.0046, loss=0.3550, entropy=7.511, advantage=0.136
2019-02-03 08:05:49,449 INFO     iteration 14/10000
2019-02-03 08:06:07,304 INFO      epoch 1/4
2019-02-03 08:06:11,466 INFO      epoch 2/4
2019-02-03 08:06:15,637 INFO      epoch 3/4
2019-02-03 08:06:19,784 INFO      epoch 4/4
2019-02-03 08:06:24,006 INFO     steps_per_s=242.56, avg_weight_age=4.4, reward_per_sec=-0.0027, loss=0.1183, entropy=7.534, advantage=0.041
2019-02-03 08:06:24,288 INFO     iteration 15/10000
2019-02-03 08:06:28,117 INFO      epoch 1/4
2019-02-03 08:06:32,300 INFO      epoch 2/4
2019-02-03 08:06:36,325 INFO      epoch 3/4
2019-02-03 08:06:40,490 INFO      epoch 4/4
2019-02-03 08:06:44,682 INFO     steps_per_s=408.59, avg_weight_age=4.5, reward_per_sec=-0.0066, loss=0.3987, entropy=7.405, advantage=0.203
2019-02-03 08:06:44,939 INFO     iteration 16/10000
2019-02-03 08:06:58,538 INFO      epoch 1/4
2019-02-03 08:07:02,430 INFO      epoch 2/4
2019-02-03 08:07:06,373 INFO      epoch 3/4
2019-02-03 08:07:10,318 INFO      epoch 4/4
2019-02-03 08:07:14,332 INFO     steps_per_s=276.31, avg_weight_age=4.3, reward_per_sec=-0.0044, loss=0.3123, entropy=7.525, advantage=0.166
2019-02-03 08:07:14,598 INFO     iteration 17/10000
2019-02-03 08:07:32,498 INFO      epoch 1/4
2019-02-03 08:07:36,640 INFO      epoch 2/4
2019-02-03 08:07:40,742 INFO      epoch 3/4
2019-02-03 08:07:44,962 INFO      epoch 4/4
2019-02-03 08:07:49,240 INFO     steps_per_s=249.25, avg_weight_age=4.9, reward_per_sec=-0.0031, loss=0.1578, entropy=7.548, advantage=-0.241
2019-02-03 08:07:49,485 INFO     iteration 18/10000
2019-02-03 08:07:52,115 INFO      epoch 1/4
2019-02-03 08:07:55,908 INFO      epoch 2/4
2019-02-03 08:07:59,834 INFO      epoch 3/4
2019-02-03 08:08:03,765 INFO      epoch 4/4
2019-02-03 08:08:07,733 INFO     steps_per_s=443.28, avg_weight_age=4.6, reward_per_sec=-0.0054, loss=0.3175, entropy=7.556, advantage=0.137
2019-02-03 08:08:07,999 INFO     iteration 19/10000
2019-02-03 08:08:15,437 INFO      epoch 1/4
2019-02-03 08:08:19,210 INFO      epoch 2/4
2019-02-03 08:08:23,094 INFO      epoch 3/4
2019-02-03 08:08:27,007 INFO      epoch 4/4
2019-02-03 08:08:30,989 INFO     steps_per_s=352.08, avg_weight_age=4.8, reward_per_sec=-0.0024, loss=0.2117, entropy=7.556, advantage=-0.033
2019-02-03 08:08:31,275 INFO     iteration 20/10000
2019-02-03 08:08:45,272 INFO      epoch 1/4
2019-02-03 08:08:49,297 INFO      epoch 2/4
Traceback (most recent call last):
  File "optimizer.py", line 746, in <module>
    run_local=args.run_local,
  File "optimizer.py", line 702, in main
    dota_optimizer.run()
  File "optimizer.py", line 441, in run
    loss, entropy_d, advantage = self.train(experiences=batch)
  File "optimizer.py", line 619, in train
    loss, policy_loss, entropy_loss, advantage_loss))
ValueError: loss=nan, policy_loss=nan, entropy_loss=0.00010979062062688172, advantage_loss=0.003007180755957961
TimZaman / dotaclient

NaN in policy loss #34
