Closed TimZaman closed 5 years ago
more:
kubectl logs job16-optimizer-master-0
2019-01-29 12:20:35,046 INFO main(rmq_host=job16-rmq.default.svc.cluster.local, rmq_port=5672, epochs=4 seq_per_epoch=32, batch_size=8, seq_len=256 learning_rate=1e-05, pretrained_model=exp2/job14/model_000002970.pt, mq_prefetch_count=4, entropy_coef=0.0005)
2019-01-29 12:20:35,047 INFO init_distribution
2019-01-29 12:20:35,047 WARNING skipping distribution: world size too small (1)
2019-01-29 12:20:35,061 INFO Checkpointing to: exp2/job16
2019-01-29 12:20:35,715 INFO Found a latest model in pretrained dir: exp2/job16/model_000001345.pt
2019-01-29 12:20:35,716 WARNING Overriding pretrained model by latest model.
2019-01-29 12:20:35,716 INFO Downloading: exp2/job16/model_000001345.pt
2019-01-29 12:20:35,829 INFO Connected to RMQ
2019-01-29 12:20:35,966 INFO iteration 1346/10000
tzaman@Tims-Mac-Pro ~ $ kubectl logs job16-optimizer-master-0 -p
2019-01-29 12:18:39,109 INFO main(rmq_host=job16-rmq.default.svc.cluster.local, rmq_port=5672, epochs=4 seq_per_epoch=32, batch_size=8, seq_len=256 learning_rate=1e-05, pretrained_model=exp2/job14/model_000002970.pt, mq_prefetch_count=4, entropy_coef=0.0005)
2019-01-29 12:18:39,109 INFO init_distribution
2019-01-29 12:18:39,109 WARNING skipping distribution: world size too small (1)
2019-01-29 12:18:39,123 INFO Checkpointing to: exp2/job16
2019-01-29 12:18:39,678 INFO Found a latest model in pretrained dir: exp2/job16/model_000001345.pt
2019-01-29 12:18:39,678 WARNING Overriding pretrained model by latest model.
2019-01-29 12:18:39,678 INFO Downloading: exp2/job16/model_000001345.pt
2019-01-29 12:18:39,805 INFO Connected to RMQ
2019-01-29 12:18:39,982 INFO iteration 1346/10000
2019-01-29 12:18:42,580 INFO epoch 1/4
2019-01-29 12:18:47,466 INFO epoch 2/4
2019-01-29 12:18:52,237 INFO epoch 3/4
2019-01-29 12:18:56,930 INFO epoch 4/4
2019-01-29 12:19:01,720 INFO steps_per_s=420.13, avg_weight_age=2.7, reward_per_sec=0.0291, loss=nan, entropy=nan
Traceback (most recent call last):
File "optimizer.py", line 737, in <module>
run_local=args.run_local,
File "optimizer.py", line 693, in main
dota_optimizer.run()
File "optimizer.py", line 495, in run
self.writer.add_histogram('losses', losses, it)
File "/root/.local/lib/python3.7/site-packages/tensorboardX/writer.py", line 406, in add_histogram
histogram(tag, values, bins), global_step, walltime)
File "/root/.local/lib/python3.7/site-packages/tensorboardX/summary.py", line 146, in histogram
hist = make_histogram(values.astype(float), bins)
File "/root/.local/lib/python3.7/site-packages/tensorboardX/summary.py", line 168, in make_histogram
counts = counts[start:end]
UnboundLocalError: local variable 'start' referenced before assignment
$ kubectl logs job16-dotaservice-75c7df74f5-4f8j6
Error from server (BadRequest): a container name must be specified for pod job16-dotaservice-75c7df74f5-4f8j6, choose one of: [agent dotaservice]
tzaman@Tims-Mac-Pro ~ $ kubectl logs job16-dotaservice-75c7df74f5-4f8j6 agent
2019-01-30 07:54:23,144 INFO main(rmq_host=job16-rmq.default.svc.cluster.local, rmq_port=5672)
2019-01-30 07:54:23,159 INFO setup_model_cb(host=job16-rmq.default.svc.cluster.local, port=5672)
2019-01-30 07:54:23,190 INFO Received new model: version=0, size=1207838b
2019-01-30 07:54:23,195 INFO === Starting Game 0.
2019-01-30 07:54:23,195 INFO Starting game.
2019-01-30 07:54:23,203 INFO Player 0 using weights version 0
2019-01-30 07:54:23,210 INFO Player 5 using weights version 0
Traceback (most recent call last):
File "agent.py", line 718, in main
await game.play(game_id=game_id)
File "agent.py", line 641, in play
action_pb = player.obs_to_action(obs=obs)
File "agent.py", line 526, in obs_to_action
world_state=obs,
File "agent.py", line 479, in select_action
action_dict = Policy.select_actions(head_prob_dict=head_prob_dict)
File "/root/dotaclient/policy.py", line 214, in select_actions
action_dict['enum'] = cls.sample_action(head_prob_dict['enum'])
File "/root/dotaclient/policy.py", line 207, in sample_action
return Categorical(probs).sample()
File "/root/.local/lib/python3.7/site-packages/torch/distributions/categorical.py", line 110, in sample
sample_2d = torch.multinomial(probs_2d, 1, True)
RuntimeError: invalid argument 2: invalid multinomial distribution (encountering probability entry < 0) at /pytorch/aten/src/TH/generic/THTensorRandom.cpp:298
2019-01-30 07:54:30,099 ERROR Unclosed connection: Channel('127.0.0.1', 13337, ..., path=None)