2019-04-25 15:35:44.984853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N Y N N
2019-04-25 15:35:44.984857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: Y N N N
2019-04-25 15:35:44.984860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 2: N N N Y
2019-04-25 15:35:44.984864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 3: N N Y N
2019-04-25 15:35:44.986746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5397 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2019-04-25 15:35:44.987684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10419 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
2019-04-25 15:35:44.988591: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10419 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:82:00.0, compute capability: 6.1)
2019-04-25 15:35:44.988889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10419 MB memory) -> physical GPU (device: 3, name: GeForce GTX 1080 Ti, pci bus id: 0000:83:00.0, compute capability: 6.1)
INFO:tensorflow:
Epoch 1 phase train (phase step 0, global step 0).
2019-04-25 15:36:05.689807: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 347 of 500
2019-04-25 15:36:09.534239: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:136] Shuffle buffer filled.
step/score/loss/zs_entropy/zs_divergence = [0, -nan, 9180.21777, 35.6603241, 3.48616147]
step/score/loss/zs_entropy/zs_divergence = [15, -nan, 9161.94629, 35.667408, 3.3734169]
WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 84 (sensor.other.collision)
WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 81 (sensor.camera.rgb)
WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 83 (sensor.camera.semantic_segmentation)
WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 82 (sensor.camera.depth)
1 current speed 0.24948974946139205 collision 0
2019-04-25 15:36:14.662652: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at tensor_array_ops.cc:447 : Invalid argument: TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False).
WARNING:tensorflow:Worker 23dc2c5e-dcca-491d-9cf5-29a62e8258ab run 00001: Exception:
Traceback (most recent call last):
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False).
[[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3}} = TensorArrayWriteV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3/Enter, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1:1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/add_1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_9:1)]]
[[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1/_293}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge1531...e/Switch_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/yan/carla94/scenario_runner/planet/planet/training/running.py", line 199, in iter
for value in self._process_fn(self._logdir, *args):
File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 93, in process
training.define_model, dataset, logdir, config):
File "/home/yan/carla94/scenario_runner/planet/planet/training/utility.py", line 179, in train
for score in trainer.iterate(config.max_steps):
File "/home/yan/carla94/scenario_runner/planet/planet/training/trainer.py", line 201, in iterate
summary, mean_score, global_step = sess.run(phase.op, phase.feed)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False).
[[node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3 (defined at /home/yan/carla94/scenario_runner/planet/planet/control/simulate.py:78) = TensorArrayWriteV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3/Enter, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1:1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/add_1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_9:1)]]
[[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1/_293}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge1531...e/Switch_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Caused by op 'graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3', defined at:
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1741, in
main()
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/yan/carla94/scenariorunner/planet/planet/scripts/train.py", line 138, in
tf.app.run(lambda : main(args_), remaining) # tf.app.run(main, argv): Runs the program with an optional 'main' function and 'argv' list.
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/yan/carla94/scenariorunner/planet/planet/scripts/train.py", line 138, in
tf.app.run(lambda : main(args_), remaining) # tf.app.run(main, argv): Runs the program with an optional 'main' function and 'argv' list.
File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 108, in main
for unused_score in run: # the method iter(self) defined in run.
File "/home/yan/carla94/scenario_runner/planet/planet/training/running.py", line 199, in iter
for value in self._process_fn(self._logdir, args):
File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 93, in process
training.define_model, dataset, logdir, config):
File "/home/yan/carla94/scenario_runner/planet/planet/training/utility.py", line 160, in train
score, summary = model_fn(data, trainer, config) # model_fn is training.define_model
File "/home/yan/carla94/scenario_runner/planet/planet/training/define_model.py", line 137, in define_model
name='shouldcollect' + params.task.name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(args, kwargs)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2086, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1930, in BuildCondBranch
original_result = fn()
File "/home/yan/carla94/scenario_runner/planet/planet/training/utility.py", line 260, in simulate_episodes
1, agent_config, name=name)
File "/home/yan/carla94/scenario_runner/planet/planet/control/simulate.py", line 42, in simulate
env_processes=env_processes)
File "/home/yan/carla94/scenario_runner/planet/planet/control/simulate.py", line 78, in collect_rollouts
initializer, parallel_iterations=1)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 718, in scan
maximum_iterations=n)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3291, in while_loop
return_same_structure)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3004, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2939, in _BuildLoop
body_result = body(packed_vars_for_body)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3260, in
body = lambda i, lv: (i + 1, orig_body(lv))
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 701, in compute
tas = [ta.write(i, value) for (ta, value) in zip(tas, flat_a_out)]
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 701, in
tas = [ta.write(i, value) for (ta, value) in zip(tas, flat_a_out)]
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped
return _add_should_use_warning(fn(*args, *kwargs))
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 842, in write
return self._implementation.write(index, value, name=name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped
return _add_should_use_warning(fn(args, kwargs))
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 276, in write
name=name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 7444, in tensor_array_write_v3
flow_in=flow_in, name=name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False).
[[node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3 (defined at /home/yan/carla94/scenario_runner/planet/planet/control/simulate.py:78) = TensorArrayWriteV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3/Enter, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1:1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/add_1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_9:1)]]
[[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1/_293}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge1531...e/Switch_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
WARNING:tensorflow:Worker 23dc2c5e-dcca-491d-9cf5-29a62e8258ab run 00001: Failed.
Traceback (most recent call last):
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False).
[[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3}} = TensorArrayWriteV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3/Enter, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1:1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/add_1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_9:1)]]
[[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1/_293}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge1531...e/Switch_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1741, in
main()
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/yan/carla94/scenariorunner/planet/planet/scripts/train.py", line 138, in
tf.app.run(lambda : main(args_), remaining) # tf.app.run(main, argv): Runs the program with an optional 'main' function and 'argv' list.
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/yan/carla94/scenariorunner/planet/planet/scripts/train.py", line 138, in
tf.app.run(lambda : main(args_), remaining) # tf.app.run(main, argv): Runs the program with an optional 'main' function and 'argv' list.
File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 108, in main
for unused_score in run: # the method iter(self) defined in run.
File "/home/yan/carla94/scenario_runner/planet/planet/training/running.py", line 210, in iter
raise e
File "/home/yan/carla94/scenario_runner/planet/planet/training/running.py", line 199, in iter
for value in self._process_fn(self._logdir, *args):
File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 93, in process
training.define_model, dataset, logdir, config):
File "/home/yan/carla94/scenario_runner/planet/planet/training/utility.py", line 179, in train
for score in trainer.iterate(config.max_steps):
File "/home/yan/carla94/scenario_runner/planet/planet/training/trainer.py", line 201, in iterate
summary, mean_score, global_step = sess.run(phase.op, phase.feed)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False).
[[node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3 (defined at /home/yan/carla94/scenario_runner/planet/planet/control/simulate.py:78) = TensorArrayWriteV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3/Enter, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1:1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/add_1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_9:1)]]
[[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1/_293}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge1531...e/Switch_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Caused by op 'graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3', defined at:
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1741, in
main()
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/yan/carla94/scenariorunner/planet/planet/scripts/train.py", line 138, in
tf.app.run(lambda : main(args_), remaining) # tf.app.run(main, argv): Runs the program with an optional 'main' function and 'argv' list.
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/yan/carla94/scenariorunner/planet/planet/scripts/train.py", line 138, in
tf.app.run(lambda : main(args_), remaining) # tf.app.run(main, argv): Runs the program with an optional 'main' function and 'argv' list.
File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 108, in main
for unused_score in run: # the method iter(self) defined in run.
File "/home/yan/carla94/scenario_runner/planet/planet/training/running.py", line 199, in iter
for value in self._process_fn(self._logdir, args):
File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 93, in process
training.define_model, dataset, logdir, config):
File "/home/yan/carla94/scenario_runner/planet/planet/training/utility.py", line 160, in train
score, summary = model_fn(data, trainer, config) # model_fn is training.define_model
File "/home/yan/carla94/scenario_runner/planet/planet/training/define_model.py", line 137, in define_model
name='shouldcollect' + params.task.name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(args, kwargs)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2086, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1930, in BuildCondBranch
original_result = fn()
File "/home/yan/carla94/scenario_runner/planet/planet/training/utility.py", line 260, in simulate_episodes
1, agent_config, name=name)
File "/home/yan/carla94/scenario_runner/planet/planet/control/simulate.py", line 42, in simulate
env_processes=env_processes)
File "/home/yan/carla94/scenario_runner/planet/planet/control/simulate.py", line 78, in collect_rollouts
initializer, parallel_iterations=1)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 718, in scan
maximum_iterations=n)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3291, in while_loop
return_same_structure)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3004, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2939, in _BuildLoop
body_result = body(packed_vars_for_body)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3260, in
body = lambda i, lv: (i + 1, orig_body(lv))
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 701, in compute
tas = [ta.write(i, value) for (ta, value) in zip(tas, flat_a_out)]
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 701, in
tas = [ta.write(i, value) for (ta, value) in zip(tas, flat_a_out)]
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped
return _add_should_use_warning(fn(*args, *kwargs))
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 842, in write
return self._implementation.write(index, value, name=name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped
return _add_should_use_warning(fn(args, kwargs))
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 276, in write
name=name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 7444, in tensor_array_write_v3
flow_in=flow_in, name=name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False).
[[node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3 (defined at /home/yan/carla94/scenario_runner/planet/planet/control/simulate.py:78) = TensorArrayWriteV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3/Enter, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1:1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/add_1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_9:1)]]
[[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1/_293}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge1531...e/Switch_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
yanyuegang
commented
5 years ago 
2019-04-25 15:35:44.984853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N Y N N 2019-04-25 15:35:44.984857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: Y N N N 2019-04-25 15:35:44.984860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 2: N N N Y 2019-04-25 15:35:44.984864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 3: N N Y N 2019-04-25 15:35:44.986746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5397 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1) 2019-04-25 15:35:44.987684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10419 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1) 2019-04-25 15:35:44.988591: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10419 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:82:00.0, compute capability: 6.1) 2019-04-25 15:35:44.988889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10419 MB memory) -> physical GPU (device: 3, name: GeForce GTX 1080 Ti, pci bus id: 0000:83:00.0, compute capability: 6.1) INFO:tensorflow:
Epoch 1 phase train (phase step 0, global step 0). 2019-04-25 15:36:05.689807: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 347 of 500 2019-04-25 15:36:09.534239: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:136] Shuffle buffer filled. step/score/loss/zs_entropy/zs_divergence = [0, -nan, 9180.21777, 35.6603241, 3.48616147] step/score/loss/zs_entropy/zs_divergence = [15, -nan, 9161.94629, 35.667408, 3.3734169] WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 84 (sensor.other.collision) WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 81 (sensor.camera.rgb) WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 83 (sensor.camera.semantic_segmentation) WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 82 (sensor.camera.depth) 1 current speed 0.24948974946139205 collision 0 2019-04-25 15:36:14.662652: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at tensor_array_ops.cc:447 : Invalid argument: TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False). WARNING:tensorflow:Worker 23dc2c5e-dcca-491d-9cf5-29a62e8258ab run 00001: Exception: Traceback (most recent call last): File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False). [[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3}} = TensorArrayWriteV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3/Enter, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1:1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/add_1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_9:1)]] [[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1/_293}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge1531...e/Switch_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/yan/carla94/scenario_runner/planet/planet/training/running.py", line 199, in iter for value in self._process_fn(self._logdir, *args): File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 93, in process training.define_model, dataset, logdir, config): File "/home/yan/carla94/scenario_runner/planet/planet/training/utility.py", line 179, in train for score in trainer.iterate(config.max_steps): File "/home/yan/carla94/scenario_runner/planet/planet/training/trainer.py", line 201, in iterate summary, mean_score, global_step = sess.run(phase.op, phase.feed) File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False). [[node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3 (defined at /home/yan/carla94/scenario_runner/planet/planet/control/simulate.py:78) = TensorArrayWriteV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3/Enter, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1:1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/add_1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_9:1)]] [[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1/_293}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge1531...e/Switch_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Caused by op 'graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3', defined at: File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1741, in
main()
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/yan/carla94/scenariorunner/planet/planet/scripts/train.py", line 138, in
tf.app.run(lambda : main(args_), remaining) # tf.app.run(main, argv): Runs the program with an optional 'main' function and 'argv' list.
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/yan/carla94/scenariorunner/planet/planet/scripts/train.py", line 138, in
tf.app.run(lambda : main(args_), remaining) # tf.app.run(main, argv): Runs the program with an optional 'main' function and 'argv' list.
File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 108, in main
for unused_score in run: # the method iter(self) defined in run.
File "/home/yan/carla94/scenario_runner/planet/planet/training/running.py", line 199, in iter
for value in self._process_fn(self._logdir, args):
File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 93, in process
training.define_model, dataset, logdir, config):
File "/home/yan/carla94/scenario_runner/planet/planet/training/utility.py", line 160, in train
score, summary = model_fn(data, trainer, config) # model_fn is training.define_model
File "/home/yan/carla94/scenario_runner/planet/planet/training/define_model.py", line 137, in define_model
name='shouldcollect' + params.task.name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(args, kwargs)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2086, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1930, in BuildCondBranch
original_result = fn()
File "/home/yan/carla94/scenario_runner/planet/planet/training/utility.py", line 260, in simulate_episodes
1, agent_config, name=name)
File "/home/yan/carla94/scenario_runner/planet/planet/control/simulate.py", line 42, in simulate
env_processes=env_processes)
File "/home/yan/carla94/scenario_runner/planet/planet/control/simulate.py", line 78, in collect_rollouts
initializer, parallel_iterations=1)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 718, in scan
maximum_iterations=n)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3291, in while_loop
return_same_structure)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3004, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2939, in _BuildLoop
body_result = body(packed_vars_for_body)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3260, in
body = lambda i, lv: (i + 1, orig_body( lv))
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 701, in compute
tas = [ta.write(i, value) for (ta, value) in zip(tas, flat_a_out)]
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 701, in
tas = [ta.write(i, value) for (ta, value) in zip(tas, flat_a_out)]
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped
return _add_should_use_warning(fn(*args, *kwargs))
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 842, in write
return self._implementation.write(index, value, name=name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped
return _add_should_use_warning(fn(args, kwargs))
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 276, in write
name=name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 7444, in tensor_array_write_v3
flow_in=flow_in, name=name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False). [[node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3 (defined at /home/yan/carla94/scenario_runner/planet/planet/control/simulate.py:78) = TensorArrayWriteV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3/Enter, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1:1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/add_1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_9:1)]] [[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1/_293}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge1531...e/Switch_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
WARNING:tensorflow:Worker 23dc2c5e-dcca-491d-9cf5-29a62e8258ab run 00001: Failed. Traceback (most recent call last): File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False). [[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3}} = TensorArrayWriteV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3/Enter, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1:1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/add_1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_9:1)]] [[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1/_293}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge1531...e/Switch_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1741, in
main()
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/yan/carla94/scenariorunner/planet/planet/scripts/train.py", line 138, in
tf.app.run(lambda : main(args_), remaining) # tf.app.run(main, argv): Runs the program with an optional 'main' function and 'argv' list.
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/yan/carla94/scenariorunner/planet/planet/scripts/train.py", line 138, in
tf.app.run(lambda : main(args_), remaining) # tf.app.run(main, argv): Runs the program with an optional 'main' function and 'argv' list.
File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 108, in main
for unused_score in run: # the method iter(self) defined in run.
File "/home/yan/carla94/scenario_runner/planet/planet/training/running.py", line 210, in iter
raise e
File "/home/yan/carla94/scenario_runner/planet/planet/training/running.py", line 199, in iter
for value in self._process_fn(self._logdir, *args):
File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 93, in process
training.define_model, dataset, logdir, config):
File "/home/yan/carla94/scenario_runner/planet/planet/training/utility.py", line 179, in train
for score in trainer.iterate(config.max_steps):
File "/home/yan/carla94/scenario_runner/planet/planet/training/trainer.py", line 201, in iterate
summary, mean_score, global_step = sess.run(phase.op, phase.feed)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False).
[[node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3 (defined at /home/yan/carla94/scenario_runner/planet/planet/control/simulate.py:78) = TensorArrayWriteV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3/Enter, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1:1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/add_1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_9:1)]]
[[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1/_293}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge1531...e/Switch_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Caused by op 'graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3', defined at: File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1741, in
main()
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/yan/pycharm-community-2019.1.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/yan/carla94/scenariorunner/planet/planet/scripts/train.py", line 138, in
tf.app.run(lambda : main(args_), remaining) # tf.app.run(main, argv): Runs the program with an optional 'main' function and 'argv' list.
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/yan/carla94/scenariorunner/planet/planet/scripts/train.py", line 138, in
tf.app.run(lambda : main(args_), remaining) # tf.app.run(main, argv): Runs the program with an optional 'main' function and 'argv' list.
File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 108, in main
for unused_score in run: # the method iter(self) defined in run.
File "/home/yan/carla94/scenario_runner/planet/planet/training/running.py", line 199, in iter
for value in self._process_fn(self._logdir, args):
File "/home/yan/carla94/scenario_runner/planet/planet/scripts/train.py", line 93, in process
training.define_model, dataset, logdir, config):
File "/home/yan/carla94/scenario_runner/planet/planet/training/utility.py", line 160, in train
score, summary = model_fn(data, trainer, config) # model_fn is training.define_model
File "/home/yan/carla94/scenario_runner/planet/planet/training/define_model.py", line 137, in define_model
name='shouldcollect' + params.task.name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(args, kwargs)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2086, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1930, in BuildCondBranch
original_result = fn()
File "/home/yan/carla94/scenario_runner/planet/planet/training/utility.py", line 260, in simulate_episodes
1, agent_config, name=name)
File "/home/yan/carla94/scenario_runner/planet/planet/control/simulate.py", line 42, in simulate
env_processes=env_processes)
File "/home/yan/carla94/scenario_runner/planet/planet/control/simulate.py", line 78, in collect_rollouts
initializer, parallel_iterations=1)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 718, in scan
maximum_iterations=n)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3291, in while_loop
return_same_structure)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3004, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2939, in _BuildLoop
body_result = body(packed_vars_for_body)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3260, in
body = lambda i, lv: (i + 1, orig_body( lv))
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 701, in compute
tas = [ta.write(i, value) for (ta, value) in zip(tas, flat_a_out)]
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 701, in
tas = [ta.write(i, value) for (ta, value) in zip(tas, flat_a_out)]
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped
return _add_should_use_warning(fn(*args, *kwargs))
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 842, in write
return self._implementation.write(index, value, name=name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped
return _add_should_use_warning(fn(args, kwargs))
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 276, in write
name=name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 7444, in tensor_array_write_v3
flow_in=flow_in, name=name)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/yan/myenv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): TensorArray graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/TensorArray_3_110: Could not write to TensorArray index 0 because the value shape is [1,96,96,5] which is incompatible with the TensorArray's inferred element shape: [1,96,96,7] (consider setting infer_shape=False). [[node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3 (defined at /home/yan/carla94/scenario_runner/planet/planet/control/simulate.py:78) = TensorArrayWriteV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/TensorArrayWrite_2/TensorArrayWriteV3/Enter, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1:1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/add_1, graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_9:1)]] [[{{node graph/collection/should_collect_carla/simulate-1/train-carla-cem-12/scan/while/Switch_1/_293}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge1531...e/Switch_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]