Upon searching online, it seems like this happens when the memory is too small?
2019-07-25 04:09:29.063615: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-25 04:09:29.088399: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2712000000 Hz
2019-07-25 04:09:29.088872: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55ec3676cd00 executing computations on platform Host. Devices:
2019-07-25 04:09:29.088950: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
I0725 04:09:29.253317 139884285708032 parallel_py_environment.py:81] Spawning all processes.
I0725 04:09:31.808124 139884285708032 parallel_py_environment.py:88] All processes started.
W0725 04:09:40.928066 139884285708032 deprecation.py:323] From /home/rlee/git/agents/tf_agents/replay_buffers/tf_uniform_replay_buffer.py:540: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
Traceback (most recent call last):
File "tf_agents/agents/ppo/examples/v2/train_eval_atari.py", line 335, in <module>
app.run(main)
File "/home/rlee/anaconda3/envs/gsoc/lib/python3.7/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/home/rlee/anaconda3/envs/gsoc/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "tf_agents/agents/ppo/examples/v2/train_eval_atari.py", line 330, in main
num_eval_episodes=FLAGS.num_eval_episodes)
File "/home/rlee/anaconda3/envs/gsoc/lib/python3.7/site-packages/gin/config.py", line 1032, in wrapper
utils.augment_exception_message_and_reraise(e, err_str)
File "/home/rlee/anaconda3/envs/gsoc/lib/python3.7/site-packages/gin/utils.py", line 49, in augment_exception_message_and_reraise
six.raise_from(proxy.with_traceback(exception.__traceback__), None)
File "<string>", line 3, in raise_from
File "/home/rlee/anaconda3/envs/gsoc/lib/python3.7/site-packages/gin/config.py", line 1009, in wrapper
return fn(*new_args, **new_kwargs)
File "tf_agents/agents/ppo/examples/v2/train_eval_atari.py", line 247, in train_eval
tf_agent._init_rnd_normalizer(experience=trajectories)
File "/home/rlee/git/agents/tf_agents/agents/ppo/rndppo_agent.py", line 330, in _init_rnd_normalizer
intrinsic_rewards, _ = self.rnd_loss(time_steps, debug_summaries=self._debug_summaries)
File "/home/rlee/git/agents/tf_agents/agents/ppo/rndppo_agent.py", line 777, in rnd_loss
self._observation_clip_value)
File "/home/rlee/anaconda3/envs/gsoc/lib/python3.7/site-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/home/rlee/anaconda3/envs/gsoc/lib/python3.7/site-packages/tensorflow_core/python/ops/clip_ops.py", line 83, in clip_by_value
t_min = math_ops.minimum(values, clip_value_max)
File "/home/rlee/anaconda3/envs/gsoc/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_math_ops.py", line 6443, in minimum
_six.raise_from(_core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.NotFoundError: Could not find valid device for node.
Node:{{node Minimum}}
All kernels registered for op Minimum :
device='CPU'; T in [DT_INT64]
device='CPU'; T in [DT_INT32]
device='CPU'; T in [DT_DOUBLE]
device='CPU'; T in [DT_BFLOAT16]
device='CPU'; T in [DT_HALF]
device='CPU'; T in [DT_FLOAT]
device='XLA_CPU'; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_INT64, DT_BFLOAT16, DT_HALF]
device='XLA_CPU_JIT'; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_INT64, DT_BFLOAT16, DT_HALF]
[Op:Minimum]
In call to configurable 'train_eval' (<function train_eval at 0x7f39591f71e0>)
Upon searching online, it seems like this happens when the memory is too small?