Error using parameter train_step_counter according to colab example

[ideenfix]

I'm using TF Agent (nightly, 0.2.0dev2019430 on Win10 and TF2.0 (GPU, 2.0.0a0).

If you run the snippet according to colab example

` train_step_counter = tf.Variable(0)

tf_agent = dqn_agent.DqnAgent(
    train_env.time_step_spec(),
    train_env.action_spec(),
    q_network=net,
    optimizer=optimizer,
    epsilon_greedy=params["epsilon_final"],
    gamma=params['gamma'],
    td_errors_loss_fn=dqn_agent.element_wise_squared_loss,
    train_step_counter=train_step_counter
)

`

After calling DqnAgent.train following error is thrown

Traceback (most recent call last): File "C:\Program Files\JetBrains\PyCharm 2019.1\helpers\pydev\pydevd.py", line 1741, in main() File "C:\Program Files\JetBrains\PyCharm 2019.1\helpers\pydev\pydevd.py", line 1735, in main globals = debugger.run(setup['file'], None, None, is_module) File "C:\Program Files\JetBrains\PyCharm 2019.1\helpers\pydev\pydevd.py", line 1135, in run pydev_imports.execfile(file, globals, locals) # execute the script File "C:\Program Files\JetBrains\PyCharm 2019.1\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "D:/git/Deep-Reinforcement-Learning-Hands-On/Chapter07/01_dqn_basic_tf.py", line 165, in train_loss = tf_agent.train(experience) File "D:\pyenv\tf2\lib\site-packages\tf_agents\agents\tf_agent.py", line 177, in train loss_info = self._train_fn(experience=experience, weights=weights) File "D:\pyenv\tf2\lib\site-packages\tf_agents\agents\dqn\dqn_agent.py", line 256, in _train weights=weights) File "D:\pyenv\tf2\lib\site-packages\tf_agents\agents\dqn\dqn_agent.py", line 353, in loss name='loss', data=loss, step=self.train_step_counter) File "D:\pyenv\tf2\lib\site-packages\tensorboard\plugins\scalar\summary_v2.py", line 65, in scalar metadata=summary_metadata) File "D:\pyenv\tf2\lib\site-packages\tensorflow\python\ops\summary_ops_v2.py", line 632, in write _should_record_summaries_v2(), record, _nothing, name="summary_cond") File "D:\pyenv\tf2\lib\site-packages\tensorflow\python\framework\smart_cond.py", line 54, in smart_cond return true_fn() File "D:\pyenv\tf2\lib\site-packages\tensorflow\python\ops\summary_ops_v2.py", line 627, in record name=scope) File "D:\pyenv\tf2\lib\site-packages\tensorflow\python\ops\gen_summary_ops.py", line 793, in write_summary writer, step, tensor, tag, summary_metadata, name=name, ctx=_ctx) File "D:\pyenv\tf2\lib\site-packages\tensorflow\python\ops\gen_summary_ops.py", line 824, in write_summary_eager_fallback step = _ops.convert_to_tensor(step, _dtypes.int64) File "D:\pyenv\tf2\lib\site-packages\tensorflow\python\framework\ops.py", line 1050, in convert_to_tensor return convert_to_tensor_v2(value, dtype, preferred_dtype, name) File "D:\pyenv\tf2\lib\site-packages\tensorflow\python\framework\ops.py", line 1108, in convert_to_tensor_v2 as_ref=False) File "D:\pyenv\tf2\lib\site-packages\tensorflow\python\framework\ops.py", line 1186, in internal_convert_to_tensor ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref) File "D:\pyenv\tf2\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py", line 1420, in _dense_var_to_tensor return var._dense_var_to_tensor(dtype=dtype, name=name, as_ref=as_ref) # pylint: disable=protected-access File "D:\pyenv\tf2\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py", line 1371, in _dense_var_to_tensor "of type {!r}".format(dtype.name, self.dtype.name)) ValueError: Incompatible type conversion requested to type 'int64' for variable of type 'int32'

if you would change the initialization of this parameter to

train_step_counter = tf.Variable(0, dtype=tf.int64)

then you have no Problems

01_dqn_basic_tf_selfrunning.txt

[egonina]

global step needs to be of type tf.int64. Could you point to the colab example you're referring to so we can fix it if that's an issue there?

[ideenfix]

https://github.com/tensorflow/agents/blob/master/tf_agents/colabs/1_dqn_tutorial.ipynb

[egonina]

Hm, I'm unable to reproduce this by running the colab you linked, the train step runs fine and the type of train_step_counter variable is tf.int32. Are you running this colab directly or are you modifying anything in your code?

[ideenfix]

At first, i use the colab example as a reference for an own RL agent.

I'm checking the colab example on my notebook after installing the jupyter package. The colab example is running after switching off some import statements as these would not run with Win10 (e.g. pyvirtualdisplay or display = pyvirtualdisplay.Display(visible=0, size=(1400, 900)).start()).

Then I have checked my own script and got some new errors like

D:\pyenv\py36tf2\Scripts\python.exe D:/git/Deep-Reinforcement-Learning-Hands-On/Chapter07/01_dqn_basic_tf.py Python:3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 03:37:03) [MSC v.1900 64 bit (AMD64)] Tensorflow: 1.14.1-dev20190603 TF-Agent:0.2.0 Traceback (most recent call last): File "D:/git/Deep-Reinforcement-Learning-Hands-On/Chapter07/01_dqn_basic_tf.py", line 62, in writer = tf.summary.create_file_writer(log_dir) File "D:\pyenv\py36tf2\lib\site-packages\tensorflow\python\util\deprecation_wrapper.py", line 104, in getattr attr = getattr(self._dw_wrapped_module, name) AttributeError: module 'tensorflow._api.v1.summary' has no attribute 'create_file_writer'

After checking my virtual environment I stated that after running the colab example tf-nightly was installed. Attached the pip list output excerpt tb-nightly 1.14.0a20190602 tensorflow-estimator-2.0-preview 1.14.0.dev2019060300 termcolor 1.1.0 terminado 0.8.2 testpath 0.4.2 tf-agents-nightly 0.2.0.dev20190528 tf-estimator-nightly 1.14.0.dev2019052901 tf-nightly 1.14.1.dev20190603 tf-nightly-gpu-2.0-preview 2.0.0.dev20190602 tfp-nightly 0.8.0.dev20190603

This has overwritten the tf-nightly-gpu-2.0-preview package preference and was the reason for the last error regarding to the attribute error as my script based on TF2.0.

Attached you can find the standalone version of my own script 01_dqn_basic_tf_standalone.txt

If you would run this script in a pure TF2 environment e.g. Python:3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] Tensorflow: 2.0.0-alpha0

than this script runs until the first training step and train_loss = tf_agent.train(experience) throws above mentioned error which can only be corrected after changing the initialization to train_step_counter = tf.Variable(0, dtype=tf.int64)

PS: Sorry last week I was on vacation leave in the Mediterranean sea.

[bartmaciszewski]

I came across the same error when trying to write summaries to Tensorboard. The fix proposed by @ideenfix to change the step counter fixed the issue.

train_step_counter = tf.Variable(0, dtype=tf.int64)

Thanks!