search

StepNeverStop / RLs

Reinforcement Learning Algorithms Based on PyTorch
https://stepneverstop.github.io
Apache License 2.0
449 stars 93 forks source link

Error applying gradient for some algorithms #8

Closed kmakeev closed 5 years ago

kmakeev commented 5 years ago

OS - Ubuntu 19.4

print(tfp.version) 0.9.0-dev20191113 print(tf.version) 2.1.0-dev20191111

In pg, ac a2c algorithms, I get the error:

in converted code: /home/konstantin/IdeaProjects/RLs/Algorithms/tf2algos/a2c.py:158 train * self.optimizer_actor.apply_gradients( /home/konstantin/anaconda3/envs/tsfl2/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py:434 apply_gradients self._create_slots(var_list) /home/konstantin/anaconda3/envs/tsfl2/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/adam.py:149 _create_slots self.add_slot(var, 'm') /home/konstantin/anaconda3/envs/tsfl2/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py:574 add_slot var_key = _var_key(var) /home/konstantin/anaconda3/envs/tsfl2/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py:1065 _var_key if var._in_graph_mode:

AttributeError: 'ListWrapper' object has no attribute '_in_graph_mode'

But with the algorithms dqn, ddqn, dddqn, dpg... everything works.

After writing this code, the error disappears:

self.optimizer_actor.apply_gradients( zip(actor_grads, self.actor_net.trainable_variables + self.log_std) )

StepNeverStop commented 5 years ago

Hi, I haven't run those algorithms in python3.7, tf 2.1.0 and on ubuntu 19.04, not sure if it is related to these running environments.

BTW, could you tell me the training environment you used, so that I can see if I can reproduce the problem.

And also, I am not sure if you have used my latest submitted code.

Now in ac, a2c, I have written this

self.optimizer_actor.apply_gradients(
                zip(actor_grads, self.actor_net.trainable_variables + [self.log_std])
            )

It looks very similar to how you solved the problem, you can try again, or submit a PR. thx.

kmakeev commented 5 years ago

OK!. I will create a new branch, make changes to it and send you a Pull Request tonight. I used the BipedalWalker-v2 Box2D Gym environment

kmakeev commented 5 years ago

PG created.

StepNeverStop commented 5 years ago

Thank you very much for you PR! Feel free to reopen this issue if still any problem about this!