search

seungjaeryanlee / agents

TF-Agents is a library for Reinforcement Learning in TensorFlow
Apache License 2.0
1 stars 0 forks source link

Unexpected TensorBoard behavior #2

Closed seungjaeryanlee closed 5 years ago

seungjaeryanlee commented 5 years ago

Thank you for the great feedback for my last blocker (#1). I moved the RND network to the agent so that it can be trained. To do so, I created a RndDqnAgent class that inherits the DqnAgent class. The _loss() function has been modified to compute RND loss as well, which is used in _train() function as intrinsic reward and to train the RND network.

However, I am noticing a strange behavior for TensorBoard.

Screenshot from 2019-06-28 18-09-29

  1. The value is only updated every 1k steps, and
  2. RND loss is not being plotted even though it summary.scalar() is invoked.

Do you have any ideas about what could be the cause of the problem?

(Please ignore the changes in rnd_wrapper.py and nest_utils.py, since they will be reverted.)

seungjaeryanlee commented 5 years ago

Closing this PR since all feedback has been incorporated and all questions have been resolved. Thank you so much!