@vkakerbeck This PR includes the changes in the first one plus a few more commits. Major changes:
Refactors the update function, so logging and loss calculation are less intertwined. That will make it easier to maintain and keep evolving the codebase. That is the 80% of the PR.
Moves backprop through reward loss calculation inside update function, in place of the standard RL loss. Note in this case auxiliary loss is also being calculated. Not sure if that is what we wanted, but it is an easy change to remove it
Changed loss to be the inverse instead of negative. It gives us the behavior we want, minimize loss favors maximizing variances, while keeping loss positive and lower bounded at 0.
Pep8 compliance updates, will be required once we activate circleCI
It would be easier to review this PR only after merging the previous one, so it will only show the diff for the recent commits.
If you prefer you could also review this directly, with the changes combined, and discard the previous one. I would merge them separately though, since I've made a lot more changes in this one, and we want to be able to rollback these changes in case it introduces a bug that impact ongoing experiments.
edit: I've done some more changes to consider what was discussed on slack, with alternate steps of updating the dynamics model and the policy when backpropagating through rewards
@vkakerbeck This PR includes the changes in the first one plus a few more commits. Major changes:
It would be easier to review this PR only after merging the previous one, so it will only show the diff for the recent commits.
If you prefer you could also review this directly, with the changes combined, and discard the previous one. I would merge them separately though, since I've made a lot more changes in this one, and we want to be able to rollback these changes in case it introduces a bug that impact ongoing experiments.
edit: I've done some more changes to consider what was discussed on slack, with alternate steps of updating the dynamics model and the policy when backpropagating through rewards