search

google-deepmind / acme

A library of reinforcement learning components and agents
Apache License 2.0
3.52k stars 426 forks source link

ClippedGaussian wrong thing is clipped #228

Closed ahsimb closed 2 years ago

ahsimb commented 2 years ago

tf/networks/noise.py class ClippedGaussian

I think it's the added noise that should be clipped rather than the whole output. The min/max should be related to the noise scale.

nikolamomchev commented 2 years ago

@ahsimb thanks for the question. Whether you want to clip the output or the noise will depend on the use-case.

Consider the following use-case: Add noise to the continuous action for an environment which expects actions in [0,1]. If you were to do output = action + clip(noise) you may end up with output > 1 which would be invalid action for the environment. In this case you do want output = clip(action + noise).