Closed danielstankw closed 1 year ago
If I wish for my parameters/ action space to only have positive values its still recommended to scale the values in the [-1,1] range and then unscale them in my env?
yes, see https://youtu.be/Ikngt0_DXJg?t=738 for a longer explanation
How is it recommended to scale action space if I try to learn parameters and I do not want to limit the algorithm from learning and exploring?
I actually answer those questions in https://youtu.be/Ikngt0_DXJg (how to define a custom env, how to choose the action space)
How can I utilize the self.squash_output when using PPO,
you need to use gSDE (use_sde=True
, you can find some examples in the RL Zoo: https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/hyperparams/ppo.yml#L28) and then pass squash_output=True
via the net_kwargs
.
But by experience, it doesn't change much compared to clipping (default for PPO).
Anyway, for continuous actions, I would recommend you to use SAC/TD3/TQC instead of PPO (I also talk about that in the video), they all handled limits properly (with squashing).
Thank you very much for the explanation !
❓ Question
Hi, There are a few issues related to my problem but none of them answers my question so I am opening a new issue.
I am using PPO to learn parameters for my controller. Therefore action = parameters of controller. I saw that it is recommended to re-scale action space between [-1,1] and that later on I can change the scale in my environment back to the desired range. Questions
policies.py
it seems that when i useself.action_space = spaces.Box(low=low, high=high)
the action space isn't re-scaled between [-1, 1] but clipped. How can I utilize the self.squash_output when using PPO, I saw that it is one of the parameters of thepolicies.ActorCriticPolicy
but how can I modify it from the PPO class?Thank you for your help :)
Checklist