[Question] Does the action clipping method affect model's performance？

[BlueBlueGrey]

Important Note: We do not do technical support, nor consulting and don't answer personal questions per email. Please post your question on the RL Discord, Reddit or Stack Overflow in that case.

Question

I find on_policy_algorithm use np.clip to avoid out of bound error, however, off_policy_algorithm use Tanh activation function make actions zoom to (- 1,1). Does the action clipping method affect model's performance？

Additional context

Add any other context about the question here.

Checklist

• [x] I have read the documentation (required)
• [x] I have checked that there is no similar issue in the repo (required)

[Miffyli]

@araffin can give more detailed response (this is his area of expertise), but as far as I know, these are the default approaches taken by other researchers in continuous action spaces, so I feel they are a pretty robust choice.

[araffin]

Does the action clipping method affect the effect of the model？

what do you mean exactly? does it affect its performance? As long as your action space is normalized, I couldn't find evidence it had an impact yet (you can actually also activate proper bound handling with on policy algorithms when using gSDE by passing squash_output, see https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/policies.py#L400)

[ghost]

@BlueBlueGrey The answer is no in my experience. The more elegant tanh (squash_output=True) might sometimes perform empirically worse than clipping due to its difficulty to approach the boundaries or due to its sensitivity around zero. I have not found situations where one works and the other doesn't yet, so I would personally treat the problem as an implementation detail.