Action bias is added twice in DDPG algorithm implementation, similar to #259

[sdpkjc]

Problem Description

A problem was reported in #259 , and the problem appears to exist in ddpg(ddpg_continuous_action.py & ddpg_continuous_action_jax.py) as well.

the action bias is added twice to the action. First, it is added during actor forward pass. Second, the random noise comes from a distribution with the center in actor.action_bias.

# In ddpg_continuous_action.py line176
# https://github.com/vwxyzjn/cleanrl/blob/17febbf88c28d3572c62ae6fa041921ef84290c4/cleanrl/ddpg_continuous_action.py#L176
actions += torch.normal(actor.action_bias, actor.action_scale * args.exploration_noise)

# In ddpg_continuous_action_jax.py line232
# (https://github.com/vwxyzjn/cleanrl/blob/17febbf88c28d3572c62ae6fa041921ef84290c4/cleanrl/ddpg_continuous_action_jax.py#L232)
jax.device_get(actions)[0] + np.random.normal(action_bias, action_scale * args.exploration_noise)[0]

Checklist

• [x] I have installed dependencies via poetry install (see CleanRL's installation guideline.
• [x] I have checked that there is no similar issue in the repo (required)

Possible Solution

• actions += torch.normal(actor.action_bias, actor.action_scale * args.exploration_noise) ➡️ actions += torch.normal(0, actor.action_scale * args.exploration_noise)
• jax.device_get(actions)[0] + np.random.normal(action_bias, action_scale * args.exploration_noise)[0] ➡️ jax.device_get(actions)[0] + np.random.normal(0, action_scale * args.exploration_noise)[0]