ACER raises an error when GaussianHeadWithFixedCovariance is used - Githubissues

pfnet / pfrl

PFRL: a PyTorch-based deep reinforcement learning library

MIT License

1.18k stars 158 forks source link

ACER raises an error when GaussianHeadWithFixedCovariance is used #144

Open muupan opened 3 years ago

muupan commented 3 years ago

Reported in #143

ACER assumes that all the parameters of a distribution (defined by get_params_of_distribution) require grad so that the algorithm can compute the gradient wrt the parameters. https://github.com/pfnet/pfrl/blob/44bf2e483f5a2f30be7fd062545de306247699a1/pfrl/agents/acer.py#L172-L180 https://github.com/pfnet/pfrl/blob/44bf2e483f5a2f30be7fd062545de306247699a1/pfrl/agents/acer.py#L218-L221

However, GaussianHeadWithFixedCovariance (https://github.com/pfnet/pfrl/blob/44bf2e483f5a2f30be7fd062545de306247699a1/pfrl/policies/gaussian_policy.py#L96) is used, the scale parameter of the torch.distributions.Normal distribution does not require grad, resulting in an assertion error.