search

nnaisense / evotorch

Advanced evolutionary computation library built directly on top of PyTorch, created at NNAISENSE.
https://evotorch.ai
Apache License 2.0
997 stars 62 forks source link

Action Clipping Wrapper #92

Closed roger-creus closed 3 months ago

roger-creus commented 11 months ago

Hi! For GymNE problems, I see the Action Clipping wrapper is added automatically to the policy when calling the to_policy() method. However, I don't think this method is called during the training, is this intended? Shouldn't actions be rescaled and clipped during training (as also done here).

Thank you!

engintoklu commented 11 months ago

Hello @roger-creus, thank you for your interest in EvoTorch, and for your question!

The action clipping wrapper is added onto the result of the to_policy() method to make sure that all the necessary behaviors to run the policy are existent within one torch.nn.Module object.

During the training, the action clipping is still done, only not by using the same wrapper: https://github.com/nnaisense/evotorch/blob/39421dccf7aaf3303be318183a51fcdd55b7ab5e/src/evotorch/neuroevolution/gymne.py#L346

This line should ensure that the gym environment does not receive any out-of-bounds action during the training.

Still, if you experience that this clipping is not working and you observe that your gym environment is receiving out-of-bounds actions during training, feel free to let us know and we will investigate.

Thanks & happy coding!