Implement Gymnasium-compliant PPO script

[dtch1997]

Description

This PR modifies ppo_continuous_action.py to work with Gymnasium environments.

Types of changes

• [ ] Bug fix
• [x] New feature
• [ ] New algorithm
• [ ] Documentation

Checklist:

• [x] I've read the CONTRIBUTION guide (required).
• [x] I have ensured pre-commit run --all-files passes (required).
• [ ] I have updated the documentation and previewed the changes via mkdocs serve.
• [ ] I have updated the tests accordingly (if applicable).

If you are adding new algorithm variants or your change could result in performance difference, you may need to (re-)run tracked experiments. See https://github.com/vwxyzjn/cleanrl/pull/137 as an example PR.

• [x] I have contacted vwxyzjn to obtain access to the openrlbenchmark W&B team (required).
• [ ] I have tracked applicable experiments in openrlbenchmark/cleanrl with --capture-video flag toggled on (required).
• [ ] I have added additional documentation and previewed the changes via mkdocs serve.
  • [ ] I have explained note-worthy implementation details.
  • [ ] I have explained the logged metrics.
  • [ ] I have added links to the original paper and related papers (if applicable).
  • [ ] I have added links to the PR related to the algorithm variant.
  • [ ] I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
  • [ ] I have added the learning curves (in PNG format).
  • [ ] I have added links to the tracked experiments.
  • [ ] I have updated the overview sections at the docs and the repo
• [ ] I have updated the tests accordingly (if applicable).

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
cleanrl	✅ Ready (Inspect)	Visit Preview	Nov 15, 2022 at 2:12AM (UTC)

[dtch1997]

Experiments ongoing: https://wandb.ai/dtch1997/cleanRL?workspace=user-dtch1997

[vwxyzjn]

@dtch1997 thanks for this PR. Great work! Your work also made me realize we could probably support gym and gymnasium simultaneously since they have different package names. We could probably save all of the gymnasium scripts under the gymnasium folder and then gradually migrate to gymnasium. #277 #263

[vwxyzjn]

Let me fix the CI

[vwxyzjn]

@dtch1997 could you give me write access to the branch?

[dtch1997]

@vwxyzjn the option doesn't show up for me, after following Github docs: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork

I added you as a collaborator on the downstream fork instead, does that help?

[vwxyzjn]

Thanks for giving it a try but it wasn't quite working...

I think the "allow edits from maintainer" option wasn't there because the PR is based on your master branch. Would you mind creating another branch and filing another PR?

[dtch1997]

Closed as a duplicate of #319