Add gymnasium support for TD3

[pseudo-rnd-thoughts]

Description

Types of changes

• [ ] Bug fix
• [x] New feature
• [ ] New algorithm
• [ ] Documentation

Checklist:

• [x] I've read the CONTRIBUTION guide (required).
• [x] I have ensured pre-commit run --all-files passes (required).
• [x] I have updated the tests accordingly (if applicable).
• [ ] I have updated the documentation and previewed the changes via mkdocs serve.
  • [ ] I have explained note-worthy implementation details.
  • [ ] I have explained the logged metrics.
  • [ ] I have added links to the original paper and related papers.

If you need to run benchmark experiments for a performance-impacting changes:

• [ ] I have contacted @vwxyzjn to obtain access to the openrlbenchmark W&B team.
• [ ] I have used the benchmark utility to submit the tracked experiments to the openrlbenchmark/cleanrl W&B project, optionally with --capture-video.
• [ ] I have performed RLops with python -m openrlbenchmark.rlops.
  • For new feature or bug fix:
    • [ ] I have used the RLops utility to understand the performance impact of the changes and confirmed there is no regression.
  • For new algorithm:
    • [ ] I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
  • [ ] I have added the learning curves generated by the python -m openrlbenchmark.rlops utility to the documentation.
  • [ ] I have added links to the tracked experiments in W&B, generated by python -m openrlbenchmark.rlops ....your_args... --report, to the documentation.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
cleanrl	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 9, 2023 2:57pm

[vwxyzjn]

Hi @pseudo-rnd-thoughts thanks for the PR! Could you do a filediff between td3 variants and ddpg variants to minimize the lines of code differences? You can select two files in vscode, right click them, and select "compare selected", which should produce something like below.

[pseudo-rnd-thoughts]

Hi @pseudo-rnd-thoughts thanks for the PR! Could you do a filediff between td3 variants and ddpg variants to minimize the lines of code differences? You can select two files in vscode, right click them, and select "compare selected", which should produce something like below.

Done in fed8aaf09c880e7e8df9a517a850c51cede99a29 The differences between DDPG and TD3 should be solely due to the heuristic improvements. The clipping part is relatively confusing and unclear but that could be looked at in another PR

[sdpkjc]

👋Hi there,

I've noticed that this PR hasn't had any activity for several months and it now only requires some clean-up work and conflict resolution to complete. I would like to volunteer to take over and finish off this work if there's no objection. Your efforts on this so far have been much appreciated, and I look forward to the possibility of being able to contribute to its completion.

[vwxyzjn]

@sdpkjc feel free to. Thanks so much!

[sdpkjc]

[sdpkjc]

[sdpkjc]

I've already run the RLops process and everything is performing as expected. If there are no other issues, I will proceed to merge this branch.

[vwxyzjn]

LGTM! Feel free to merge.

[pseudo-rnd-thoughts]

@sdpkjc Apologies for not finishing this and thanks for doing the rest of it