[Feature Request] Implement TQC for the example algorithms

pytorch / rl

A modular, primitive-first, python-first PyTorch library for Reinforcement Learning.

https://pytorch.org/rl

MIT License

2.2k stars 290 forks source link

[Feature Request] Implement TQC for the example algorithms #1623

Open maxweissenbacher opened 11 months ago

maxweissenbacher commented 11 months ago

Motivation

I suggest adding an implementation of TQC to the examples. I suggest adding this as a request in the 'call for distributions' stack. I would be happy to take on the implementation.

Solution

Add a performant, clear and minimal implementation of TQC to the examples. I would base the implementation on the already existing implementation of SAC in the examples. This seems reasonable due to the overall similar structure of the algorithms. In the Stable-Baselines3 library implementation of TQC, the implementation is likewise based on their corresponding implementation of SAC.

Checklist

[x] I have checked that there is no similar issue in the repo (required)
[x] Add a request to the call for contributions.
[x] Adapt the implementation of SAC from the examples to TQC
[ ] Benchmark?

### Tasks

vmoens commented 11 months ago

Hi @maxweissenbacher Adding TQC would be fantastic and greatly appreciated! Any help we can provide, just let us know. I'm adding this to the call for contributions straight away!

maxweissenbacher commented 11 months ago

Hi @vmoens, that's great, thank you! I'll get started right away and come back here in case I need assistance (more likely than not).

maxweissenbacher commented 11 months ago

I cannot seem to get the logger to work for me. By running the (unchanged) training script for the SAC example, trying to execute theget_logger function gives me the following error:

wandb: WARNING Path sac_logging/wandb/ wasn't writable, using system temp directory.

Any suggestions?

maxweissenbacher commented 11 months ago

Another (very silly) question... this is my first open source contrib - how do I contribute the code to the github? Do I need to fork the repository first, and submit a pull request?

vmoens commented 11 months ago

For the logger it seems you lack write access in the folder where you're executing the code.

For the OSS contribution:

make a fork of torchrl on your username
write code in your separate branch locally on your machine
when you're at a stage where you feel the code is mature and/or you want to show it to us to get feeeback, commit the changes on that branch and push it to your fork. Then create a PR from that branch towards main on the PyTorch org.

You will need to format the code etc for which I invite you to read the CONTRIBUTE.md file in torchrl.

Thanks again for your help!

maxweissenbacher commented 11 months ago

Thank you for all this info. I have submitted a pull request now, hopefully I did it right!

I couldn't get the logging to work, so I implemented a custom logger to use on my local machine instead - however the logging is just as in the other example files in the pull request, so it should hopefully work as expected for you.