DDPG documnetation tweaks; added Q loss equations and light explanation

[dosssman]

Description

• Fixed a few typos and proposed some reformulations of a few sentences.
• Added a little bit more details regarding DDPG's Q loss.

Other comments

• Regarding the hard time reproducing ddpg on Mujoco-v1, I was wondering how feasible it would be to run fujimoto's DDPG.py etc.. on free-mujoco

Other than that, great job on the pretty complete documentation for DDPG @vwxyzjn @yooceii , and sorry for being late to the party :bow:

Types of changes

• [ ] Bug fix
• [ ] New feature
• [ ] New algorithm
• [x] Documentation

Checklist:

• [x] I've read the CONTRIBUTION guide (required).
• [x] I have ensured pre-commit run --all-files passes (required).
• [x] I have updated the documentation and previewed the changes via mkdocs serve.
• [x] I have updated the tests accordingly (if applicable).

If you are adding new algorithms or your change could result in performance difference, you may need to (re-)run tracked experiments. See https://github.com/vwxyzjn/cleanrl/pull/137 as an example PR.

• [ ] I have contacted @vwxyzjn to obtain access to the openrlbenchmark W&B team (required).
• [ ] I have tracked applicable experiments in openrlbenchmark/cleanrl with --capture-video flag toggled on (required).
• [x] I have added additional documentation and previewed the changes via mkdocs serve.
  • [x] I have explained note-worthy implementation details.
  • [x] I have explained the logged metrics.
  • [ ] I have added links to the original paper and related papers (if applicable).
  • [ ] I have added links to the PR related to the algorithm.
  • [ ] I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
  • [ ] I have added the learning curves (in PNG format with width=500 and height=300).
  • [ ] I have added links to the tracked experiments.
• [ ] I have updated the tests accordingly (if applicable).

[vercel[bot]]

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/vwxyzjn/cleanrl/CSE1uakxpjPwtLa1Dm9cmjwxxE4g
✅ Preview: https://cleanrl-git-fork-dosssman-ddpg-docs-tweaks-vwxyzjn.vercel.app

[gitpod-io[bot]]

[vwxyzjn]

This PR is a follow-up on #137. Thanks @dosssman for this fix! I will take a look at it tomorrow :)

Regarding the hard time reproducing ddpg on Mujoco-v1, I was wondering how feasible it would be to run fujimoto's DDPG.py etc.. on free-mujoco

There it is: https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/MuJoCo-sfujim-TD3--VmlldzoxNzIyODIz

[dosssman]

Thanks. The report seems privated though:

[vwxyzjn]

Could you try it again?

[dosssman]

All good now