[Proposal] Tutorials

[pseudo-rnd-thoughts]

Proposal

To encourage the use of Gymnasium and build up the RL community, I would propose that a large range of tutorials are created.

This is a list of tutorials that could be made

• [x] Implementation of a custom environment
• [x] Frozenlake training results with different map sizes
• [x] Gymnasium vectorisation (gym.make_vec)
• [x] Training agents for the blackjack environment, it has Tuple observation space
• [ ] DQN for atari implementation (doesn't need to be fast)
• [ ] Train an agent with stable-baselines-3
• [ ] Train an agent with tianshou
• [ ] Training a deep RL agent with pytorch fromscratch
• [ ] Training a deep RL agent with jax from scratch
• [ ] How to use the action sample masking, with example from Taxi
• [ ] Car racing, comparing agents with continuous and discrete action spaces
• [ ] Exploring the impact of bipedal walker hardcore parameter on agent performance
• [ ] Experimenting with classic control reset options random state bounds
• [ ] Add environment or example using the Graph space

[kir0ul]

Will there be a required format for the tutorials?

[kir0ul]

Isn't the first tutorial on Implementation of a custom environment already done in: https://gymnasium.farama.org/content/environment_creation/ ?

[pseudo-rnd-thoughts]

We are using the sphinx-gallery project, therefore, we expect single python files with figures stored in a folder with the same name. We are in the process of transferring our current tutorials to this style but pytorch use this style for their tutorials

[BalajiAI]

Since I'm a beginner, I'll take DQN for Atari tutorial.

[kir0ul]

I'm working on the Frozenlake one, still work in progress but I hope to have it ready soonish.

[siddarth-c]

I am working on "Training a deep RL agent with pytorch (with the new step API)".

[hahas94]

Could someone make an update on this proposal? For instance, which tasks are done, which ones are work in progress, so that one may know what to work on.

[pseudo-rnd-thoughts]

Could someone make an update on this proposal? For instance, which tasks are done, which ones are work in progress, so that one may know what to work on.

I've have updated

[kir0ul]

@pseudo-rnd-thoughts also the Frozenlake training results with different map sizes tutorial is done.

[hahas94]

Thanks!

I am thinking to work on Training a deep RL agent with pytorch fromscratch. However, doesn't the REINFORCE implementation fit that requirement?

[pseudo-rnd-thoughts]

Good point @hahas94 I think we are always happy for more tutorials. You can work on another tutorial if you want

[hahas94]

Okay, if similar tutorials is a good thing I will stick to my aforementioned choice. I am thinking of writing a tutorial on implementing the C51 agent in PyTorch.

[jonnytracker]

We need this. For practical use case. Need external environment tutorials not another memory environment

[dantp-ai]

I can add a tutorial for "Train an agent with tianshou". The exact agent and env is free to choose right, as long as env is from Gymnasium?

[Kallinteris-Andreas]

@dantp-ai we have chosen to link tutorials, who purpose is to show how to use external libraries, to

https://gymnasium.farama.org/main/tutorials/third-party-tutorials/#third-party-tutorials-about-libraries-that-use-gymnasium

[dantp-ai]

@Kallinteris-Andreas I see, so for "Train an agent with tianshou" it would simply mean listing a link to the library with a brief explanation of its focus (similar to AgileRL in the section).

Regarding, the tutorial "How to use the action sample masking, with example from Taxi", I can tackle this one. Do I understand correctly that for this tutorial we want to show the impact of the RL agent learning, when we use at each state the info["action_mask"] to pick up an action that results in a state change versus when we are not using the action_mask info?

[Kallinteris-Andreas]

1. It does mean that now
2. @pseudo-rnd-thoughts, can elaborate

[pseudo-rnd-thoughts]

@dantp-ai yes that is general idea. Both a discussion of what action masking is, how it is impacted in gymnasium with space.sample(mask) and a small training implementation