Does Catalyst support multi-agent environments?

[rfali]

Hi

1. Can Catalyst implementations be used in multi-agent environments?
2. Does Catalyst support vectorized environments?

I am more specifically talking about DQN/Rainbow (and not interested in PPO etc) as my action domain is discrete.

FAQ

Please review the FAQ before submitting an issue:

• [x] I have read the documentation and FAQ
• [ ] I have reviewed the minimal examples section
• [ ] I have checked the changelog for main framework updates
• [ ] I have read the contribution guide
• [ ] I have joined Catalyst slack (#__questions channel) for issue discussion

[github-actions[bot]]

Hi! Thank you for your contribution! Please re-check all issue template checklists - unfilled issues would be closed automatically. And do not forget to join our slack for collaboration.

[Scitator]

Hi!

Catalyst.RL 2021 is currently in the development stage, you could check the demo notebook here. tl;dr

• you could implement any architecture you would like
• and any policy logic you want
• connect all these through the custom "GameCallback"
• and watch you training :)

By saying this, I mean, that it's on your side to write the environment logic and how to sample trajectories from it (vectorized environments, etc). Catalyst is a high-level experiment runner on top of PyTorch and correctly focused on Deep Learning (CV/NLP/RecSys) and Representation Learning mostly. Reinforcement Learning is under development stage - you could run Reinforcement Learning with Catalyst, but it still requires a lot of custom code from your side.

You could also check Catalyst.RL 2020 and an introduction for your own environment usage. Nevertheless, the 2020 version is frozen nowadays

[rfali]

Thanks @Scitator. I actually was looking towards Catalyst-RL because of this multi-headed DQN as per this PR that I wanted to use in a multi-agent environment.

I looked at this catalyst atari example. I see that the multi-headed dqn is part of this dqn.py, and the config file is this. In order to use multi-headed DQN, I only need to uncomment line46 right?

1. Can I use this DQN only on the Gym Atari, and not any other custom environments?
2. If I can use it on other custom environments (similar to Atari, I am actually looking at a multi-agent Atari from PettingZoo), can you show me how by an example code or reference to some other implemented work using catalyst-rl?

Thanks!

[Scitator]

Sorry, Catalyst.RL is deprecated for now – you could use it, run it and do all other interesting research stuff on your own. Nevertheless, active support is not possible for free time. If you still would like to collaborate on Catalyst.RL, you could write us at team@catalyst-team.com. Moreover, speaking of the open-source Catalyst.RL 2, you could check out new examples here.

[rfali]

I just wanted to use the catalyst v1 out of the box for one of its implementation and was wondering how can I use catalyst agents in other environments (gym-like but not Gym itself). the catalyst v2 does not have the specific implementation I am looking for. Anyways, seems like I will have to figure this out myself. Thanks!