Multi-agent with pettingzoo API

[labanca]

I'm trying multi-agent PyFlyt using pettingzoo and SuperSuit.

Where can I find material to learn how to do it and where to start? I tried following the Multi-Agent Deep Reinforcement Learning in 13 Lines of Code Using PettingZoo but I get stuck on many errors such as:

assert aec_env.metadata.get("is_parallelizable", False), (
AssertionError: Converting from an AEC environment to a Parallel environment with the to_parallel wrapper is not generally safe (the AEC environment should only update once at the end of each cycle). If you have confirmed that your AEC environment can be converted in this way, then please set the `is_parallelizable` key in your metadata to True

And after adding the metada is_parallelizable to True, the following:

for pipe, proc in zip(self.pipes, self.procs):
AttributeError: 'ProcConcatVec' object has no attribute 'pipes'

Any help would be greatly appreciated.

Thanks in advance!

[jjshoots]

PyFlyt doesn't actually have any native multi-agent environments, as such, you can't convert its environments to multi-agent directly. You'll probably have to create your own environment for the task you want.

For reference, there are fundamentally 3 types of environments within the Farama Foundation (and also the most supported environments):

1. Gymnasium environments: these are single agent environments
2. AEC environments: these environments come from pettingzoo, and are multiagent environments that are used for turn-based games (think of games like chess)
3. Parallel environments: these environments also come from pettingzoo, and are multiagent environments that are used for games where all agents step at once (think of more modern games like first person shooters)

The tutorial you mention is for converting AEC environments to Parallel environments (although it is pretty old and likely to be outdated). Meanwhile, all the environments within PyFlyt are only single agent Gymnasium environments - we don't have any environment with multiagent tasks yet, though this would be an interesting addition. For this reason, the code you are trying doesn't work.

If you could describe what you are trying to achieve (list out what you want the drones to do in a multiagent setting), then I can probably guide you into creating your own multiagent environment using the parallel API from PZ with PyFlyt as the UAV simulation engine.

[labanca]

Hi @jjshoots.

I am currently pursuing my master's thesis, which focuses on the application of Reinforcement Learning (RL) in the context of drone simulation.

The primary scenario of interest involves the implementation of a Cooperative Threat Engagement using Drone Swarms. This scenario encompasses two distinct groups of drones: the defending team, comprised of Loyal Wingman (LW) UAVs, and the attacking team, composed of loitering munition UAVs (kamikazes).

My objective is to employ RL techniques to train kamikaze agents to perform effective attacks, with a secondary goal of fostering cooperative behavior among these agents. Specifically, I aim to train kamikaze agents to coordinate their actions in consideration of the actions undertaken by other drones in the swarm.

There is a colleague of mine working with the LW defensive team and the ultimate goal of the project will be to use the policy obtained by the Kamikazes to create a more challenging scenario in which the defending agents will be trained.

To achieve this, I will construct a multi-agent environment where each agent contributes to the collective experience. In this environment, every agent possesses its own set of actions, observations, and reward mechanisms, enabling them to learn and adapt their behavior through the RL framework.

About the 3 fundamental types of environments described by you, I believe that my work would fit into a Parallel Environment.

I'm already using PyFlyt as a simulator and so far it's working perfectly, everything is very easy to use and well built (I previously used gym-pybullet-drones and had a lot of problems).

Thank you for your response and I hope it was possible to get an idea of the objectives of the work.

I really hope I can contribute to the PyFlyt project!

[jjshoots]

Hi, that sounds awesome. It sounds similar to another project I'm working on - Aerial dogfighting in PyFlyt, although the environment API that I'm using now is a custom one that suits my learning workflow. It is possible to convert it into a Parallel API using PZ's Parallel Environment Creation Tutorial, but at the moment my bandwidth is a bit full. This will ultimately end up being something that I aim to have included in PyFlyt though, so if you think you can help get that project going, I'd be more than happy to review the PR.

[labanca]

Hi @jjshoots,

Right. I had already seen this other project of yours, pyflyt_dogfight. But as in my case the drones are Kamikazes (they do not have weapons) and are of the quadrator type, so I continued to work with PyFlyt only.

But I have a question: wouldn't I also need to adapt PyFlyt or could I build the parallel environment just by changing pyflyt_dogflight? I say this because dogfight uses PyFlyt as a base.

I ask this because from what I understand in your last answer you told me to take dogfight as a basis. Or did I misunderstand and you actually meant for me to work on creating the parallel environment in PyFlyt?

[jjshoots]

Ah, I didn't see the point where you are using the quadrotor model. (Note that PyFlyt actually supports all forms of UAVs, the quadrotor just seems to be the most popular for now).

In your case, if you already have a PyFlyt Aviary where you have your "game" defined (the quadrotors have been initialized and the rules of the "game" have been built), you can probably just wrap that in the Parallel API, basically making a ParallelEnv from your base Aviary (this would be the equivalent of building a parallel environment in PyFlyt). I would think that would be much easier than modifying the dogfight env.

Do let me know if you need some guidance with this, I am more than happy to discuss this over a Discord call if you wish. You can join Farama Foundation's Discord server, then look for the user @jjshoots, that's me. You can send me a friend invite and we can organize a call there.

[jjshoots]

PZ Support has been added, closing this.