Official implementation of the AAMAS 2024 paper Measruing Policy Distance for MARL.
To run this repository you need:
i) Install the code's requirements. A virtual environment based on Conda is recommended (We will update the docker approach soon). Install with
conda create --name madps --file requirements.txt
ii) Install the supported MARL-environments, for example:
madps/large_spread_example.py
with pettingzoo/mpe/scenarios/large_spread.py
to access the updated task.You can then simply run ac_NF.py using:
python ac_NF.py with env_name='pettingzoo:pz-mpe-large-spread-v1' time_limit=50
This command runs our multi-agent actor-critic training framework, with pettingzoo:pz-mpe-large-spread-v1
as the training scenario and the maximum episode step length as 50. The scenario versions, from large-spread-v1
to v6
, correspond to the 15a_3c
, \ 30a_3c
, \ 30a_5c
, \ 30a_5c_super
, \ 15a_3c_shuffle
, and 30a_3c_shuffle
scenarios in our paper.
The MADPS code is structured as follows:
ac_NF.py includes:
More details can be found in the comments of main function in ac_NF.py.
MADPS_NF.py includes:
compute_fusions
(including MAPD and MADPS):
calculate_N_Gaussians_BD
: This function is used for parallel calculation of the Bhattacharyya distance between multiple Gaussian distributions using PyTorch.calculate_N_Gaussians_Hellinger_through_BD
: This function is used for parallel calculation of the Hellinger distance between multiple Gaussian distributions using PyTorch. It requires results from the calculate_N_Gaussians_BD
function.calculate_N_Gaussians_WD
: This function is used for the parallel calculation of the Wasserstein distance between multiple Gaussian distributions using PyTorch.model_NF.py includes:
MADPSNet
: Multi-agent neural network models that support dynamic adjustment and hierarchical adjustment of parameter sharing.MultiAgentFCNetwork
: Multi-agent neural network models that support adjustment of parameter sharing (Replication of SePS algorithm).Policy
: Multi-agent policy models.ConditionalVAE
: Conditional VAE model.Note: More details are coming soon.
Before the AAMAS 2024 conference, please cite the arxiv version.
@inproceedings{tianyihu-MAPD,
title={Measruing Policy Distance for Multi-agent Reinforcement Learning},
author={Tianyi Hu et.al},
booktitle={International Conference on Autonomous Agents and Multiagent Systems (AAMAS)},
year={2024}
}