This is a major update for qlearning baselines which includes:
Refactor: Complete refactor of the code, aiming to make it lighter and more readable. Some features are now excluded (such as td_lambda, which follows the PyMARL2 implementation and doesn't seem valid, and non-parameter sharing, which can be easily re-included if needed).
Better results: The new code fixes some bugs and adds the possibility to train for multiple epochs per update, resulting in improved sample efficiency and overall performance in Smax (results of the NeurIPS paper).
PQN: PQN scripts are included with FF, RNN, and CNN versions. PQN appears to be the state-of-the-art baseline in many JAXMARL environments like Smax, Hanabi, and Overcooked.
Overcooked: Scripts with convolutional network and shaped reward are included for PPO, VDN, and PQN-VDN.
Additional features: Extra features included in all scripts, such as separate logging of vmapped seeds on wandb, saving of (vampped) parameters after training, and hyperparameter tuning.
Configuration alignment: Configuration usage is more aligned with PPO scripts, using a single config YAML file for each script. A general config.yaml is kept to avoid duplicating general hyperparameters like wandb setup in all the configuration files.
VDN_FF without RNN: A VDN_FF version without RNN is included.
This is a major update for qlearning baselines which includes:
td_lambda, which follows the PyMARL2 implementation and doesn't seem valid, and non-parameter sharing, which can be easily re-included if needed).