Pufferlib PPO

[daphnecor]

Description

Integration of Pufferlib for fast PPO. Additionally, it restructures algorithms into integrations for improved naming.

Readme: https://github.com/Emerge-Lab/gpudrive/tree/integrations/smoketested-puffer-ppo

Todo

• [x] (dc) Add README for running Puffer IPPO
• [x] Running implementation on cpu and cuda
• [x] (dc) Check and fix configs
• [x] (dc) Fix checkpointing and data_dir
• [x] (dc) Fix observations
• [x] (dc) Add video logging
• [x] (dc) Check / improve other stats
• [x] Train with multiple scenes
• [x] Benchmark against SB3 IPPO implementation
• [x] Understand bottleneck in Puffer implementation
• [x] Cleanup and refactor

Notes

• this works with the pufferlib gpudrive branch. To install it, use

  pip install git+https://github.com/PufferAI/PufferLib.git@gpudrive

Benchmarking

SB3

• SB3 PPO run command to train on a single scene with 3 controlled agents

  python baselines/ippo/ippo_sb3.py  --k_unique_scenes=1  --resample_scenarios=False --render_n_worlds=1

• SB3 PPO run command to train on a single dense scene with [many] controlled agents

  python baselines/ippo/ippo_sb3.py  --data_dir='data/processed/dense' --k_unique_scenes=1  --resample_scenarios=False --render_n_worlds=1

Puffer

• Pufferlib PPO run command to train on a single dense scene

  python baselines/ippo/ippo_pufferlib.py --track --data-dir 'data/processed/dense'

[jsuarez5341]

Use this instead of dev: https://github.com/PufferAI/PufferLib/tree/gpudrive It's a fork I just made off of dev right now with no changes. I break dev a lot and will probably break it even more in the process of preparing for 1.1. You can pin 1.1 after I finish that