How can I generate replays and win-rates of a population?

[ajlangley]

Hi, great work! Would it be possible to get some instructions for creating replays of a population of agents playing against one another? I am trying to create a replay using the usual method for PyMARL-based repos, but it seems to work differently here.

Also, how can I collect win-rates between pairs of teams, so I could for instance make a 5x5 matrix of win-rates?

Thanks in advance for your help! And let me know if you would like to see the commands I have already tried.

[PaLeroy]

Hey there!

I'd be happy to discuss what has been done already! Happy to see someone uses this work Also would be glad to help improve some parts of this repo, especially if you use the competSMAC, and already have some ideas.

In the past, I've manually executed the test script with the saved models I wanted to obtain replays. I think it should be possible to adapt the main_popu_to_test.py to create a list of matchmaking to observe and to make everyone plays again each other with the replay.

IMO, it should work like pymarl, but it should be done carefully since in pymarl, you have the same model to save the replay.

I don't think we can collect precise matchmaking win rates with what is implemented yet, but by tracking the opponent IDs in the buffer, the matrix should be easily obtained after your experiments.

[ajlangley]

I see, I will be digging into the code later to understand main_popu_to_test.py better. By executing it manually to obtain replays, do you mean that you run the script for each pair of agents for which you want a replay?

Also, when you say "it should be possible to adapt the main_popu_to_test.py to create a list of matchmaking to observe and to make everyone plays again each other with the replay," I had assumed that this was what the script is doing. Would it be possible to get a quick explanation of what main_popu_to_test.py does, exactly?

I will definitely let you know if I have any ideas for improvement as I get more familiar with the code, and thanks for your response!

[PaLeroy]

I see, I will be digging into the code later to understand main_popu_to_test.py better. By executing it manually to obtain replays, do you mean that you run the script for each pair of agents for which you want a replay?

Yeah exactly, you can see the run_test_duo.sh for example

So the role of main_popu_to_test is to run games with already trained models. Based on the config, the script load all the models and prepare the agent_dict and then based on the args.matchmaking, it plays some matches.

I guess it is possible there to add a new type of matchmaking to have a replay of everyone playing together.

[ajlangley]

Thanks Leroy , this helps a lot. I will close this issue for now and will keep in touch in case I have any useful suggestions. :)