search

chaovven / SMIX

Code for "SMIX(λ): Enhancing Centralized Value Functions for Cooperative Multi-Agent Reinforcement Learning" AAAI 2020
Apache License 2.0
26 stars 5 forks source link

How can I calculate the winrate? #5

Closed GoingMyWay closed 4 years ago

GoingMyWay commented 4 years ago

Hi, in your paper, I fund there is a winrate metric to evaluate the performance of algorithms. How can I manually calculate the value in StarCraft environment?

chaovven commented 4 years ago

Hi, in our paper, we we run the algorithms for 24 episodes with each agent performing greedy action selection during testing. The percentage of episodes where the agents defeat all enemy units within the time limit is called the win rate. This metric is automatically saved in your tensorboard event file. Refer to https://arxiv.org/abs/1902.04043 for more details on this metric.

GoingMyWay commented 4 years ago

Hi, in our paper, we we run the algorithms for 24 episodes with each agent performing greedy action selection during testing. The percentage of episodes where the agents defeat all enemy units within the time limit is called the win rate. This metric is automatically saved in your tensorboard event file. Refer to https://arxiv.org/abs/1902.04043 for more details on this metric.

Thanks, when training, you use dense rewards and when evaluation, you use winrate (win or loose) right?

chaovven commented 4 years ago

Yes. Like the standard setting in RL, rewards are used to train agents. Win rate is just a standard metric to report your final performance in the decentralized StarCraft benchmark.

GoingMyWay commented 4 years ago

Yes. Like the standard setting in RL, rewards are used to train agents. Win rate is just a standard metric to report your final performance in the decentralized StarCraft benchmark.

Thanks.

GoingMyWay commented 4 years ago

@chaovven Hi, for training the SMIX(\lambda) and QMIX, how many CPUs and how much GPU memory do we need? Also, to get the best results, how many hours training do we need? Recently, I am trying to run your code, but I have limited CPUs and GPUs.

chaovven commented 4 years ago

@chaovven Hi, for training the SMIX(\lambda) and QMIX, how many CPUs and how much GPU memory do we need? Also, to get the best results, how many hours training do we need? Recently, I am trying to run your code, but I have limited CPUs and GPUs.

Hi, running the simplest map 3m for 2 million timesteps would spend nearly 3.5 hours on my PC, with nearly 1g memory usage in my GPU and 5g of system memory (batch_size_run=4). more complex scenarios with increasing agents (e.g., 8m, 3s5z) may cost 8~10 hours or more. In your case, you'd better set batch_size_run to 1 to save your memory. Also, you can disable GPU by setting use_cuda to False.

ps: my CPU is Intel(R) Xeon(R) W-2133 CPU @ 3.60GHz and GPU is GeForce GTX TITAN X.

GoingMyWay commented 4 years ago

@chaovven Hi, for training the SMIX(\lambda) and QMIX, how many CPUs and how much GPU memory do we need? Also, to get the best results, how many hours training do we need? Recently, I am trying to run your code, but I have limited CPUs and GPUs.

Hi, running the simplest map 3m for 2 million timesteps would spend nearly 3.5 hours on my PC, with nearly 1g memory usage in my GPU and 5g of system memory (batch_size_run=4). more complex scenarios with increasing agents (e.g., 8m, 3s5z) may cost 8~10 hours or more. In your case, you'd better set batch_size_run to 1 to save your memory. Also, you can disable GPU by setting use_cuda to False.

ps: my CPU is Intel(R) Xeon(R) W-2133 CPU @ 3.60GHz and GPU is GeForce GTX TITAN X.

Thanks, 8-10 hours is not that long compared with Atari games. I think 5 CPU cores and one P100 are enough for SC training, do you think so? Another question is did you try the mini-games from DeepMind?

chaovven commented 4 years ago

@chaovven Hi, for training the SMIX(\lambda) and QMIX, how many CPUs and how much GPU memory do we need? Also, to get the best results, how many hours training do we need? Recently, I am trying to run your code, but I have limited CPUs and GPUs.

Hi, running the simplest map 3m for 2 million timesteps would spend nearly 3.5 hours on my PC, with nearly 1g memory usage in my GPU and 5g of system memory (batch_size_run=4). more complex scenarios with increasing agents (e.g., 8m, 3s5z) may cost 8~10 hours or more. In your case, you'd better set batch_size_run to 1 to save your memory. Also, you can disable GPU by setting use_cuda to False. ps: my CPU is Intel(R) Xeon(R) W-2133 CPU @ 3.60GHz and GPU is GeForce GTX TITAN X.

Thanks, 8-10 hours is not that long compared with Atari games. I think 5 CPU cores and one P100 are enough for SC training, do you think so? Another question is did you try the mini-games from DeepMind?

Yes, though more training time may be needed. I didn't try the mini-games.

GoingMyWay commented 4 years ago

@chaovven Hi, for training the SMIX(\lambda) and QMIX, how many CPUs and how much GPU memory do we need? Also, to get the best results, how many hours training do we need? Recently, I am trying to run your code, but I have limited CPUs and GPUs.

Hi, running the simplest map 3m for 2 million timesteps would spend nearly 3.5 hours on my PC, with nearly 1g memory usage in my GPU and 5g of system memory (batch_size_run=4). more complex scenarios with increasing agents (e.g., 8m, 3s5z) may cost 8~10 hours or more. In your case, you'd better set batch_size_run to 1 to save your memory. Also, you can disable GPU by setting use_cuda to False. ps: my CPU is Intel(R) Xeon(R) W-2133 CPU @ 3.60GHz and GPU is GeForce GTX TITAN X.

Thanks, 8-10 hours is not that long compared with Atari games. I think 5 CPU cores and one P100 are enough for SC training, do you think so? Another question is did you try the mini-games from DeepMind?

Yes, though more training time may be needed. I didn't try the mini-games.

Good, thanks.