Right now, I am creating rating plots with the generation number as the x variable. For the sorts of tests I am doing, this is fine.
However, we may in the future wish to test things that make this choice of x variable not appropriate. For instance, we may want to test using more mcts iterations per move in self-play. This will likely lead to more improvement per generation, while making each generation take longer to produce. A good x-variable should capture this trade-off appropriately.
Clock time is another possible choice, but it has its downsides as well.
Still, another choice is some sort of measure of GPU usage. The KataGo paper uses "self-play cost in billions of equivalent 20 block x 256 channel queries", as can be seen by the below plot:
I have questions about how "equivalent" is defined if using different architecture sizes, or how query-batching factors in, but the high level idea behind this choice seems right. Supporting it would demand adding some query-count-tracking in the c++, and then dumping this count at the end of the run to be picked up by the python plotting logic.
shindavid
commented
1 year ago
Right now, I am creating rating plots with the generation number as the x variable. For the sorts of tests I am doing, this is fine.
However, we may in the future wish to test things that make this choice of x variable not appropriate. For instance, we may want to test using more mcts iterations per move in self-play. This will likely lead to more improvement per generation, while making each generation take longer to produce. A good x-variable should capture this trade-off appropriately.
Clock time is another possible choice, but it has its downsides as well.
Still, another choice is some sort of measure of GPU usage. The KataGo paper uses "self-play cost in billions of equivalent 20 block x 256 channel queries", as can be seen by the below plot:
I have questions about how "equivalent" is defined if using different architecture sizes, or how query-batching factors in, but the high level idea behind this choice seems right. Supporting it would demand adding some query-count-tracking in the c++, and then dumping this count at the end of the run to be picked up by the python plotting logic.