search

skotz / volcanoes

Prototype for a new board game
4 stars 1 forks source link

add some measure of AI performance (maybe time-to-think or NPS) to tournament results #43

Closed simondorfman closed 3 years ago

simondorfman commented 4 years ago

I'm running tournaments on two different computers and I want to be able to see if i can treat the results the same when I'm doing analysis. So I want to make sure the AI performance (or should I call it computational-strength?) is about the same. Any idea for some measure that could be added to the results to capture this?

Maybe the number of seconds for the AI to think, from the tournament settings.

Or maybe NPS?

Or maybe time-to-think multiplied by NPS?

Or maybe some hardware specs like CPU speed?

Note to self: quote from Scott "NPS = Nodes Per Second and is how many different positions it's calculating every second to get the best move."

skotz commented 4 years ago

This'll be dependent on the type of AI, but for MCTS the best thing you could measure to compare one run to another would be simulation count, which is the total number of full games the AI played against itself while searching for the best move. Total nodes would be fine too, although that includes every node in every game, so it'll be much higher. At large sample sizes I doubt there would be much difference between the two for this purpose.

Any measurement over seconds (like nodes per second) will be tied to how much CPU was available while executing, which will be affected by a lot of things.

simondorfman commented 4 years ago

Coolio. MCTS AIs are usually the strongest AIs so they're the ones I really care about most. So yeah, simulation count or node count, whichever you think is best.

simondorfman commented 3 years ago

I've been running all my simulations on one computer so I don't really care about this feature idea anymore. I'll close it. Feel free to reopen if you think it would be useful still.

simondorfman commented 3 years ago

I’m interested in this feature again. I’d like to add secondsPerMove to the output file. I’ve been tracking that separately for every simulation I run. It would be nice to have it included in the file. And if it’s not too hard, I’ll add simulationCount as a column too.

simondorfman commented 3 years ago

added secondsPerMove here: https://github.com/skotz/volcanoes/pull/50/commits/3eed1030f47a115511a37c1677dbefc68426fd4c gave up on adding simulationCount column for now.