headless monte carlo for starting player advantage analysis

[ksjogo]

I have some abstract game I would like to analyse for starting player advantage and how likely it is to draw. Is there some support for running a lot of full AI runs and see aggregate results?

[nicolodavis]

This is something that I've always wanted to do (it falls under the general category of Simulations that is on my TODO list), but it isn't really on the current roadmap for v1.

We could probably have something that can just take a Game object and do this sort of analysis on the commandline via a Node CLI program of some sort.

[danmkent]

I'm playing around with things in this area. Since my PR (https://github.com/nicolodavis/boardgame.io/pull/493) was kindly accepted, the Simulate function is now exported, so it is easy to write code that uses the boardgame.io AI framework to run simulations.

At the most basic, you just need to configure a game object and then have a loop with a bot and a call to Simulate in it:

for(let i=0; i<testCount; i++){    
    const randomBot = new RandomBot({
        seed:i,
        enumerate: enumerateMovesFunction
    })

    let { state: endState} = Simulate({
        game: game,
        bots: [randomBot, randomBot],
        state: {G,ctx},
    })

    //use results from this test run
}

You just grab the winner details from the results and compile your statistics. I am also getting the game score from the final game state to produce some statistics on that.

One thing to note is that you need to generate the bot seed so that it is different for each iteration of the loop, otherwise the bots just behave the same way each time.

I'm currently doing this from my React interface for convenience of creating starting positions to test, so the boardgame.io Client provides the G and ctx objects. At some point I will want a commandline tool to test batches of starting positions – it looks like the exported InitializeGame function will enable that.

[nicolodavis]

@danmkent Nice! We should probably create a nicer public API rather than use the internal ones like InitializeGame (although that's fine in the meantime).

In the ideal world, you should be able to do:

import { Simulate } from 'boardgame.io/ai';

const statistics = Simulate(game);   // simulate with random bots
const statistics = Simulate(game, bots);   // also customize the bots
const statistics = Simulate(game, bots, initialState);  // also override the initial state to simulate from.

[danmkent]

I'd be happy to contribute to this area. This is one of the main aspects of boardgame.io that I expect to make use of, so it would be great to firm up the API.

There are lots of stats that can be collected regardless of the details of the game being simulated, so it would definitely make sense to have Simulate return them.

I created a separate issue (https://github.com/nicolodavis/boardgame.io/issues/494) for the idea of providing hooks that code that uses Simulate could use to get more game-specific data from simulation runs. I would imagine these would be passed into Simulate in another parameter.

[nearwood]

This is awesome, good work. I also would love to run some stats on some simple board games.