Open SteveDraper opened 10 years ago
So, following our conversations, I think there are two independent things there.
And thinking about this, we also need to consider how "virtual losses" will work in the new scheme (i.e. how we'll avoid all threads working in the same area of the tree).
MCTS statistics are far from perfect under two circumstances: 1) When transition occur, because the child node can have been sampled far more than the number of times expected relative to a particular parent 2) When a child completes (non-decisively) it will typically undergo a discontinuous jump in its score, which will not reflect into the parent for some time thereafter
So, instead of propagating results up a particular path in the tree/graph just mark the selection path as dirty and on querying average score or num visits, calculate (and cache, marking non-dirty) suitable approximations from first principles given the values of those properties in the immediate children. If this can be pulled off (it's far from obvious in terms of the details) then any node will reflect maximal information (given the distribution of samples amongst children) regardless of transitions, and adjusting immediately to non-continuous value changes due to child completions.