search

suragnair / alpha-zero-general

A clean implementation based on AlphaZero for any game in any framework + tutorial + Othello/Gobang/TicTacToe/Connect4 and more
MIT License
3.74k stars 1.01k forks source link

mcts is reset after every episode? #292

Open tueboesen opened 1 year ago

tueboesen commented 1 year ago

I am not an expert in mcts at all, but I have been playing around with your code and I noticed that in the learn method the mcts is reset after every single episode. Meaning the mcts is reset after every single game when it is generating data to train on. Hence the mcts never grows beyond the information of a single game, this seems weird to me since I would expect you to want to keep the mcts and continue to evolve it and only reset it once the neural network has been trained on the data generated?

Is this really intended?

arnemileswinter commented 1 year ago

If i understand your question correctly, it's what they do in AlphaGoZero too:

The search tree is reused at subsequent time-steps: the child node corresponding to the played action becomes the new root node; the subtree below this child is retained along with all its statistics, while the remainder of the tree is discarded.

so i think it is intended. Have you tried reusing the tree? (I would guess that the memory footprint will be ginormous, though)

cyshe commented 1 day ago

args.numMCTSSims games are played when calling getActionProb for each MCTS.