haroldsultan
commented
6 years ago James,
You are correct, the algorithm is built to optimize reward/visit. This is the most common approach in the literature and is called MAXCHILD. There are other options like ROBUSTCHILD which optimize by selecting the most visited child, although that is not what the code is doing.
In your example, Node 1 and 3 are close. Node 3 has a reward/visit ratio of 0.89125163772 while Node 1 has a reward/visit ratio of 0.89089080495. So you correct, node 3 should have been selected.
If you look at the code, the end of UCTSEARCH is a call to BESTCHILD(,0). Setting the scalar in BESTCHILD to 0 means we do not care about exploring at all and will pick the best child by selecting the child with the highest reward/visit. If there is a tie, we randomly pick between the bests. In your case I don't see what happened. Would you mind including some debug prints in the BESTCHILD function and see what is happening. Let me know if you want me to check in some code with the debug prints.
Syaoran87
Hi,
Thank you for your awesome code (mcts.py). However, I think the algorithm is not working according to the policy network. For example, in the link you gave, under section 2.1, it is stated that it aims to find the policy that yields the highest reward. When I ran the algorithm, the nodes/child selected were not the highest rewards. The root child selected also did not have the highest reward. Does this mean that the policy network is not optimized, and therefore, did not choose the best action/search?
E.g. as follows: level 0 Num Children: 4 (0, Node; children: 4; visits: 354; reward: 295.106667) (1, Node; children: 4; visits: 928; reward: 826.746667) <--- This was selected by the algorithim (2, Node; children: 4; visits: 582; reward: 504.422222) (3, Node; children: 4; visits: 933; reward: 831.537778) <--- Shouldn't this be selected? Best Child: Value: 20; Moves: [20]
The input parameters I ran was 10000 loops and 8 levels.
Thank you once again for your help. Hope to hear from you soon.
James