`Temperature` variable in Monte Carlo Tree Search

[AndreasKaratzas]

What is the physical meaning of the temperature variable? I know that it is the exploration parameter. My question is about it's initialization. In other words, what does 200 mean in the default .json configuration file? I understand that it means that it means that the agent has still mush to explore. It's just what is the quantization rule here? I mean is there a theoretical max value? Any input on the matter besides my question would be helpful to better understand the physical purpose of the temperature hyperparameter. Thanks :)

[eleurent]

Indeed, it is a parameter (often denoted c) which controls the tradeoff between exploration (sampling an action that hasnt been played a lot before) and exploitation (sampling the action with the highest current empirical value estimate). There is probably a theoretical value that guarantees asymptotic convergence (just like there is one for the UCB algorithm), but these are typically quite conservative, and in practice they are often tuned manually to get better performance. But let me try to give you an intuition: we sample the action that has the highest value_estimate + temperature * prior_probability / visits

• The first term is exploitation (trust your value estimate), and the second term is exploration (sample low visit counts).
• If temperature = 0, this is pure exploitation, as we greedily maximise the value estimate
• If temperature = infinity, it is pure exploration, as the contribution of the value becomes negligible, and we are basically sampling from a uniform distribution (assuming that the prior probabilities are uniform here, for simplicity).
• The order of magnitude that you want if for the exploration term (temperature * prior / (visit=1)) to about the scale of Vmax, the maximum value (sum of discounted rewards) achievable in your problem. Indeed, assume that the optimal action got unlucky has a and very low value estimate (say, 0), while another (suboptimal action) has a higher value estimate, say slightly lower that Vmax. Then, the exploration bonus of the optimal action will be sufficiently large to counteract the unlucky value estimate and get a total value >= Vmax and ensure the optimal action is sampled at least once.
• TLDR: having a temperature high enough ensures that you explore sufficiently to not discard too early the optimal action if it gets unlucky. But if the temperature is too high, the mcts will spend too much time exploring (and favour breadth instead of depth), so your sample efficiency will suffer. A good rule of thumb should be temperature ~= value_max / prior_probability(optimal action) ~ value_max * number_of_actions, for a uniform prior policy.

Does that make sense?

[AndreasKaratzas]

Yes, perfectly :) Thank you for the thorough explanation!