search

LeelaChessZero / lc0

The rewritten engine, originally for tensorflow. Now all other backends have been ported here.
GNU General Public License v3.0
2.41k stars 526 forks source link

Improve default WDLDrawRateTarget handling #2029

Closed Naphthalin closed 4 months ago

Naphthalin commented 4 months ago

This fixes the following unintended behavior:

If we want to turn off the WDL sharpening when using WDLCalibrationElo by setting it to 0, the code falls back to directly using WDLDrawRateTarget. This however means that when using the correct WDLDrawRateReference in the config without also setting WDLDrawRateTarget to the same value, turning WDLCalibrationElo off doesn't actually retain the raw WDL values, since it internally works with the default 0.5 value.

By changing it to the default behavior "value 0 means no effect", it doesn't need to be set in the config anymore, and turning off WDLCalibrationElo also turns off any WDL sharpening.

borg323 commented 4 months ago

Further testing shows that very small WDLDrawRateTarget values (e.g. 0.00000001) cause issues.

Naphthalin commented 4 months ago

Further testing shows that very small WDLDrawRateTarget values (e.g. 0.00000001) cause issues.

I assume you're talking about numerical issues? Probably coming from 1 - eps being rounded to 1 numerically before taking the log.

Because from a chess point of view, already values like 0.02 (which were allowed in the old range) cause completely erratic evals, play etc. We could either sanitize for numerical issues only, or give a warning/exception/... when a too low value is used. One such value could be whatever the Elo model automatically assigns to a game between 0 (or 1000) rated players.

borg323 commented 4 months ago

After we decide on a safe minimum value, what is the preferred solution? Make smaller values zero or increase them to this safe minimum? What about WDLDrawRateReference, should this also have the same minimum? And if we decide on a minimum, what about the corresponding maximum?

Naphthalin commented 4 months ago

After we decide on a safe minimum value, what is the preferred solution? Make smaller values zero or increase them to this safe minimum?

Since any "safe minimum" is arbitrary anyway, I would treat them as zero instead and thus turn off the WDL sharpening. I'd be happy with using the current minimum (0.001) as that value so we have consistent behavior for example, even if it allows one order of magnitude more than reasonable.

What about WDLDrawRateReference, should this also have the same minimum?

It has per design resp. definition, since the reference draw rate comes from the raw WDL of the net, and we don't have any nets with values outside the current value range, nor will we ever have unless on purpose.

And if we decide on a minimum, what about the corresponding maximum?

Mathematically, it's max = 1 - min, and the current max value of 0.999 already is 1 order of magnitude above what I personally consider to be reasonable.