official-stockfish / Stockfish

A free and strong UCI chess engine
https://stockfishchess.org/
GNU General Public License v3.0
11.57k stars 2.28k forks source link

NNUE = false tests shouldn't be done even if it just affects 'classical eval' #3590

Closed NightlyKing closed 2 years ago

NightlyKing commented 3 years ago

If I remember correctly, during the "classical eval era", we never tested smaller parts of it, like PSQTs and lazy eval, on its own. Now the classical eval only serves as a lazy way to get a value for a position. Not using nnue = false would be consistent with how it's has always been. My hope is that this change will encourage people to make more significant changes to the classical, or handwritten part, of our evaluation since slowdowns for that part of the engine barely show at all. This could mean that adding a large specialized endgame heuristic won't pass our nnue = false testing but would actually pass nnue = true.

Sopel97 commented 3 years ago

I've never even stumbled on a single person that uses nnue=false

Vizvezdenec commented 3 years ago

I can only agree. There is no actual reason to test smth with NNUE=false, especially when fully disabling classical eval from usage is something like 1-2 elo.

NightlyKing commented 3 years ago

It's the same for tuning as well. If we tune classical eval without nnue active we'll probably end up fixing problems in pure classical mode that are caused by search patches that can only pass with nnue on. If we tune classical eval with nnue on we (hopefully) end up fixing some problems/'eval holes' that hybrid has which aren't necessarily the same as the problems in pure classical mode.

vondele commented 2 years ago

just for the record, we consider classical eval frozen. The effect of improving it is minimal on what is most important now, namely game quality with NNUE. As such it is not a good use of resources.