official-stockfish / fishtest

The Stockfish testing framework
https://tests.stockfishchess.org/tests
281 stars 130 forks source link

Fix NPS measurement for TC scaling #2081

Open Viren6 opened 3 months ago

Viren6 commented 3 months ago

This PR modifies the NPS measurement for TC scaling to more closely resemble actual testing conditions. In particular, it addresses the point raised in https://github.com/official-stockfish/fishtest/issues/2077

Currently we run one process with a bench and one process with the search of n-1 threads. This doesn't account for these RAM bandwidth limitations discussed and therefore the measured nps is far faster than the real nps.

Instead, this PR runs a bench process for each active core and takes the average NPS.

Viren6 commented 3 months ago

Tested with https://tests.stockfishchess.org/tests/view/66729718602682471b065101?show_task=66 Works as expected

vondele commented 3 months ago

I think that's reasonable direction. Things to consider (idk the right answer)