Should collect a database of test positions and the scores/moves indicated by a reputable engine with a reasonable search time (5 seconds should be plenty). Then see how our engine compares with their decisions under different time constraints.
It is unreasonable to say that it should guess the correct move on each position, or even to set a percentage that must be correct. However it would be a useful regression test for when accuracy or performance drops, as well as providing insight into the approximate ELO the engine is playing at, and what positions it is struggling with.
Should collect a database of test positions and the scores/moves indicated by a reputable engine with a reasonable search time (5 seconds should be plenty). Then see how our engine compares with their decisions under different time constraints.
It is unreasonable to say that it should guess the correct move on each position, or even to set a percentage that must be correct. However it would be a useful regression test for when accuracy or performance drops, as well as providing insight into the approximate ELO the engine is playing at, and what positions it is struggling with.