[Solver] Record statistics which will help in analysing Sudoku difficulty

[kirtivr]

Comparing statistics for a known "hard" sudoku and a known "easy" sudoku (number of iterations = 5, number of candidate cells to shuffle between = 5].

Easy Sudoku:

{id : 1, givens_to_empty_ratio : 0.883721, unique_solutions : [ 451962738736185249982374651813649527627531894594827163369218475175496382248753916 ], avg_number_of_steps : 57.6, avg_failure_to_success_ratio : 3.8, avg_candidates_for_picked_cell : 4, avg_median_of_candidates_for_cell : 4.02986, }

Hard Sudoku:

{id : 872, givens_to_empty_ratio : 0.396552, unique_solutions : [ 478653921365912784219874365736128459591746238842395176987531642623489517154267893 ], avg_number_of_steps : 12117, avg_failure_to_success_ratio : 2940.4, avg_candidates_for_picked_cell : 4, avg_median_of_candidates_for_cell : 4.20893, }

It seems the straightforward way to tell if the Sudoku is easy or hard is to compare the number of givens (pre-filled assignments) for the Sudoku.

While solving the Sudoku, the hard sudoku had an average 12 10^3 steps, out of which 3 10^2 steps resulted in an outcome [%outcome = 24%].

On the other hand the easy sudoku had an average 58 steps, out of which 4 steps resulted in an outcome [%outcome = 6.9%].

There was not much of a difference in the average number of candidates for the top 5 best pickable cells.

A lot of these results are counter-intuitive, in terms of what it actually means for a Sudoku to be easy or hard.

[kirtivr]

Another metric that may be useful to collect: average number of steps until an outcome.

[kirtivr]

Also calculate the success/fail outcomes separately, and present them as %failed, %succeeded.

It may also make sense to remove the "avg_candidates_for_picked_cell : 4," metric since an existing optimization messes up this number. And we are getting much more representative data from "avg_median_of_candidates_for_cell".