search

kiudee / chess-tuning-tools

A collection of scripts aimed at efficiently tuning chess engine parameters.
https://chess-tuning-tools.readthedocs.io/en/latest/
Other
52 stars 13 forks source link

Add more validation of cutechess output #220

Open thomasahle opened 1 year ago

thomasahle commented 1 year ago

Description

I had some really strange errors (see trace below) which turned out to be because my engine would sometimes (in branches it didn't expect to happen) write some stuff to stdout. This would mess with the regular expression matching the cutechess-cli results.

Instead of the cryptical "all the input arrays must have same number of dimensions" error, it would be nice with something like "Unexpected output: {...}"

What I Did

2023-01-06 08:50:36,242 INFO     Start experiment
2023-01-06 08:53:08,948 INFO     Experiment finished (152.704942s elapsed).
Traceback (most recent call last):
  File "/fsx/users/ahle/conda/bin/tune", line 8, in <module>
    sys.exit(cli())
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/tune/cli.py", line 505, in local
    score, error_variance, draw_rate = parse_experiment_result(out_exp, **settings)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/tune/local.py", line 1047, in parse_experi
ment_result
    diffs = np.diff(array, axis=0, prepend=np.array([[0, 0, 0]]))
  File "<__array_function__ internals>", line 5, in diff
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/numpy/lib/function_base.py", line 1281, in
 diff
    a = np.concatenate(combined, axis)
  File "<__array_function__ internals>", line 5, in concatenate
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2
 dimension(s) and the array at index 1 has 1 dimension(s)
srun: error: dev-dy-p3dn24xlarge-1: task 0: Exited with exit code 1
kiudee commented 1 year ago

Do you have an example output of what your engine produced, which messed with the parsing? It should definitely be possible to make the parsing even stricter and raise errors if an unexpected output occurs.