search

kiudee / chess-tuning-tools

A collection of scripts aimed at efficiently tuning chess engine parameters.
https://chess-tuning-tools.readthedocs.io/en/latest/
Other
51 stars 14 forks source link

Categorical parameter does not work #90

Open Claes1981 opened 4 years ago

Claes1981 commented 4 years ago

Description

Experiment with categorical UCI parameter. The tuner tries to set it to a real number instead of one of specified texts.

What I Did

tune local --tuning-config Test6.json --data-path Test6.npz --logfile Test6.log --plot-path Test6 --verbose 
2020-09-02 15:28:50,540 INFO     Starting iteration 0
2020-09-02 15:28:50,541 INFO     Testing {'TimeManager': '0.38854381999831666'}
2020-09-02 15:28:50,541 DEBUG    engines.json is prepared:
[{'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine1', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}, {'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine2', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}]
2020-09-02 15:28:50,542 INFO     Start experiment
2020-09-02 15:29:29,865 INFO     Experiment finished (39.32348s elapsed).
2020-09-02 15:29:29,865 DEBUG    Raw result:
Indexing opening suite...
Started game 1 of 2 (engine1 vs engine2)
Terminating process of engine engine1(0)
Finished game 1 (engine1 vs engine2): 0-1 {White's connection stalls}
Score of engine1 vs engine2: 0 - 1 - 0  [0.000] 1
Started game 2 of 2 (engine2 vs engine1)
Terminating process of engine engine1(2)
Finished game 2 (engine2 vs engine1): 1-0 {Black's connection stalls}
Score of engine1 vs engine2: 0 - 2 - 0  [0.000] 2
Elo difference: -inf +/- nan, LOS: 7.9 %, DrawRatio: 0.0 %
Finished match

2020-09-02 15:29:30,199 INFO     Got score: 1.021090020413224 +- 2.087254826323683
2020-09-02 15:29:30,200 INFO     Updating model
2020-09-02 15:29:30,200 INFO     GP sampling finished (0.000135s)
2020-09-02 15:29:30,201 DEBUG    GP kernel: None
2020-09-02 15:29:30,241 INFO     Starting iteration 1
2020-09-02 15:29:30,241 INFO     Testing {'TimeManager': '0.7705098312484218'}
2020-09-02 15:29:30,242 DEBUG    engines.json is prepared:
[{'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine1', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}, {'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine2', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}]
2020-09-02 15:29:30,242 INFO     Start experiment
^C
Aborted!

Test6.json:

{
    "engines": [
        {
            "command": "nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden",
            "fixed_parameters": {
                                "WeightsFile": "/partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz",
                                "Backend": "cudnn-fp16",
                                "NNCacheSize": 800000,
                                "Ponder": false,
                                "MoveOverheadMs": 3000
            }
        },
        {
            "command": "nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden",
            "fixed_parameters": {
                                "WeightsFile": "/partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz",
                                "Backend": "cudnn-fp16",
                                "NNCacheSize": 800000,
                                "Ponder": false,
                                "MoveOverheadMs": 3000
            }
        }
    ],
    "parameter_ranges": {
       "TimeManager": "('legacy', 'smooth')"
    },
    "engine1_st": "4",
    "engine2_st": "4",
    "rounds": 1,
    "opening_file": "empty.pgn",
    "adjudicate_draws": false,
    "adjudicate_resign": false
}
AlexisOlson commented 4 years ago

Maybe it needs to be specified as categorical like here?

https://github.com/kiudee/chess-tuning-tools/blob/c9931ed2b4f5a2d26e36cfa5157be9c902fbc741/tune/io.py#L112

kiudee commented 4 years ago

Could also be that there is a bug regarding categoricals. I will check it tomorrow. edit: But yes, try AlexisOlson’s suggestion first:

"TimeManager": "Categorical([legacy, smooth])"
or
"TimeManager": "Categorical(['legacy', 'smooth'])"
Claes1981 commented 4 years ago

Thanks, tried it now, "TimeManager": "Categorical([legacy, smooth])":

tune local --tuning-config Test6.json --data-path Test6.npz --logfile Test6.log --plot-path Test6 --verbose 
Traceback (most recent call last):
  File "/home/claes/Chess-tuning-tools/env/bin/tune", line 8, in <module>
    sys.exit(cli())
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/tune/cli.py", line 265, in local
    settings, commands, fixed_params, param_ranges = load_tuning_config(json_dict)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/tune/io.py", line 185, in load_tuning_config
    param_ranges = parse_ranges(json_dict["parameter_ranges"])
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/tune/io.py", line 136, in parse_ranges
    parsed = literal_eval(s)
  File "/usr/lib64/python3.8/ast.py", line 99, in literal_eval
    return _convert(node_or_string)
  File "/usr/lib64/python3.8/ast.py", line 82, in _convert
    return list(map(_convert, node.elts))
  File "/usr/lib64/python3.8/ast.py", line 98, in _convert
    return _convert_signed_num(node)
  File "/usr/lib64/python3.8/ast.py", line 75, in _convert_signed_num
    return _convert_num(node)
  File "/usr/lib64/python3.8/ast.py", line 66, in _convert_num
    _raise_malformed_node(node)
  File "/usr/lib64/python3.8/ast.py", line 63, in _raise_malformed_node
    raise ValueError(f'malformed node or string: {node!r}')
ValueError: malformed node or string: <_ast.Name object at 0x7fd09a625280>

"TimeManager": "(['legacy', 'smooth'])":

tune local --tuning-config Test6.json --data-path Test6.npz --logfile Test6.log --plot-path Test6 --verbose 
2020-09-03 05:51:07,431 INFO     Starting iteration 0
2020-09-03 05:51:07,435 INFO     Testing {'TimeManager': '0.38854381999831666'}
2020-09-03 05:51:07,436 DEBUG    engines.json is prepared:
[{'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine1', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}, {'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine2', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}]
2020-09-03 05:51:07,436 INFO     Start experiment
^C
Aborted!

Before this simple test configuration with just one tuning parameter, I also tried tuning some categorical parameters (two possible text values) and some number parameters together in a multiparameter tune. Then it seemed to work and set the categorical parameters correctly. However, if I paused it, and then resumed, it seemed like it wanted to prune some data (or "reduce_ranges") and stopped with an error doing so.

Just tried to do a similar test: Test7.json:

    "engines": [
        {
            "command": "nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden",
            "fixed_parameters": {
                                "WeightsFile": "/partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz",
                                "Backend": "cudnn-fp16",
                                "NNCacheSize": 800000,
                                "Ponder": false
            }
        },
        {
            "command": "nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden",
            "fixed_parameters": {
                                "WeightsFile": "/partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz",
                                "Backend": "cudnn-fp16",
                                "NNCacheSize": 800000,
                                "Ponder": false,
                                "MoveOverheadMs": 3000
            }
        }
    ],
    "parameter_ranges": {
       "MoveOverheadMs": "Integer(0, 3000)",
       "TimeManager": "(['legacy', 'smooth','alphazero'])"
    },
    "engine1_st": "4",
    "engine2_st": "4",
    "rounds": 1,
    "opening_file": "empty.pgn",
    "adjudicate_draws": false,
    "adjudicate_resign": false
} 
tune local --tuning-config Test7.json --data-path Test7.npz --logfile Test7.log --plot-path Test7 --verbose 
2020-09-03 20:42:25,791 INFO     Starting iteration 0
2020-09-03 20:42:25,791 INFO     Testing {'MoveOverheadMs': 1734, 'TimeManager': 'legacy'}
2020-09-03 20:42:25,791 DEBUG    engines.json is prepared:
[{'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine1', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False'], 'protocol': 'uci'}, {'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine2', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}]
2020-09-03 20:42:25,792 INFO     Start experiment
2020-09-03 20:44:13,814 INFO     Experiment finished (108.022283s elapsed).
2020-09-03 20:44:13,814 DEBUG    Raw result:
Indexing opening suite...
Started game 1 of 2 (engine1 vs engine2)
Finished game 1 (engine1 vs engine2): 1/2-1/2 {Draw by 3-fold repetition}
Score of engine1 vs engine2: 0 - 0 - 1  [0.500] 1
Started game 2 of 2 (engine2 vs engine1)
Finished game 2 (engine2 vs engine1): 1/2-1/2 {Draw by 3-fold repetition}
Score of engine1 vs engine2: 0 - 0 - 2  [0.500] 2
Elo difference: 0.0 +/- 0.0, LOS: nan %, DrawRatio: 100.0 %
Finished match

2020-09-03 20:44:14,142 INFO     Got score: 0.0 +- 0.8734251875761166
2020-09-03 20:44:14,143 INFO     Updating model
2020-09-03 20:44:14,143 INFO     GP sampling finished (0.000138s)
2020-09-03 20:44:14,144 DEBUG    GP kernel: None
2020-09-03 20:44:14,187 INFO     Starting iteration 1
2020-09-03 20:44:14,187 INFO     Testing {'MoveOverheadMs': 2469, 'TimeManager': 'legacy'}
2020-09-03 20:44:14,188 DEBUG    engines.json is prepared:
[{'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine1', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False'], 'protocol': 'uci'}, {'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine2', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}]
2020-09-03 20:44:14,188 INFO     Start experiment
^C
Aborted!
(env) claes@Lenovo-OpenSUSE:~/Chess-tuning-tools> tune local --tuning-config Test7.json --data-path Test7.npz --logfile Test7.log --plot-path Test7 --verbose 
Traceback (most recent call last):
  File "/home/claes/Chess-tuning-tools/env/bin/tune", line 8, in <module>
    sys.exit(cli())
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/tune/cli.py", line 320, in local
    reduction_needed, X_reduced, y_reduced, noise_reduced = reduce_ranges(
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/tune/local.py", line 363, in reduce_ranges
    if value < lb or value > ub:
TypeError: '<' not supported between instances of 'str' and 'int'
kiudee commented 4 years ago

That message is helpful and it appears that reduce_ranges does not properly account for categorical variables.

Claes1981 commented 4 years ago

Should it even process something called "reduce_ranges"? I did not change any settings or ranges before resuming. (I do not know how the program works though.)