Highest density interval not implemented for categorical parameters

Chess Tuning Tools version: 0.9.3
Python version: 3.9
Operating System: Linux

Description

I ran a simple tune job with some categorical parameter ranges. After a few hours the job failed with a NotImplementedError. I didn't have this problem before when using just Integer ranges.

What I Did

I ran a simple tune local -c simple.json -vv with the following parameter ranges:

   "parameter_ranges": {                    
       "QS_A": "Integer(0, 500)",           
       "QS_B": "Integer(0, 300)",           
       "EVAL_ROUGHNESS": "Integer(0, 40)",  
       "USE_BOUND_FOR_CHECK_TEST": "(0, 1)",
       "IID_LIMIT": "Integer(0, 5)",        
       "IID_REDUCE": "Integer(1, 5)",       
       "IID_TYPE": "(0, 1, 2, 3)"           
   },

After completing the first 15 iterations, suddenly tune crashed with the following trace:

...
2023-01-06 02:08:56,406 INFO     Experiment finished (720.901534s elapsed).
2023-01-06 02:08:57,041 INFO     Got Elo: 18.302996224270025 +- 28.660450847409518
2023-01-06 02:08:57,043 INFO     Estimated draw rate: 18.66%
2023-01-06 02:08:57,043 INFO     Updating model
2023-01-06 02:10:13,454 INFO     GP sampling finished (76.410967s)
2023-01-06 02:10:13,455 DEBUG    GP kernel: 0.988**2 * Matern(length_scale=[0.202, 0.242, 0.346, 0.3
2, 0.37, 0.402, 0.208], nu=2.5) + WhiteKernel(noise_level=5.94e-06)
2023-01-06 02:10:13,591 INFO     Starting iteration 16
2023-01-06 02:10:29,150 INFO     Current optimum:
{'QS_A': 171, 'QS_B': 40, 'EVAL_ROUGHNESS': 26, 'USE_BOUND_FOR_CHECK_TEST': 1, 'IID_LIMIT': 0, 'IID_
REDUCE': 3, 'IID_TYPE': 1}
2023-01-06 02:10:29,152 INFO     Estimated Elo: 104.8064 +- 31.2139
2023-01-06 02:10:29,153 INFO     90.0% confidence interval of the Elo value: (nan, nan)
Traceback (most recent call last):
  File "/fsx/users/ahle/conda/bin/tune", line 8, in <module>
    sys.exit(cli())
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/tune/cli.py", line 437, in local
    current_optimum, estimated_elo, estimated_std = print_results(
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/tune/local.py", line 577, in print_results
    confidence_out = confidence_intervals(
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/tune/summary.py", line 57, in confidence_i
ntervals
    intervals = optimizer.optimum_intervals(
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/bask/optimizer.py", line 650, in optimum_i
ntervals
    raise NotImplementedError(
NotImplementedError: Highest density interval not implemented for categorical parameters.

I've now changed my settings to use Integer(0, 1, 2, 3) instead of the categorical (0, 1, 2, 3). But of course this is not ideal.

kiudee / chess-tuning-tools

Highest density interval not implemented for categorical parameters #217

Description

What I Did