search

automl / SMAC3

SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
https://automl.github.io/SMAC3/v2.1.0/
Other
1.06k stars 219 forks source link

Add an example for Algorithm Configuration facade #1127

Open olegzaikin opened 1 month ago

olegzaikin commented 1 month ago

I am going to use SMAC3 to tune a CDCL SAT solver's parameters to minimize the total runtime on a set of instances (CNFs). I suppose that the Algorithm Configuration facade is the right choice. The provided examples clarify the usage of the following facades: Hyperparameter Optimization; Multi Fidelity; BlackBox. However, there is no example where the Algorithm Configuration facade is used. It would be really helpful if such and example is added. Thank you for your work!

dengdifan commented 1 month ago

Hi @olegzaikin thanks for the issue,

the other facades are nearly the same as the AC facades except that they might have different surrogate model, acquisition function optimizer. Could you please provide us more information on the content that should be contained in the examples?

For example, how does your solver look like? Is it a python script or a binary file that can be executed? Does your problem contain instances? What is the optimization target for your problem, is it run time or evaluation cost? Is the problem deterministic or not (this means that the cost is highly influenced by the random seeds)? It would be best if you could also provide a basic script on how to evaluate one configuration on your problem.

With these information, we could add a new example on AC facade. Otherwise, the existing examples on the other facade should suffice.

olegzaikin commented 1 month ago

Hi @olegzaikin thanks for the issue,

Could you please provide us more information on the content that should be contained in the examples?

For example, how does your solver look like? Is it a python script or a binary file that can be executed? Does your problem contain instances? What is the optimization target for your problem, is it run time or evaluation cost? Is the problem deterministic or not (this means that the cost is highly influenced by the random seeds)? It would be best if you could also provide a basic script on how to evaluate one configuration on your problem.

With these information, we could add a new example on AC facade. Otherwise, the existing examples on the other facade should suffice.

Thank you for the reply. Having a pcs file and a set of instances (CNFs), I need to tune a SAT solver to decrease the total runtime (in fact, PAR2, PAR10, or PAR100 are preferable) on the instances. A minimal example is here https://drive.google.com/drive/folders/1L6EE0eDTCCkrUnbk6DOkAe917UuAOd_6?usp=sharing

  1. Download a folder
  2. Download and build kissat from here https://github.com/arminbiere/kissat/releases/tag/rel-3.1.1) and copy the binary file kissat to the folder
  3. Install SMAC3 version 1.4 from here https://github.com/automl/SMAC3/releases/tag/v1.4.0
  4. In the folder, run smac --scenario ./scenario --mode SMAC4AC --seed 0

The algorithm is deterministic while the objective function is PAR10. As a result, in 1 minute the best incumbent is updated 2 times.

@dengdifan I believe it would be really helpful for the community if you provide the corresponding example for SMAC3 of version 2.2 since tuning SAT solvers is of high demand.

mwever commented 1 month ago

Hi @olegzaikin,

thanks for all the details. However, I feel that your minimal example will already be a little bit too complicated for a broader audience as it requires third party software that needs to be built before the example can be executed and also increases maintenance efforts for this example since we need to cross-check whether the example is still working with updated libraries etc.

Do you think that a mockup example might be already enough for the AC community? I.e., a simple example that comes with all the specifics of AC (like a dataset of problem instances, cutoff time, etc.) but instead of actually running a SAT solver we would integrate some dummy config space and executable doing nothing but rolling a dice how long it takes for a certain problem instance? Would that be as useful as the minimal example you described?

olegzaikin commented 1 month ago

Hi @olegzaikin,

thanks for all the details. However, I feel that your minimal example will already be a little bit too complicated for a broader audience as it requires third party software that needs to be built before the example can be executed and also increases maintenance efforts for this example since we need to cross-check whether the example is still working with updated libraries etc.

Do you think that a mockup example might be already enough for the AC community? I.e., a simple example that comes with all the specifics of AC (like a dataset of problem instances, cutoff time, etc.) but instead of actually running a SAT solver we would integrate some dummy config space and executable doing nothing but rolling a dice how long it takes for a certain problem instance? Would that be as useful as the minimal example you described?

Hi @mwever

Sure, an example with a dummy config space and a dummy executable would be sufficient.