search

openml-labs / gama

An automated machine learning tool aimed to facilitate AutoML research.
https://openml-labs.github.io/gama/master/
Apache License 2.0
96 stars 30 forks source link

Hyperparameter check #188

Open HadiKutabi opened 1 year ago

HadiKutabi commented 1 year ago

Currently checking out cool AutoML Frameworks as this one :) for my master thesis.

I noticed that pipelines with faulty hyperparamters are getting compiled and evaluated, despite the implementation of lambda functions for "param_check" in the configuration dictionaries.

is that a bug or is this feature yet to be implemented?

Thank you ^^

PGijsbers commented 1 year ago

It is (temporarily) disabled. They were used to catch configurations which would raise an error anyway (e.g., using some prohibited combination of loss and penalty). In the end there was actually very little benefit to catching these situations early, since all you avoid is some interprocess communication (failure is instantaneous in these situations). It ended up being much simpler to just let the pipelines fail.

One use case where it may be useful is to manually restrict specific combinations which the algorithm may allow but you as a user want to avoid (within your specified hyperparameter ranges). But I have so far not had anyone ask for this behaviour, so I don't know that I want to support this.

simonprovost commented 11 months ago

@HadiKutabi Just FYI: #210 will allow you to resolve your raised issue #188. Feel free to contribute by adding the appropriate parameter checks as in the PR review if you have access to that, if you see any!