modelfoxdotdev / modelfox

ModelFox makes it easy to train, deploy, and monitor machine learning models.
Other
1.46k stars 63 forks source link

Allow fixing a single parameter value #46

Closed otsaw closed 2 years ago

otsaw commented 2 years ago

I have two potential use cases where I'd like to fix a single parameter value, but otherwise get the full default parameter grid. If I understand correctly, currently I'd have to specify that full grid in quite verbose JSON, which is a bit much.

  1. For the data I have (lots of dummy variables), both linear and GBDT models do well and, perhaps depending on the downsampling stochastics, sometimes I get a linear model as best, sometimes GBDT. I'd like to fix that so that only linear models are tried (or only GBDT), because I don't want a discontinuity in production of having now a linear model, a month later GBDT, then linear again, etc.
  2. If I oversample (which I'm not currently doing), I believe I need to fix the min_examples_per_node to more than the upsampling replication count. I'd like to set that but otherwise get the default grid.

If you want to keep the CLI simple, having these via Python (#12) would be fine for me too.

otsaw commented 2 years ago

All of the parameters that you do not specify will be filled with their default parameter values.

Instead of the default value, I'd like to get the default grid search for those, so that the unspecified parameters would be optimized.

nitsky commented 2 years ago

Hey @otsaw, we have been thinking of adding an "autogrid" config option that will allow you to customize the default hyperparameter grid.

For example, to restrict the default grid to linear models, you would be able to do something like this:

{
  "train": {
    "autogrid": {
      "model_types": ["linear"]
    }
  }
}

For more complex cases like your example with min_examples_per_node, I think that will be best supported by training directly from python, when #12 is implemented.

How does that sound?

otsaw commented 2 years ago

That sounds good :+1:

deciduously commented 2 years ago

66 closes