search

automl / auto-sklearn

Automated Machine Learning with scikit-learn
https://automl.github.io/auto-sklearn
BSD 3-Clause "New" or "Revised" License
7.54k stars 1.27k forks source link

[Request] Allow portfolio and selector models to be set through hyperparameters in ASKL2 #1197

Open PGijsbers opened 3 years ago

PGijsbers commented 3 years ago

As per the title, it will be useful for ASKL2 to have a configurable portfolio and policy selector. It's beneficial for research (avoiding 'cheating' through meta-learning in a benchmark) or for customization.

Issue opened on the request of @mfeurer

eddiebergman commented 3 years ago

Hi @PGijsbers, this would certainly be a nice feature and definitely something to have in once asklearn 2 becomes non-experimental. I had a quick look to see how possible this is and I don't imagine it's too bad depending on the definition of configurable for policy selector and portifolio.

Simply changing the portifolio file location and enabling disabling it should be quite straightforward and likewise with white listing or black listing certain policys. However if there is a need for injecting a custom policy selector object it might be a bit more involved ... but also manageable.

Do you need this for the automlbenchmark and do you have a time urgency for this?

PGijsbers commented 3 years ago

Yes it's something we'd ultimately like to use in the AutoML benchmark. Though realistically I don't think we can wait for this to be all done and set up properly before doing our next set of experiments, which have been delayed for months already.

eddiebergman commented 3 years ago

No problem, seeing as this will probably not be done in time for the benchmarks, I will talk to @mfeurer about how we do this. Perhaps autosklearn2 will become the default before the next benchmark run.

zhuygln commented 1 year ago

Hi, We have been using autosklearn and automlbenchmark in R&D at SAS Institiute for autoML pipeline. Huge applauses to the great work of you all first! @eddiebergman @PGijsbers

I have a few questions regarding portfolio.

  1. I am trying to replicate the portfolio building in the autosklearn2 paper. Is these any code shared in this repo? I search the portfolio file but I couldn't find it. Maybe I have don't have the permission to the version that you are referring to.
  2. For my current research project, I'd like to add/delete portfolio. Is this supported? If not, I am glad to contribute by adding this feature.

Thanks, Yonglin