search

microsoft / FLAML

A fast library for AutoML and tuning. Join our Discord: https://discord.gg/Cppx2vSPVP.
https://microsoft.github.io/FLAML/
MIT License
3.93k stars 512 forks source link

A Curious Question about Prediction Quality #214

Closed reshalfahsi closed 3 years ago

reshalfahsi commented 3 years ago

Hi there,

Is it possible to obtain prediction quality? Is there any computation under the hood within this package to find the conformal prediction? If there weren't any could you please add it as the feature request?

Thank you.

sonichi commented 3 years ago

I'm making it a feature request. What package do you use today to get prediction quality for models like lightgbm or random forest?

reshalfahsi commented 3 years ago

Have you heard about nonconformist package? I currently use that one.

sonichi commented 3 years ago

It seems that you can first run flaml to do hyperparameter tuning and model selection, and then pass flaml.model to the package. Give that a try?

reshalfahsi commented 3 years ago

I haven't but maybe I will give it a try.

reshalfahsi commented 3 years ago

Hi @sonichi,

It seems the nonconformist package requires the user to train alongside the wrapper from the package. You have to wrap the model and do the magic. I recommend that you should implement the algorithm by yourself because when I apply XGBoost model I have to tweak the package a little bit.

sonichi commented 3 years ago

Yes it uses a wrapper. I was thinking that you could take the tuned model from flaml, wrap it with the package and retrain. Is there any issue with this approach? What tweak is required for XGBoost?

sonichi commented 3 years ago

@reshalfahsi I'm closing this issue as it is inactive for a month. If you still request this feature, feel free to reopen and follow up.