search

py-why / EconML

ALICE (Automated Learning and Intelligence for Causation and Economics) is a Microsoft Research project aimed at applying Artificial Intelligence concepts to economic decision making. One of its goals is to build a toolkit that combines state-of-the-art machine learning techniques with econometrics in order to bring automation to complex causal inference problems. To date, the ALICE Python SDK (econml) implements orthogonal machine learning algorithms such as the double machine learning work of Chernozhukov et al. This toolkit is designed to measure the causal effect of some treatment variable(s) t on an outcome variable y, controlling for a set of features x.
https://www.microsoft.com/en-us/research/project/alice/
Other
3.82k stars 714 forks source link

Using FLAML in tune() methods #557

Open EgorKraevTransferwise opened 2 years ago

EgorKraevTransferwise commented 2 years ago

It's great to have tune() methods in some of the model classes (though sadly not all as yet). But they all seem to do tuning via grid search, which is quite wasteful. Could you possibly add an option to use FLAML instead of grid search, to search the hyperparameter space - should be quite easy if there's already a spec in the code for the parameter space of each class to be tuned.

kbattocchi commented 2 years ago

Thanks for the suggestion - we'll look into it.

EgorKraevTransferwise commented 2 years ago

A solution for some usecases (for now, CATE for binary treatments, extensions pending) is implemented in our auto-causality library, which searches over both individual model's parameters and across models to optimize a given score. Requests for extending the usecases it supports (via raising issues) are very welcome!

EgorKraevTransferwise commented 2 years ago

@kbattocchi as we discussed, I'm preparing a PR to have the hyperparameter search spaces available as @classmethods of the respective model. To do this, I need to import flaml.tune. Would you be happy for me to add flaml to requirements.txt, or if not, how should I do this? I could do a try-catch block inside the class that tries to import flaml.tune and only creates the classmethod if successful, but that feels kinda ugly...

kbattocchi commented 2 years ago

I think ideally we should be providing the ranges in an optimizer-agnostic way, so that users can use FLAML if they want, but could use something like hyperopt instead if that suits their environment better.

If there is a need to use FLAML-specific functionality somewhere, then you may be able to try to import it but use the MissingModule class from econml.utilities in its place if it can't be imported - see e.g. https://github.com/microsoft/EconML/blob/ac12f54f4304c730d1501edaaeae2b96e9af4507/econml/_tree_exporter.py#L18-L28

EgorKraevTransferwise commented 2 years ago

How about I wrap the FLAML imports into MissingModule for now, and add an optimizer-agnostic spec as soon as someone raises an issue asking for it?