Could you tell me the details of the validity of confidence interval of MetaLearners?

py-why / EconML

ALICE (Automated Learning and Intelligence for Causation and Economics) is a Microsoft Research project aimed at applying Artificial Intelligence concepts to economic decision making. One of its goals is to build a toolkit that combines state-of-the-art machine learning techniques with econometrics in order to bring automation to complex causal inference problems. To date, the ALICE Python SDK (econml) implements orthogonal machine learning algorithms such as the double machine learning work of Chernozhukov et al. This toolkit is designed to measure the causal effect of some treatment variable(s) t on an outcome variable y, controlling for a set of features x.

Other

3.88k stars 720 forks source link

Background

Thank you for developing such a great library.

I'm reading the document of MetaLearners of EconML.

The document states that the confidence interval is not typically valid:

due to their unrestricted flexibility, they typically do not offer valid confidence intervals, since it is not clear how arbitrary ML methods trade off bias and variance

However, this paper introduced algorithms to estimate confidence interval by bootstrap, and EconML have effect_interval method to estimate confidence interval of MetaLearners.

I also confirmed that effect_interval returns lower and upper bounds of each point.

T_learner.fit(Y, T, X=X, inference="bootstrap")
T_learner.effect_interval(X_test)

Questions

Should we understand that the result of effect_interval could be invalid? How can we interpret the invalidity?
Could you introduce several papers or documents that explain the reason why the confidence interval of MetaLearners could be invalid? (In my understanding, confidence interval estimated by bootstrap could be said to be valid)

py-why / EconML

Could you tell me the details of the validity of confidence interval of MetaLearners? #647

Background

Questions