search

py-why / EconML

ALICE (Automated Learning and Intelligence for Causation and Economics) is a Microsoft Research project aimed at applying Artificial Intelligence concepts to economic decision making. One of its goals is to build a toolkit that combines state-of-the-art machine learning techniques with econometrics in order to bring automation to complex causal inference problems. To date, the ALICE Python SDK (econml) implements orthogonal machine learning algorithms such as the double machine learning work of Chernozhukov et al. This toolkit is designed to measure the causal effect of some treatment variable(s) t on an outcome variable y, controlling for a set of features x.
https://www.microsoft.com/en-us/research/project/alice/
Other
3.77k stars 713 forks source link

Comparison of confidence intervals between CausalForestDML and the R grf package #479

Open gcasamat opened 3 years ago

gcasamat commented 3 years ago

I trained the same model, both with the econml CausalForestDML and the R grf causal_forest algorithms. The average marginal effects obtained are comparable, around 0.12. The confidence intervals are however quite different: [0.064,0.166] with causal_forest, and [-0.360,0.621] with CausalForestDML. Do you have an idea of why there is such a difference? Thanks

vsyrgkanis commented 3 years ago

Interesting… we had done similar comparisons and wasnt seeing such big differences.

could you provide code reproducing these results?

vsyrgkanis commented 3 years ago

Oh is this for the average effect??

make sure you use: ate__inference() Which is a doubly robust ate on the training data. That’s the comparable result.

ate_inference(Xtest) because it is on an arbitraty test set we can only provide for now conservative intervals as we also point in the asterisk in the summary frame. So these would be much wider

vsyrgkanis commented 3 years ago

You could also just call “summary()” on the cfdml object and you’ll see these results (assuming your treatment is categorical)

gcasamat commented 3 years ago

Thank you for your reply. I have omitted to mention that my treatment is continuous.

gcasamat commented 3 years ago

Here is the code: 

est_forest = CausalForestDML(model_y = RegressionForest(random_state = 123), model_t = RegressionForest(random_state = 123), random_state = 123)
est_forest.fit(Y, T, X = W, W = W, inference = 'auto') 
est_forest.marginal_ate_inference(T, X=W)
vsyrgkanis commented 3 years ago

Unfortunately then you'll need to wait a few weeks for the doubly robust correction to be implemented for continuous treatments. That correction requires yet one more ML model to be provided by the user, which was why we were hesitant to add that in the first round, but we've found other users/use cases where this is important so we'll add it soon.

Currently you are getting the conservative intervals by using ate_inference(Xtest), which is why you see the difference on the average effects.

vsyrgkanis commented 3 years ago

For now if you want a "non-conservative" (the other direction) interval, you could look at the "std_point" on the second table of the ate_inference results and divide that by the sqrt{number of samples} and then create an 95%-confidence intervals via:

lb = scipy.stats.norm.ppf(.025, loc=mean_point, scale = std_point / sqrt{number of samples})
ub = scipy.stats.norm.ppf(1 - .025, loc=mean_point, scale = std_point / sqrt{number of samples})

but this could potentially by too optimistic. Though I suspect this will be much closer to what the grf package provides.

gcasamat commented 3 years ago

Thanks a lot for the advice! I will try this and also the new econml version with the doubly robust correction.