Closed vinnyarmentano closed 3 years ago
(Bringing this to the attention of @isabelonate & @dsantamaria94)
Thanks for reaching out. The main difference between Duflo's DML tutorial and our test_calibration function is that we assume we're in an observational study where the propensity score need to be estimated, whereas Duflo works in a randomized study setting where the propensity score is known (usually constant). In response to the specific questions:
observation_weight
is not the same thing as an inverse-propensity weight; and in fact observation_weight
is typically 1 for everyone. We use observation weights when we want to add in an additional layer of weighting (e.g., if some observations are censored, so we want to do inverse-probability of censoring weighting with the data we have), but that usually only comes up in special cases.Overall, if you're in an RCT (i.e., working with known propensities), then doing the calibration check as in Duflo's slides should be perfectly fine -- and would presumably give very similar results to what we get using test_calibration
.
Dear @swager and @vinnyarmentano,
thank you for asking this question, @vinnyarmentano. I had a very similar question.
The modification of Chernozhukov et al.'s approach to "work" in observational studies seems very useful (and I am currently working on applying it). Have you mentioned that in a paper we could cite? Or shall we just cite grf?
Thanks a lot for developing these tools and answering questions!
Hans
Hi GRF Team,
I am interested in understanding the differences between the Best Linear Predictor (BLP) specifications employed by GRF’s
test_calibration()
command and Chernozhukov et al.’s "Generic Machine Learning Inference on Heterogenous Treatment Effects in Randomized Experiments." arXiv preprint arXiv:1712.04802 (2017)”.I understand from the
test_calibration()
help file that there is a slightly different aim for the GRF BLP specification than the Chernozhukov et al. BLP, with Beta1 (mean.forest.prediction
) being a test of the ability of the forest to have identified an ATE. Whereas with the Beta1 from the Chernozhukov et al. BLP, the coefficient itself should be the ATE in the same units as the dependent variable.Referencing the code provided for Dr. Duflo’s 2018 NBER SI Lecture for the DML BLP and the
test_calibration()
function’s code as the CF BLP there are 5 differences I have been able to identify. (These differences will be listed below)My questions of interest are;
The 5 differences I’m interested in understanding, with relevant lines for each method;
mean.forest.prediction
aka Beta1 construction: The DML simply subtracts observations’ propensity of being treated while GRF multiplies that by a weighted mean of observations CATEs. (DML Line 271 & CF line 52)differential.forest.prediction
aka Beta2 term construction: The DML uses a simple mean of CATEs when demeaning CATEs while GRF uses a weighted mean according to each observations inverse propensity of being treated. (DML Line 269 & CF lines 49 & 53)I hope all of the above helps define the context of my questions. I’m more than happy to clarify any of the above thoughts if it’ll be of assistance. I’m really looking forward to understanding this. Thank you in advance for your help.
Best, Vinny Armentano
References: Double Machine Learning Reference Links here;
The
test_calibration()
code I referenced is from here.I also wrote a brief test script that runs both methods’ BLP specifications on the simulated dataset created by the example in the
test_calibration()
help file. It is attached to this issue. The two different BLP specifications result in similarly interpreted Beta2 terms. Both also come to the conclusion that an ATE is present. BestLinearPredictor_GRF_DML.txt