p-value of the Average ATT Estimate

[azev77]

Hi @ebenmichael and thank you for this package!

I'm getting some results that seem inconsistent.

syn_Ridge = augsynth(lnComposite_HPI ~ tr, loc.f, Date.f, d_O, progfunc = "Ridge", scm = T)
a = summary(syn_Ridge, inf_type = "conformal") 
plot(syn_Ridge, inf_type = "conformal")

summary reports:

Average ATT Estimate (p Value for Joint Null):  -0.111   ( 0.411 )
L2 Imbalance: 0.180
Percent improvement from uniform weights: 88.1%
Avg Estimated Bias: 0.184
Inference type: Conformal inference
 Time Estimate 95% CI Lower Bound 95% CI Upper Bound p Value
  149   -0.040             -0.073             -0.007   0.027
  150   -0.049             -0.082             -0.016   0.027
  151   -0.062             -0.095             -0.029   0.013
...
  189   -0.120             -0.200             -0.021   0.027
  190   -0.143             -0.223             -0.044   0.020

The corresponding plot is:

The plot matches nicely w/ the individual year ATTs reported by summary. The Average ATT Estimate (-0.111) is equal to the average of the individual annual ATTs. mean(a$att$Estimate[149:190]) However, the p-value of the Average ATT Estimate (0.411) is so much bigger than all the individual annual ATT p-values. The biggest individual annual p-value is .09, the average is .019. We would expect the pv of the Average ATT Estimate to not be so much bigger than the individual annual ATT p-values Why is this? Is there some kind option I need to use? Or is there some kind of bug?

[ebenmichael]

Hmm that is odd. It might be due to the test statistic. By default for the average post-treatment effect the inference procedure uses the sum of the absolute treatment effects as the test statistic, but it could be better to use the absolute sum of the effects. I tried making that quick change in this branch. Can you install from that branch with devtools::install_github("ebenmichael/augsynth", ref = "conformal_test_stat") and see what happens with the average p-value?

[azev77]

@ebenmichael thanks for your quick response! The "p Value for Joint Null" falls from 0.411 to 0.077. But it still seems too high.

I have 42 treatment periods, and only 2/42 p_values are >.033, but somehow the "p Value for Joint Null" = 0.077.

To be clear, here is my understanding: the joint null is: ATT_t = 0 for each treatment period "t" is this correct? Can you point me to where in the code the average p_value is computed?

[azev77]

Also, is it possible for the "Jacknife+" method to provide inference for the joint null hypothesis?

[ebenmichael]

That is what the joint null is, but different test statistics and procedures will have different amounts of power for the joint null. You could try using iid permutations rather than block permutations by choosing type = "iid". Here is where the test statistic is computed, if you want to try other test statistics. One future feature I want to add is the ability to pass in custom test stats.

As for the jackknife+, empirically it won't work for inference on the average treatment effect. In general I think inference for the average treatment effect is still a bit of a murky area for these sorts of approaches, while the point-wise p-values seem to work pretty well and I might take those more seriously. Here that would imply strong evidence for an effect.