search

xuyiqing / fect

fixed effect counterfactual estimators
Other
51 stars 18 forks source link

Power analysis for ife model #40

Open nthrpy opened 1 month ago

nthrpy commented 1 month ago

Hi- what is the recommended way to check the statistical power (for instance, when the resulting ATT=5.0, with p>0.1, where H0: ATT=0)?

xuyiqing commented 1 month ago

I'm sorry. I don't fully understand the question. Could you elaborate?

nthrpy commented 1 month ago

I would like to identify the statistical power to detect a certain ATT (e.g. ATT=5), given the quasi-experiment conditions (e.g. 15 treated / 30 control units, pre-treatment periods = 50, assuming significance level = 0.1).

The background of this question is that, I used to measure impact of treatment (test vs control units, during treatment periods) simply by t-test. In that case, whenever I encounter a non-significant result (e.g. p=0.43), then I can use standard R/python packages to solve for statistical power by assuming effect size = Cohen's d, significance level = 0.1, 2-tailed test. If the power is way below 80%, then I know the quasi-experiment is underpower to detect the observed impact (e.g. mean diff = 5).

To overcome assumptions of parallel trend and to control for unobserved covariates, I have applied the IFEct model to the same test, and obtained somewhat similar impact estimate (similar to t-test), and a non-significant p-value (consistent with the t-test).

How would I check the statistical power with the IFEct model?

Let me know if I can further clarify.