search

xuyiqing / fect

fixed effect counterfactual estimators
Other
57 stars 20 forks source link

p value and placebo test #35

Closed ireneb612 closed 1 year ago

ireneb612 commented 1 year ago

maybe a stupid question, but what if my p values are not significant but my plavebo test pass ? What about the opposite? What can I deduce from it? Is the p value essential to have relevant result?

xuyiqing commented 1 year ago

Hi, from a frequentist's perspective, it's either rejecting the null or not rejecting the null. When you have a large p-value, with which you don't reject the null, it can be that your null is correct, or you simply do not have enough power. Hope this helps!

ireneb612 commented 1 year ago

@xuyiqing Sorry I probably did not explain correctly, I meant if I get a p values that is not significative for the ATT value when printing for example : print(out.mc)

ATT: ATT S.E. CI.lower CI.upper p.value Tr obs equally weighted 57.14 41.31 -23.83 138.1 0.1666 Tr units equally weighted 57.14 41.31 -23.83 138.1 0.1666

But then I pass both placebo test, Placebo test p-value: 0.738 Placebo equivalence test p-values : 0.00

What does it mean?

xuyiqing commented 1 year ago

Hi it means that you cannot reject the null that ATT = 0 while at the same time, you cannot reject the null that pretreatment residual averages are zero. Please refer to our paper when you have a conceptual question like this. Thank you!