search

edunford / tidysynth

A tidy implementation of the synthetic control method in R
Other
98 stars 14 forks source link

grab_significance() - fewer controls question #18

Closed Je-gh closed 1 year ago

Je-gh commented 2 years ago

Dear @edunford, I'm sorry if this question is trivial, but I have struggled to find an answer elsewhere. In your R documentation, when talking about grab_significance() you mention: One needs at least 20 control case to use the conventional .05 level. With fewer cases, significance levels need to be adjusted to accommodate the low total rank. This is a bug of rank based significance metrics.

Could you please point out to a resource that explains how this adjustment is done (to e.g., 9 instead of 20 units)?

Thank you in advance!

edunford commented 1 year ago

Wow @Je-gh, so sorry for the 6 month delay in this response. This 20 controls logic simply stems from logic underpinning a p-value (i.e. if re-sampled the data again and again, we'd only see this result 1 time out of 20, or 5% of the time). The less units you have, the harder it is to make that claim. I can't remember the exact adjustment, did you ever figure out the correct approach?