search

asheshrambachan / HonestDiD

Robust inference in difference-in-differences and event study designs
Other
175 stars 45 forks source link

Inquiry Regarding Confidence Interval Behavior in createSensitivityResults_relativeMagnitudes() Function #49

Closed lzy318 closed 9 months ago

lzy318 commented 10 months ago

I hope this message finds you well. Firstly, I would like to express my appreciation for the development of your package, which I've found to be extremely convenient and useful in my work.

However, I am encountering a puzzling issue while utilizing the createSensitivityResults_relativeMagnitudes() command, specifically in relation to the generation of confidence intervals under varying values of M_bar. When I adjust M_bar to larger values, the resulting confidence intervals become narrower, which appears somewhat counter-intuitive.

To illustrate, consider the following output:

honest.twfe <- createSensitivityResults_relativeMagnitudes(betahat = beta.twfe,
                                                           sigma = vcov.twfe,
                                                           numPrePeriods = 12,
                                                           numPostPeriods = 1,
                                                           l_vec = c(1),
                                                           Mbarvec = c(0,0.25,0.5,0.75,1),
                                                           gridPoints = 3000)
honest.twfe
      lb    ub method Delta    Mbar
   <dbl> <dbl> <chr>  <chr>   <dbl>
1 0.0544 0.177 C-LF   DeltaRM  0   
2 0.0565 0.175 C-LF   DeltaRM  0.25
3 0.0516 0.178 C-LF   DeltaRM  0.5 
4 0.0359 0.193 C-LF   DeltaRM  0.75
5 0.0187 0.209 C-LF   DeltaRM  1

Notably, the confidence interval for M_bar = 0.25 is narrower than for M_bar = 0. Additionally, this interval contrasts with the one generated using constructOriginalCS(), which is considerably wider:

constructOriginalCS(betahat = beta.twfe,
                    sigma = vcov.twfe,
                    numPrePeriods = 12,
                    numPostPeriods = 1,
                    l_vec = c(1))
  lb[,1] ub[,1] method   Delta
   <dbl>  <dbl> <chr>    <lgl>
1 0.0786  0.154 Original NA

I am seeking some insights or explanations regarding this behavior. Your guidance on understanding these discrepancies would be immensely helpful. Additionally, if necessary, I can provide a working example for further investigation.

Thank you for your time and assistance.

Best regards,

jonathandroth commented 9 months ago

I would guess it's a numerical precision issue (HonestDiD inverts tests over a finite grid of points and then collects them to form the CI). Can you try re-running with fixed values of the upper and lower bounds for the grid (parameters grid.ub and grid.lb) and tell me if that fixes it?

On Mon, Jan 8, 2024 at 6:10 PM Ziyi Liu @.***> wrote:

I hope this message finds you well. Firstly, I would like to express my appreciation for the development of your package, which I've found to be extremely convenient and useful in my work.

However, I am encountering a puzzling issue while utilizing the createSensitivityResults_relativeMagnitudes() command, specifically in relation to the generation of confidence intervals under varying values of M_bar. When I adjust M_bar to larger values, the resulting confidence intervals become narrower, which appears somewhat counter-intuitive.

To illustrate, consider the following output:

honest.twfe <- createSensitivityResults_relativeMagnitudes(betahat = beta.twfe, sigma = vcov.twfe, numPrePeriods = 12, numPostPeriods = 1, l_vec = c(1), Mbarvec = c(0,0.25,0.5,0.75,1), gridPoints = 3000) honest.twfe A tibble: 5 x 5

lb ub method Delta Mbar

1 0.0544 0.177 C-LF DeltaRM 0 2 0.0565 0.175 C-LF DeltaRM 0.25 3 0.0516 0.178 C-LF DeltaRM 0.5 4 0.0359 0.193 C-LF DeltaRM 0.75 5 0.0187 0.209 C-LF DeltaRM 1 Notably, the confidence interval for M_bar = 0.25 is narrower than for M_bar = 0. Additionally, this interval contrasts with the one generated using constructOriginalCS(), which is considerably wider:

constructOriginalCS(betahat = beta.twfe, sigma = vcov.twfe, numPrePeriods = 12, numPostPeriods = 1, l_vec = c(1)) A tibble: 1 x 4

lb[,1] ub[,1] method Delta

1 0.0786 0.154 Original NA

I am seeking some insights or explanations regarding this behavior. Your guidance on understanding these discrepancies would be immensely helpful. Additionally, if necessary, I can provide a working example for further investigation.

Thank you for your time and assistance.

Best regards,

— Reply to this email directly, view it on GitHub https://github.com/asheshrambachan/HonestDiD/issues/49, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE6EXFDM6W2V42XKQ42EYA3YNR4FTAVCNFSM6AAAAABBSGLMRSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA3TCMZWHEZTGNI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

lzy318 commented 9 months ago

Thanks for your prompt response, I re-run the codes with fixed grid but the problem persists. Here is my code:

honest.twfe <- createSensitivityResults_relativeMagnitudes(betahat = beta.twfe,
                                                           sigma = vcov.twfe,
                                                           numPrePeriods = 12,
                                                           numPostPeriods = 1,
                                                           l_vec = c(1),
                                                           Mbarvec = c(0,0.25,0.5,0.75,1),
                                                           grid.ub = 0.8,
                                                           grid.lb = -0.8)

      lb    ub method Delta    Mbar
   <dbl> <dbl> <chr>  <chr>   <dbl>
1 0.0553 0.175 C-LF   DeltaRM  0   
2 0.0569 0.174 C-LF   DeltaRM  0.25
3 0.0521 0.177 C-LF   DeltaRM  0.5 
4 0.0360 0.191 C-LF   DeltaRM  0.75
5 0.0200 0.209 C-LF   DeltaRM  1

Let me know if you need the data to replicate this issue. And I am still confused about the difference between the confidence interval when Mbar equals to zero and the confidence interval given by constructOriginalCS(). They should be similar right? But in my case they are not.

jonathandroth commented 9 months ago

Can you please share a reproducible example with data?

On Tue, Jan 9, 2024, 2:33 PM Ziyi Liu @.***> wrote:

Thanks for your prompt response, I re-run the codes with fixed grid but the problem persists. Here is my code:

honest.twfe <- createSensitivityResults_relativeMagnitudes(betahat = beta.twfe, sigma = vcov.twfe, numPrePeriods = 12, numPostPeriods = 1, l_vec = c(1), Mbarvec = c(0,0.25,0.5,0.75,1), grid.ub = 0.8, grid.lb = -0.8)

  lb    ub method Delta    Mbar
1 0.0553 0.175 C-LF DeltaRM 0 2 0.0569 0.174 C-LF DeltaRM 0.25 3 0.0521 0.177 C-LF DeltaRM 0.5 4 0.0360 0.191 C-LF DeltaRM 0.75 5 0.0200 0.209 C-LF DeltaRM 1 Let me know if you need the data to replicate this issue. And I am still confused about the difference between the confidence interval when Mbar equals to zero and the confidence interval given by constructOriginalCS(). They should be similar right? But in my case they are not. — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you commented.Message ID: ***@***.***>
lzy318 commented 9 months ago

Thanks for the quick reply, this is the link to the data.

https://drive.google.com/file/d/14IoDrnCulGmUfCyBJkqmNp-iObSIgw_3/view?usp=sharing

Here is the code for the event study.

`rm(list = ls()) library(HonestDiD) library(fixest) load("example.RData") twfe.est <- feols(general_sharetotal_A_all ~ i(Time_to_Treatment, treat, ref = -1) + cand_H_all + cand_B_all| district_final + cycle,
data = data.twfe, cluster = "district_final")

beta.twfe <- twfe.est$coeftable[c(4:16),1] vcov.twfe <- summary(twfe.est)$cov.scaled[c(4:16),c(4:16)]

honest.twfe <- createSensitivityResults_relativeMagnitudes(betahat = beta.twfe, sigma = vcov.twfe, numPrePeriods = 12, numPostPeriods = 1, l_vec = c(1), Mbarvec = c(0,0.25,0.5,0.75,1), grid.ub = 0.8, grid.lb = -0.8) honest.twfe original.twfe <- constructOriginalCS(betahat = beta.twfe, sigma = vcov.twfe, numPrePeriods = 12, numPostPeriods = 1, l_vec = c(1)) original.twfe`

jonathandroth commented 9 months ago

Hmm, that is weird. I am able to replicate your strange result. However, if I include the second post-treatment coefficient and then tell it I'm only interested in the first one (l_vec = c(1,0)), then I get the expected result that the CI for M=0 is nearly identical to the original CI, and the CIs expand as M increases. 

library(fixest)
load("~/Downloads/example.RData")
twfe.est <- feols(general_sharetotal_A_all ~ i(Time_to_Treatment, treat, ref = -1) + cand_H_all + cand_B_all| district_final + cycle,
                  data = data.twfe, cluster = "district_final")

beta.twfe <- twfe.est$coeftable[c(4:17),1]
vcov.twfe <- summary(twfe.est)$cov.scaled[c(4:17),c(4:17)]

honest.twfe <- createSensitivityResults_relativeMagnitudes(betahat = beta.twfe,
                                                           sigma = vcov.twfe,
                                                           numPrePeriods = 12,
                                                           numPostPeriods = 2,
                                                           l_vec = c(1,0),
                                                           Mbarvec = c(0,0.25,0.5,0.75,1),
                                                           grid.ub = 0.8,
                                                           grid.lb = -0.8)

IIRC, HonestDiD has a special subroutine for the case without nuisance parameters (i.e. only one post-treatment period), and I'm wondering if there is a bug there. @mcaceresb when you have some RA time, could you look into this please?

CCing @asheshrambachan since I think you wrote this code initially and may have a better idea where the bug is.

mcaceresb commented 9 months ago

@jonathandroth The critical values are different.

  1. The least-favorable critical value is larger here when using the "no-nuisance" parameter logic.

  2. If the least-favorable CV is not rejected, the next critical value is obtained from the generalized normal, which is also different.

    • This no-nuisance call uses the values returned from .VLoVUpFN for the lower and upper bounds, whereas this nuisance call uses the values from .lp_dual_fn; I'm not quite sure why but they end up being rather different, but they do.