search

insongkim / PanelMatch

117 stars 35 forks source link

summary results don't match data #37

Closed marcgrinberg closed 4 years ago

marcgrinberg commented 5 years ago

There seems to be a problem with the output of summary. I noticed this with my own data and see a similar problem with the sample data on the main page.

The evident problem is that reported confidence intervals do not fit with the reported coefficient estimates and standard errors. My guess is this is just a problem in the way the intervals are computed.

  1. The intervals are not centered around the coefficient estimate. Consider the sample data at t+0: 0.4221848-(-0.7932684) = 1.215453 -0.7932684-(-2.029409) = 1.236141

  2. And neither interval is at coef (+/-) 1.96*se

Also, since the package update last week, the output of the sample code (from https://github.com/insongkim/PanelMatch) when I run it in R no longer matches the output on this site.

marcgrinberg commented 5 years ago

This is my output...

library(PanelMatch) PM.results <- PanelMatch(lag = 4, time.id = "year", unit.id = "wbcode2", treatment = "dem", refinement.method = "mahalanobis", data = dem, match.missing = T, covs.formula = ~ I(lag(tradewb, 1:4)) + I(lag(y, 1:4)), size.match = 5, qoi = "att" ,outcome.var = "y", lead = 0:4, forbid.treatment.reversal = FALSE)

get_covariate_balance(PM.results$att, dem, covariates = c("tradewb"), plot = F, ylim = c(-2,2)) tradewb t_4 0.14247452 t_3 0.08363034 t_2 0.11718424 t_1 0.25036846 t_0 0.28859923

PE.results <- PanelEstimate(inference = "bootstrap", sets = PM.results, data = dem) summary(PE.results)

Weighted Difference-in-Differences with Mahalanobis Distance Weighted Difference-in-Differences with Covariate Balancing Propensity Score Matches created with 4 lags

Standard errors computed with 1000 Weighted bootstrap samples

Estimate of Average Treatment Effect on the Treated (ATT) by Period: $summary estimate std.error 2.5% 97.5% t+0 -0.8913640 0.6400197 -2.186967 0.3394582 t+1 -0.4709856 1.0837097 -2.633837 1.6861810 t+2 0.4803681 1.4381344 -2.367757 3.4436938 t+3 1.3447573 1.7370143 -2.153228 4.9218421 t+4 1.0782767 1.9030224 -2.670944 4.8875058

$lag [1] 4

$iterations [1] 1000

$qoi [1] "att"

adamrauh commented 5 years ago

hey @marcgrinberg sorry for taking so long to get back with you. The second issue about the summary results is on my radar. The results on the main page are from an old version of the package and I just need to update them. I believe the first thing you mention also makes sense. The confidence interval is calculated from the bootstrap results empirically, so there's no guarantee about it being exactly centered at the original estimate I don't think.

adamrauh commented 4 years ago

I believe this should be resolved.