covariate balance

[chrisw475]

Hi PanelMatch team,

I'm getting an error on get_covariate_balance with this code.

Thanks, Chris

data<-read_dta(file="http://chriswarshaw.com/papers/export.dta") data<-as.data.frame(data) data$year2<-as.integer(data$year2) PM.results <-PanelMatch(lag = 4, time.id = "year2", unit.id = "D_ST2", treatment = "right_to_work", refinement.method = "mahalanobis", data = data, match.missing = FALSE, listwise.delete = TRUE, # listwise deletion used covs.formula = ~ I(lag(policy_updated, 1:4))+ I(lag(GovDem, 1:4)), size.match = 5, qoi = "att", outcome.var = "policy_updated", lead = 0:4, forbid.treatment.reversal = FALSE, use.diagonal.variance.matrix = TRUE) PE.results <- PanelEstimate( sets = PM.results, data = data)

get_covariate_balance(PM.results$att, data = data, covariates = c("GovDem"), plot = FALSE)

[adamrauh]

@chrisw475 Hi Chris,

Thanks so much for continuing to use the package and raising this! I believe I found the issue and pushed up a patch to the master branch. Could you try switching to that version and seeing if it works:

library(devtools)
install_github("insongkim/PanelMatch", dependencies=TRUE)

I was able to get it to run on my end with this change.

Thanks again! Adam

[chrisw475]

That worked. Thanks!!

Also, it looks like you guys turned off the earlier functionality to calculate "treatment effects" in pre-treatment periods using something like 'lead = -2:4'. It's definitely possible I'm thinking about it wrong, but I thought that was a helpful way to diagnose possible pre-treatment trends (aka, non-parallel trends).

On Mon, Jul 27, 2020 at 7:11 PM Adam Rauh notifications@github.com wrote:

@chrisw475 https://github.com/chrisw475 Hi Chris,

Thanks so much for continuing to use the package and raising this! I believe I found the issue and pushed up a patch to the master branch. Could you try switching to that version and seeing if it works:

library(devtools) install_github("insongkim/PanelMatch", dependencies=TRUE)

I was able to get it to run on my end with this change.

Thanks again! Adam

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insongkim/PanelMatch/issues/64#issuecomment-664684132, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEHYHOJQ4XOAMTFDP5BUC3R5YCSFANCNFSM4PJFCJJA .

[insongkim]

@chrisw475 You can check the parallel trend with the pre-treatment trend of the outcome variable. We removed the functionality as PanelMatch function was not actually designed to take any negative values in the lead argument (we had to remove it when we learned that users are actually using it for “placebo tests” although we did not write the function for that task). I hope this is useful.

[junhyeokjang1]

@insongkim Hello, Professor Kim.

Could you elaborate what you mean by "You can check the parallel trend with the pre-treatment trend of the outcome variable"? When I run a PanelMatch, I acquire a number of matched sets (PM.results). Do you mean that I can check the pre-treatment trend within these matched sets? Also, these matched sets (PM.results) do not seem to account for matching refinement. If my understanding is correct, the parallel trend assumption must hold within a refined matched set. Any practical suggestions for testing this assumption, using the current version of PanelMatch?

[insongkim]

@hijjang13 Thanks for asking these questions.

1. The covariate balance plot will help you evaluate whether the outcome variable difference (between treated vs. control units) is relatively flat in the pre-treatment period. The plot returns a standardized mean difference: of course, you can do various other diagnostics if you need using the matched sets object.

2. I suggest you look into each element of the matched sets object. Here is an example. I hope this makes sense.

## No refinement

PM.results.none <- PanelMatch(lag = 4, time.id = "year", unit.id = "wbcode2", 
                              treatment = "dem", refinement.method = "none",  #note that refinement method = "none"
                              data = dem, match.missing = TRUE, 
                              size.match = 5, qoi = "att", outcome.var = "y",
                              lead = 0:4, forbid.treatment.reversal = FALSE, 
                              use.diagonal.variance.matrix = TRUE)

msets <- PM.results.none$att #extract the matched sets

attr(msets[[1]], "weights") #all weights for controls in the first matched set are equal

## applying refinement
PM.results <- PanelMatch(lag = 4, time.id = "year", unit.id = "wbcode2", 
                         treatment = "dem", refinement.method = "mahalanobis", 
                         data = dem, match.missing = TRUE, 
                         covs.formula = ~ I(lag(tradewb, 1:4)) + I(lag(y, 1:4)), # lags
                         size.match = 5, qoi = "att", outcome.var = "y",
                         lead = 0:4, forbid.treatment.reversal = FALSE, 
                         use.diagonal.variance.matrix = TRUE)

msets <- PM.results$att #extract the matched sets

attr(msets[[1]], "weights") #Different weights. 5 most similar control units have non-zero weight.
sum(attr(msets[[1]], "weights")) #these weights are normalized and will always sum to 1

[pietrosantoleri]

Hello PanelMatch team, and thanks for the great work. Could you kindly elaborate a bit more on why including negative values in lead would be wrong/misleading (and not informative on parallel trends) given the current features of the PanelMatch? Many thanks!

[adamrauh]

Hi @pietrosantoleri , thanks for using the package! The issue with negative lead values isn't that they're "wrong" or misleading, per se, just that there are implementation issues that haven't been worked out yet. In writing the package, the assumption was made that the "lead window" (as specified by the lead argument) was the set of periods after or including the treatment period where the att/atc/ate is estimated, which implies that these values would be >=0. So, the code just wasn't set up to handle negative values and will have undefined behavior in these cases. This seems like something a fair number of people want though, so we're working on it!

[pietrosantoleri]

@adamrauh thank you very much for your answer!

[antonpeez]

Dear PanelMatch team,

Thank you very much for the terrific package!

Using the exact code in the original post above (as well as my own projects), I am currently getting an error when using the get_covariate_balance command:

Error in FUN(newX[, i], ...) : unused argument (simplify = TRUE)

I am running the PanelMatch version from the se_comparison branch.

Many thanks again!

[adamrauh]

Hi @antonpeez , thanks for using the package! I've seen this error pop up a few times, but I have never been able to replicate it on my end. Would you be able to try updating to the latest version of the package (maybe uninstalling + reinstalling) and if that doesn't work, tell me what version of R and OS you are using? Sorry, I know that's not terribly helpful.

[antonpeez]

Thank you, @adamrauh! I reinstalled back in June, which did not solve the problem, but it does seem to be resolved now with the newest round of updates, as I am fortunately no longer receiving the error message. Thanks again!