The issue of unbalanced panels

[Halvard123]

Hey,

I am slightly uncertain as to the effect of the necessarily unbalanced panels that arise due to the absorbing state of the treatment making the composition of the control group change within each cohort-effect estimation. Surely, this has the potential to cause selection in the extreme ends of the dynamic treatment estimation, and could be a point of criticism for the model when compared to a normal diff-in-diff. I understand that it is used in order to get the additional data from future treated units that have not yet been treated, but changing the composition in the control group between different periods must have some implications for the cohort-fixed effects and could easily be a point of criticism for the methodology. I guess I am wondering whether there is any literature on this issue of changes in composition of the control group and its effects or the assumptions under which it has a major or negligible influence.

The only way I could see to completely avoid this issue is to only use the never-treated as the control group, but that may not always be a viable option. Alternatively, is it potentially possible to restrict the amount of pre-treatment and post-treatment periods that we estimate (so that we don't estimate all the way up to T-7 and T+7 for example when that is not of major interest and may cause extra imbalance)?

Best, Halvard

[davidnov]

Hi @Halvard123,

I'm not sure I'm understanding the issue in this case, but does the response in issue #21 address your concern about cohort effects? Otherwise, seems you can always ignore event time estimates that you are not interested in as the package estimates a broad range of possible estimates.

[Halvard123]

Thank you for all the answers, and sorry both for taking so long to respond and for the unclear post.

I guess DiD-sample fixed effects, as you describe them in issue #21, may cause changes in composition between the samples to be less of a problem if these are included separately as well and not only as part of the interaction with cohort-fixed effects. I will however try to rephrase the issue into a question: Is it possible that any of the variation we are estimating in the CATTs can be attributed to changes in composition in the control group samples? Or is it rather the case that a cohort-fixed or unit fixed effect will alleviate most of these concerns?

I am partly asking because of the literature on systematic bias in pre-treatment estimation in event studies. Abraham and Sun talk about the convexity-issues of two-way fixed effects, and Borusyak and Jaravel (2017) raise several concerns about negative weighting of cohorts and about potential issues related to unbalanced panels as well as how fixed effects (and balanced panels) may not be a fruitful solution in their paper on revisiting event studies.

In essence, I am wondering how big of a problem the unbalanced panels may be in worst-case and average-case scenarios in terms of potential biases that may be caused by us trying to approximate averages of actually heterogeneous effects between treatment groups (due to the event times being more than a decade apart for example, differentiating the settings in ways that cannot necessarily be homogenized through cohort-fixed effects).

Best regards and thank you for taking the time to respond, Halvard

[davidnov]

I understand the issue, but it seems like this is more about whether the identifying assumptions are satisfied, and less about the workings of the package. By definition, if the identifying assumptions are satisfied (namely, parallel trends between any two calendar times in the untreated state across cohorts), then the only relevant source of heterogeneity across cohorts is the underlying treatment effect heterogeneity. I'm going to close this issue for now, would be happy to discuss your specific question further offline.