Recover triple-difference and subgroup effects from same model

Thanks for the comment in #12 on how to do triple differences. Following on from this I wanted to check if there was a good way of using this setup to return both the interaction coefficient and the two effect estimates for subgroups?

Constructing a population where:

Population observed over 10 time periods
Treatment of group ‘b’ at t = 6
Treatment base effect of +2
Group ‘b’ has a baseline of +1 over ‘a’
Female pop has a baseline of +4 over male
Female treated has additional effect of +8

library(tidyverse)
library(did2s)

set.seed(101)

df <- tibble(
  group = c(rep("a", 20), rep("b", 20)),
  gender = rep(c(rep("m", 10), rep("f", 10)), 2) |>
    factor(levels = c("m", "f")),
  t = rep(1:10, times = 4),
  t_f = factor(t),
  period = (t > 5),
  treat = period * (group == "b"),
  treat_f = treat * (gender == "f"),
  treat_m = treat * (gender == "m"),
  y = 1 * (group == "b") +
    2 * treat + 
    4 * (gender == "f") +
    8 * treat_f +
    rnorm(40, 0, 0.5)
)

df |> 
  ggplot(aes(t, y, colour = group, shape = period, group = group)) +
  geom_point() + 
  geom_line() +
  facet_wrap(~gender)

To check with a linear model (as one treatment time point), this recovers:

Baseline b +1 groupb
Female baseline +4 genderf
Treatment base effect (on males) +2 groupb:periodTRUE
Female additional treatment effect +8 groupb:periodTRUE:genderf

lm(y ~ group*period*gender, data = df)
#> 
#> Call:
#> lm(formula = y ~ group * period * gender, data = df)
#> 
#> Coefficients:
#>               (Intercept)                     groupb  
#>                  0.007661                   0.948069  
#>                periodTRUE                    genderf  
#>                  0.229718                   3.937925  
#>         groupb:periodTRUE             groupb:genderf  
#>                  1.804825                   0.236216  
#>        periodTRUE:genderf  groupb:periodTRUE:genderf  
#>                 -0.560564                   8.310232

In models with did2s, these also return the right numbers. Non-subgrouped effect of +6 (male treatment = 2, female treatment = 2 + 8)

did2s(
  df,
  "y",
  first_stage = ~0 | group,
  second_stage = ~i(treat, ref = 0),
  treatment = "treat",
  cluster_var = "group",
  bootstrap = FALSE
)
#> Running Two-stage Difference-in-Differences
#>  - first stage formula `~ 0 | group`
#>  - second stage formula `~ i(treat, ref = 0)`
#>  - The indicator variable that denotes when treatment is on is `treat`
#>  - Standard errors will be clustered by `group`
#> OLS estimation, Dep. Var.: y
#> Observations: 40
#> Standard-errors: Custom 
#>          Estimate Std. Error      t value  Pr(>|t|)    
#> treat::1  5.90938   4.88e-16 1.209703e+16 < 2.2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> RMSE: 3.43963   Adj. R2: 0.339757

To do triple-differences, building from your suggested model (although time/group fixed effects not working here - have substituted equivalent factors) recovers the interaction effect (difference from male effect seen in females):

Treatment of female group +8

did2s(
  df,
  "y",
  first_stage = ~ gender*t_f + gender*group + group*t_f,
  # first_stage = ~0 | gender^t + gender^group + group^t,
  second_stage = ~i(treat_f, ref = 0),
  treatment = "treat_f",
  cluster_var = "group",
  bootstrap = FALSE
)
#> Running Two-stage Difference-in-Differences
#>  - first stage formula `~ gender * t_f + gender * group + group * t_f`
#>  - second stage formula `~ i(treat_f, ref = 0)`
#>  - The indicator variable that denotes when treatment is on is `treat_f`
#>  - Standard errors will be clustered by `group`
#> OLS estimation, Dep. Var.: y
#> Observations: 40
#> Standard-errors: Custom 
#>            Estimate Std. Error      t value  Pr(>|t|)    
#> treat_f::1  8.31023   3.55e-16 2.339122e+16 < 2.2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> RMSE: 0.164607   Adj. R2: 0.996334

Fitting a separate model with two interacting variables in i() can recover the treatment effect of each subgroup - i.e. their difference from respective baselines:

Male treatment effect of +2
Female treatment effect of +10 (8 + 2)

did2s(
  df,
  "y",
  first_stage = ~ gender | t + group,
  second_stage = ~i(treat, gender, ref = 0),
  treatment = "treat",
  cluster_var = "group"
)
#> Running Two-stage Difference-in-Differences
#>  - first stage formula `~ gender | t + group`
#>  - second stage formula `~ i(treat, gender, ref = 0)`
#>  - The indicator variable that denotes when treatment is on is `treat`
#>  - Standard errors will be clustered by `group`
#> OLS estimation, Dep. Var.: y
#> Observations: 40
#> Standard-errors: Custom 
#>                    Estimate Std. Error      t value  Pr(>|t|)    
#> treat::1:gender::m  1.91294   3.90e-16 4.907801e+15 < 2.2e-16 ***
#> treat::1:gender::f 10.00694   9.79e-16 1.022635e+16 < 2.2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> RMSE: 0.420649   Adj. R2: 0.982962

My questions relating to the above are twofold. Firstly, is the final model a valid way of getting subgroup estimates, rather than splitting the dataframe? Secondly, can both of these estimates - interaction term and estimates of subgroups from respective baselines - be recovered from the same model? In theory the coefficients in the two-group model can be subtracted, but standard errors I imagine would be a lot more complex.

In my dataset which is a lot more complex than this example, I want to perform subgroup analyses to estimate both an interaction term for additional effects on vulnerable groups and a set of subgroup differences. The changes to the first-stage formula between these models is making the estimates differ slightly from a straightforward subtraction sum, so if recovering both from the same model was possible it would give more continuity across analyses.

Thanks in advance for any pointers on this.

Hi Andrew,

I might be missing something, but I don't think you're doing triple-diff from your example code. In the classic triple-difference design, you would have either the male or the female group be treated in a treated region, not both. One gender's region-trend is used as a stand in for the other. So I would not call this a triple-diff. Things are working out in your DGP since you are not generating differential trends by gender or by group. See example code here https://github.com/Mixtape-Sessions/Madrid-2024/blob/main/Labs/DDD/ddd2.R and https://academic.oup.com/ectj/article-pdf/25/3/531/45842047/utac010.pdf

I believe what you're doing here is not triple diff but just doing diff-in-diff using treated groups compare to untreated groups. Additionally, you're interacting treatment with gender to get subgroup ATT estimates which is an okay thing to do. Note you're implicitly assuming no gender-specific trends in the first-stage. Doing first_stage = ~ 0 | gender^t + group would estimate gender-specific trends which is probably preferable.

You can compare two subgroup-ATTs with proper standard errors using marginaleffects::hypotheses. You could also split the df and use vcovSUR package to perform hypothesis tests using separate diff-in-diff.

library(tidyverse)
library(did2s)

set.seed(101)

df <- tibble(
  group = c(rep("a", 20), rep("b", 20)),
  gender = rep(c(rep("m", 10), rep("f", 10)), 2) |>
    factor(levels = c("m", "f")),
  t = rep(1:10, times = 4),
  t_f = factor(t),
  period = (t > 5),
  treat = period * (group == "b"),
  treat_f = treat * (gender == "f"),
  treat_m = treat * (gender == "m"),
  y = 1 * (group == "b") +
    2 * treat + 
    4 * (gender == "f") +
    8 * treat_f +
    rnorm(40, 0, 0.5)
)

est = did2s(
  df,
  "y",
  first_stage = ~ gender | t + group,
  second_stage = ~i(treat, gender, ref = 0),
  treatment = "treat",
  cluster_var = "group"
)

library(marginaleffects)
hypotheses(est, "`treat::1:gender::m` - `treat::1:gender::f` = 0")

kylebutts / did2s

Recover triple-difference and subgroup effects from same model #32