Issue getting bootstrapped standard errors with csdid

[timothykohler]

Hi Mauricio, Jonathan, and Ashesh,

I'm hoping you can provide some guidance regarding the combined use of Callaway and Sant-Anna's csdid Stata package and your honestdid Stata package. I'm using csdid to get event study estimates with bootstrap standard errors. Because I'm concerned about a pre-trend, I'm using honestdid thereafter for partial identification. However, it looks like honestdid is returning confidence intervals which are calculated using the asymptotic standard errors, not the bootstrapped ones. I've tried storing the latter after csdid and then running honestdid, but haven't been able to succeed. I've included a MWE below. Hoping you'd be able to help!

*****************
* load packages *
*****************

ssc install csdid, replace

local github https://raw.githubusercontent.com
net install honestdid, from(`github'/mcaceresb/stata-honestdid/main) replace
honestdid _plugin_check

************************
* generate sample data *
************************

clear

local units = 10
local start = 1
local end   = 10

local time = `end' - `start' + 1
local obsv = `units' * `time'
set obs `obsv'

egen id    = seq(), b(`time')  
egen t     = seq(), f(`start') t(`end')     

sort  id t
xtset id t

set seed 20211222

gen Y           = 0     // outcome variable 
gen D           = 0     // intervention variable
gen cohort      = .     // treatment cohort
gen effect      = .     // treatment effect size
gen first_treat = .     // when the treatment happens for each cohort
gen rel_time    = .     // time - first_treat

levelsof id, local(lvls)
foreach x of local lvls {
    local chrt = runiformint(0,5)   
    replace cohort = `chrt' if id==`x'
}

levelsof cohort , local(lvls) 
foreach x of local lvls {

    local eff = runiformint(2,10)
        replace effect = `eff' if cohort==`x'

    local timing = runiformint(`start',`end' + 20)  // 
    replace first_treat = `timing' if cohort==`x'
    replace first_treat = . if first_treat > `end'
        replace D = 1 if cohort==`x' & t>= `timing' 
}

replace rel_time = t - first_treat
replace Y = id + t + cond(D==1, effect * rel_time, 0) + rnormal()

// generate leads and lags (used in some commands)

summ rel_time
local relmin = abs(r(min))
local relmax = abs(r(max))

    // leads
    cap drop F_*
    forval x = 2/`relmin' {  // drop the first lead
        gen F_`x' = rel_time == -`x'
    }

    //lags
    cap drop L_*
    forval x = 0/`relmax' {
        gen L_`x' = rel_time ==  `x'
    }

// generate the control_cohort variables  (used in some commands)
gen never_treat = first_treat==.

sum first_treat
gen last_cohort = first_treat==r(max) // dummy for the latest- or never-treated cohort

// generate the gvar variabls (used in some commands)
gen gvar = first_treat
recode gvar (. = 0)

*************
*   csdid   *
*************

// estimate ATT(g,t)'s
csdid Y, i(id) t(t) gvar(gvar) long2 notyet wboot(reps(1000)) cluster(id) rseed(12345) saverif(rif_example) post replace

// estimate event study aggregation
use rif_example.dta, clear
csdid_stats event, wboot(reps(1000)) rseed(12345) post estore(csdid) // returns bootstrapped standard errors
estimates restore csdid // store results

*****************
*   honestdid   *
*****************

honestdid, pre(3/9) post(10/11) mvec(0.5(0.5)2) // clear that t=0 confidence interval is not the "original" using bootstrapped standard errors...

* END *

[mcaceresb]

@timothykohler Ok, but, in our defense csdid help file says "Even if WBootstrap SE are requested, asymptotic SE are stored in e()." We're not prescient enough to have divined that (:

But, in seriousness, I can't seem to find where csdid does store the bootstrap vcov. If you can figure that out then you can pass that on to honstdid.

[timothykohler]

Thanks for your prompt response, Mauricio! I also haven't been able to store csdid's bootstrap vcov matrix. I'll contact the authors and reply here once I receive their response :)

[jonathandroth]

Tim, thanks for raising the issue! Let us know if you find anything from the csdid creators.

Mauricio, thanks for the quick response!

Best, Jon

On Mon, Jun 24, 2024 at 8:38 AM Timothy Köhler @.***> wrote:

Thanks for your prompt response, Mauricio! I also haven't been able to store csdid's bootstrap vcov matrix. I'll contact the authors and reply here once I receive their response :)

— Reply to this email directly, view it on GitHub https://github.com/mcaceresb/stata-honestdid/issues/23#issuecomment-2186483343, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE6EXFGIWNMCDARHLRR7LPTZJAHNLAVCNFSM6AAAAABJZTB3PSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBWGQ4DGMZUGM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[timothykohler]

Hi all. Regarding the above, Fernando Rios-Avila and Pedro Sant'Anna informed me that csdid does not compute the bootstrap vcov matrix, because doing so would require one to compute the vcov for each multiplier bootstrap draw, i.e. like taking the vcov of the bootstrapped influence function. Hence, one has to rely on the asymptotic standard errors for honestdid, because you need the whole covariance structure.