Sensitivity analysis using smoothness restrictions for other periods in staggered treatment timing

[mu2023]

Hi, Thank you for writing the Stata package for honest did! I am trying to run the sensitivity analysis for other periods in a staggered treatment setting. I use csdid to get my estimates, and run honestdid. I can get he sensitivity results for the default event study (Tp0) time with no issues, and I can also do the sensitivity analysis for other periods using relative magnitudes. For example, using the example dataset from mixtape, I can get the sensitivity results using relative magnitude restriction by running honestdid, pre(3/6) post(8/12) mvec(0.5(0.5)2) coefplot The original CIs in the sensitivity table are the same for the relevant relative time produced from csdid.

However, when I try to run the analysis using smoothness restrictions, I can not replicate the analysis for other time periods. I tried playing with l_vec as suggested for the non-staggered case, but it seems like I am missing a point.

Could you please guide me on how to test the sensitivity of other periods using smoothness restrictions? Thank you very much.

[jonathandroth]

Thank you for your message! To help debug, could you please provide a minimum working example of the code you ran, the output you expected, and the output/error that you received. Thanks!

On Thu, Nov 10, 2022, 7:23 AM mu2023 @.***> wrote:

Hi, Thank you for writing the Stata package for honest did! I am trying to run the sensitivity analysis for other periods in a staggered treatment setting. I use csdid to get my estimates, and run honestdid. I can get he sensitivity results for the default event study (Tp0) time with no issues, and I can also do the sensitivity analysis for other periods using relative magnitudes. For example, using the example dataset from mixtape, I can get the sensitivity results using relative magnitude restriction by running honestdid, pre(3/6) post(8/12) mvec(0.5(0.5)2) coefplot The original CIs in the sensitivity table are the same for the relevant relative time produced from csdid.

However, when I try to run the analysis using smoothness restrictions, I can not replicate the analysis for other time periods. I tried playing with l_vec as suggested for the non-staggered case, but it seems like I am missing a point.

Could you please guide me on how to test the sensitivity of other periods using smoothness restrictions? Thank you very much.

— Reply to this email directly, view it on GitHub https://github.com/mcaceresb/stata-honestdid/issues/7, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE6EXFBBDH56H35ITNJR3ATWHTSM5ANCNFSM6AAAAAAR4P6XDQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[mu2023]

Hi Jonathan,

Thank you for your swift reply. I am using the same example dataset, and my code is the following:

local mixtape https://raw.githubusercontent.com/Mixtape-Sessions
use `mixtape'/Advanced-DID/main/Exercises/Data/ehec_data.dta, clear
qui sum year, meanonly
replace yexp2 = cond(mi(yexp2), r(max) + 1, yexp2)
qui csdid dins, time(year) ivar(stfips) gvar(yexp2) long2 notyet
csdid_estat event, window(-4 5) estore(csdid)
*csdid_plot, name(es, replace)
estimates restore csdid

// Sensitivity analysis - relative magnitude 
honestdid, pre(3/6) post(7/12) mvec(0.5(0.5)2) coefplot // default relative time (Tp0)

// Other post-periods (e.g. Tp1, Tp2), changing post(./12)
honestdid, pre(3/6) post(8/12) mvec(0.5(0.5)2) coefplot
honestdid, pre(3/6) post(9/12) mvec(0.5(0.5)2) coefplot

// Sensitivity analysis using smoothness restriction
honestdid, pre(3/6) post(7/12) mvec(0(0.01)0.05) delta(sd) omit coefplot // default period

// Other periods, e.g. Tp1:
matrix l_vec= (0\1\0\0\0\0) 
honestdid, pre(3/6) post(7/12) mvec(0(0.01)0.05) l_vec(l_vec) delta(sd) omit coefplot 

When I run the last section of the code, I receive the following error message:

I am sure I am missing some point for the smoothness restriction specification, but cannot see what it is at the moment.

Thank you.

[jonathandroth]

Hmm, that is odd. The code that you sent works well for me:

[image: image.png]

Can you please provide your operating system and Stata version? Mauricio will probably be better-equipped to help you with this than I will.

On Thu, Nov 10, 2022 at 9:15 AM mu2023 @.***> wrote:

Hi Jonathan,

Thank you for your swift reply. I am using the same example dataset, and my code is the following:

local mixtape https://raw.githubusercontent.com/Mixtape-Sessions use `mixtape'/Advanced-DID/main/Exercises/Data/ehec_data.dta, clear qui sum year, meanonly replace yexp2 = cond(mi(yexp2), r(max) + 1, yexp2) qui csdid dins, time(year) ivar(stfips) gvar(yexp2) long2 notyet csdid_estat event, window(-4 5) estore(csdid) *csdid_plot, name(es, replace) estimates restore csdid

// Sensitivity analysis - relative magnitude honestdid, pre(3/6) post(7/12) mvec(0.5(0.5)2) coefplot // default relative time (Tp0)

// Other post-periods (e.g. Tp1, Tp2), changing post(./12) honestdid, pre(3/6) post(8/12) mvec(0.5(0.5)2) coefplot honestdid, pre(3/6) post(9/12) mvec(0.5(0.5)2) coefplot

// Sensitivity analysis using smoothness restriction honestdid, pre(3/6) post(7/12) mvec(0(0.01)0.05) delta(sd) omit coefplot // default period

// Other periods, e.g. Tp1: matrix l_vec= (0\1\0\0\0\0) honestdid, pre(3/6) post(7/12) mvec(0(0.01)0.05) l_vec(l_vec) delta(sd) omit coefplot

When I run the last section of the code, I receive the following error message:

[image: image] https://user-images.githubusercontent.com/91473938/201114491-e2ae54e5-9e97-4352-8e88-f02816b41053.png

I am sure I am missing some point for the smoothness restriction specification, but cannot see what it is at the moment.

Thank you.

— Reply to this email directly, view it on GitHub https://github.com/mcaceresb/stata-honestdid/issues/7#issuecomment-1310341680, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE6EXFAP3VLJ2KXMZGHMDMLWHT7QFANCNFSM6AAAAAAR4P6XDQ . You are receiving this because you commented.Message ID: @.***>

[mu2023]

I am using Stata 16.1 in Windows.

When I re-ran the code I got a slightly different error message. I can see that it grabs the correct estimate to check the sensitivity for (from original CIs), but no results for Ms:

Thanks once again.

[mcaceresb]

@mu2023 Can you install from the issue branch and re-try?

net uninstall honestdid
local github https://raw.githubusercontent.com
net install honestdid, from(`github'/mcaceresb/stata-honestdid/issue7) replace
honestdid _plugin_check

Should tell you you're using version 1.1.2; then try again from a fresh Stata session.

---

@jonathandroth You were right that it was a time thing. I missed it when I greped the code quickly in between our meetings, but this is actually documented here. I suppose this is done for the sake of computational speed, but it seems to be messing up sometimes.

To be fair to them, it looks to be on average ~10% slower on my machine (without parallel I reckon this could be worse). I'm not too concerned with this, however, since the Stata version was already pretty fast.

In any case, I've pushed a fix to an issue branch (which is to just always adapt rho). If @mu2023 confirms the issue is fixed, can you check doc/readme.do and doc/examples.do run in your OSX machines? After that I can merge to main.

[mu2023]

I re-installed honestdid and now the code works without any warning messages. Thanks @mcaceresb and @jonathandroth.

[mcaceresb]

@jonathandroth Sounds like this did the trick; LMK when you've tested it on OSX and I'll merge this branch to main. For this issue, honestdid _plugin_check should give version 1.1.2.

[jonathandroth]

Mauricio -- I've run this on both Intel and M1 silicon without a problem, so I think we should be good to merge.

On Mon, Nov 14, 2022 at 9:42 AM Mauricio Caceres Bravo < @.***> wrote:

@jonathandroth https://github.com/jonathandroth Sounds like this did the trick; LMK when you've tested it on OSX and I'll merge this branch to main. For this issue, honestdid _plugin_check should give version 1.1.2.

— Reply to this email directly, view it on GitHub https://github.com/mcaceresb/stata-honestdid/issues/7#issuecomment-1313862516, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE6EXFGAQTFVICPGHDWANKLWIJFV3ANCNFSM6AAAAAAR4P6XDQ . You are receiving this because you were mentioned.Message ID: @.***>

[mcaceresb]

Fix merged in #8; see here for details.