macartan
commented
5 years ago Thanks Tara.
@jaspercooper I did some edits in branch iv_small_edits that adds in ITT and changes default; love to get your thoughts
@all I think the ate should definitely be kept alongside late; but I see that concern that late_het could be confusing for non pedagogical usages
@all we do have designers for designs such as block-cluster-factorial which might be too much for some pedagogical purposes but useful for learning day usages
Mentioned this to Macartan today. Some of this was written in June, so apologies if you have updated the templates mentioned here in the meantime.
In June, I taught Learning Days again and have a couple thoughts on the DesignLibrary package. I haven't used DD in a couple years, largely because I find it more efficient to code and troubleshoot Monte Carlos outside DD (some of the component packages are very helpful generally for this though). Since the Chile LDs in 2016, we have been trying to help people simulate designs to (a) visualize the dataset; and (b) calculate power. This means coding ~15 designs in a day on the Thursday of Learning Days while interacting with students. For the last 3 LDs, I have not used DD, given time constraints and that it is easier to adapt my own starter code for Monte Carlos to participants' interests/questions.
Some of the other instructors in Benin were more committed to DD than I have been so I tried to use some DesignLibrary designs there. As a resource, I'm confused about the intent. Is the point to illustrate somewhat more subtle properties of designs to advanced "research designers" or to be a resource for people that are not ready to code an entire design (or people artificially racing to code Monte Carlos as the case may be)?
Some of the templates are quite nice. I liked the factorial design template and found it useful. (However, this "intuitiveness" may be only because I designed the first version of the template back in the day.) On the whole, however, my reactions were: (1) lack of coverage of commonly used design elements; and (2) somewhat obscure default properties of some templates. On #1, there are not enough clustered design capabilities for anything except a 2-arm design, as far as I could tell.
Wrt #2, some of the designs seem oddly specific and likely to give confusing output unless users study the defaults really carefully. For example, I tried to use binary_ivdesigner for an encouragement design. In some sense, it would be great to diagnose estimators of the ITT and LATE (among compliers) as estimands and possibly the first-stage ATE for teaching purposes. The template does not include the ITT as an estimand, and seems to emphasize the (endogenous) ATE and LATE{het} instead. Further, there are some other odd things that took me time with the back end of the function to figure out. For example, why would we want heterogeneous intercepts by principal stratum as a default? (The default is an intercept of 1 amongst always-takers and 0 for all other groups.)
What is the intended audience for DesignLibrary at this point? At least for Learning Days, as a teaching tool, it would be great to emphasize the most standard estimands/estimators for a given design and include only completely "vanilla" defaults as to eliminate confusion. Thanks!