Elaborate on the evolution of our teaching practices and consider including failures

[joelostblom]

Throughout the manuscript, the authors offer many useful examples and illustrations. However, I think it would further benefit the manuscript to be more explicit about the evolution of the authors’ teaching practices. As noted in the introduction (p. 3), the manuscript is informed by five years of experience teaching in the data science program. I’m assuming that the strategies and techniques identified in the manuscript evolved in some way over this period of time. If so, it would be helpful to more explicitly detail what had been attempted and, if it changed, what motivated the change. “Cases of failure” can be a great learning experience from a teaching perspective too (and I certainly have my own fair share of those cases!).

[ttimbers]

Wow, this is a big question, I am not sure we can fully answer in the space given... But here are some things we might want to mention if we can:

• We obtain student feedback on teaching practices from mid-course meetings and surveys from student-elected student representatives, course evaluations and alumni surveys. Feedback on teaching reproducibility has been obtained from all of these mediums.
• We learned that more practice mattered, and in capstone we observed students could do the things that we gave the lots of practice on (e.g., basic version control skills, e.g., working linearly on the main branch) but did not attempt or struggled with other reproducibility concepts/tasks they were taught, but where we provided less practice (e.g., more advanced version control topics like branching and pull-requests, intermediate/advanced features of reproducible reports and presentations).
• Guided instruction - so the lecture worksheets and/or autograding questions in the lab are newer. Block rep feedback in the earlier years indicated that labs were hard to get started on (i.e., there was a big jump from what was learned in lecture and what was expected on lab assessments). When we implemented lecture worksheets and/or autograding questions in the lab this complaint decreased.
• Lot's of practice applies to projects too - in early years of the program we didn't have any project courses outside of the project courses. The capstone projects from that time had a lot of technical debt and were not that reproducible. Thus, we implemented project courses for several reproducibility related topics (522, 524 and 525) and we observed (and received feedback from capstone partners) that the technical debt was lower, and that the projects were more reproducible.