lecy
commented
2 years ago This first assignment is a chance to do a deep dive into a rigorous real-world evaluation. It is meant as a warm-up exercise of sorts that is designed to give you some sense of all of the moving parts that an evaluator needs to keep track of in a study, as well as how challenging it can be to make sense of all of the details of a study from the report.
Try to answer the questions as best you can. I don't expect all of the terminology to make perfect sense at this point. We will revisit all of these concepts throughout the semester, and we will revisit the program impact estimates from this study in Lab 5.
A rigorous impact evaluation will treat the program as a black box. The evaluator is not concerned about the program model or details of implementation. The impact framework typically approaches the world is more simplistic terms - did you receive the treatment or not? If you did, how much impact did it have?
The theory of change breaks open the black box. It is a succinct and parsimonious explanation of the program model, represented as a causal chain of events: the program does A, and that causes B, and that results in more of C.
You don't have to create a diagram to answer the question, but try to describe the key elements of the program and why you think they would impact the outcome. There are always lots of plausible explanations - for example for the claim that coffee decreases depression you could posit:
(1) Coffee stimulates the part of the brain responsible for executive functioning, which makes people more alert and productive, which means they are more likely to finish work on time, which reduces stress and anxiety about work, which results in better mental health.
(2) Coffee has antioxidants and improves blood flow and digestion, as well as stimulates dopamine in the brain, which results in better health and more energy, which then has a positive impact on a persons mental health.
One is a psychological pathway (locus of control) and one is a physical pathway (impact on circulation and dopamine). Either could be the primary reason that coffee impacts depression (both of these are made up for this example).
It's called a theory because its your understanding of the causal pathways that matter in a program. The skill is leaning how to describe essential elements of a program. For example, for a tutoring program don't simply state, "kids meet after school and get tutored." That is just restating that it's a tutoring program. How does the program enact change? Specialized curriculum designed for English Language Learners? Compensation for the types of support other kids are getting at home in order to keep up with the class? Enhancement of self-confidence using positive reinforcement in order to build identity as a reader? What's the secret sauce, in your view?
You will get more practice part 3 of your evaluation design project.
Question 4 states : "Explain the theory of change of this study".
I have never heard of a theory of change before. From what I gather from a quick Google of the term, a theory of change seems to be a methodology for planning, participation, and evaluation. It defines the long-term goals, hypotheses, necessary preconditions, and indicators used for assessment. As such I have answered the question according to that interpretation.
Dr. Lecy, I was wondering if you could please confirm this interpretation and/or clarify what you are seeking from this question?