In academic studies on experts and expertise one consistent finding is that experts are better at filtering information to quickly determine what is noise and can be ignored and where to focus attention.
This insight definitely applies to expertise in evaluation. You will find that seasoned evaluators will not read a report from front to back - they will flip to specific sections first to identify things like the estimator and sample before reading the rest of the report because knowing those details impacts how they process the rest of the information.
Interpreting a table of results is not a straight-forward task. Making sense of what numbers are included in a table and what they represent takes some practice. Getting comfortable with standard errors, t-values, and p-values so you can make sense of whichever the authors decided to use takes some practice. Looking for details on selection and attrition takes some practice. Learning how to identify potential competing hypothesis or internal validity problems takes practice.
I find that even rigorous studies might do a poor job of organizing key findings. For example, what are we supposed to make of Table 4 in CH5, the RCT on early childhood education?
I'm hoping at this point you feel a little more confident about reading a technical report and absorbing key details to make your own assessment about the quality of evidence presented in a study.
What do you find most challenging about reading a new study? Which information is hardest to make sense of?
In academic studies on experts and expertise one consistent finding is that experts are better at filtering information to quickly determine what is noise and can be ignored and where to focus attention.
This insight definitely applies to expertise in evaluation. You will find that seasoned evaluators will not read a report from front to back - they will flip to specific sections first to identify things like the estimator and sample before reading the rest of the report because knowing those details impacts how they process the rest of the information.
Interpreting a table of results is not a straight-forward task. Making sense of what numbers are included in a table and what they represent takes some practice. Getting comfortable with standard errors, t-values, and p-values so you can make sense of whichever the authors decided to use takes some practice. Looking for details on selection and attrition takes some practice. Learning how to identify potential competing hypothesis or internal validity problems takes practice.
I find that even rigorous studies might do a poor job of organizing key findings. For example, what are we supposed to make of Table 4 in CH5, the RCT on early childhood education?
I'm hoping at this point you feel a little more confident about reading a technical report and absorbing key details to make your own assessment about the quality of evidence presented in a study.
What do you find most challenging about reading a new study? Which information is hardest to make sense of?