Lab 03 Campbell Scores CH 20-21

[lecy]

Question Posed:

I’m working through CPP 524 Lab-04 and using the Campbell Score to evaluate the two studies in Chapter 20 and 21. I read the extra notes and hints you provided in the assignment instructions.

I do understand that several of the scoring categories like Time-Frame and Regression to the Mean will be different, but what about sections like Selection / omitted Variable Bias, or Seasonality? Since the two studies are from the same dataset, I would imagine them to be the same. Am I correct in this assumption?

[lecy]

It's a very good question.

For a lot of the items there are really two parts of the question - (1) does the issue potentially exist, and (2) if so have the authors done something to address it?

In order to assign a +1 you could argue either point - the issue is unlike to exist (that is usually the case for regression to the mean unless program participation is triggered by poor performance like drug rehab or summer school). Or you can argue it likely exists but the authors have addressed it.

I am having a hard time saying there is any hard rule about this (each study and each type of research design has its own nuances), but just because two authors are drawing upon the same data does not mean the scores will be well-aligned.

One of the very interesting facets of this particular case study looking at 20 and 21 is that the two research teams made very different choices about how to analyze the data, which resulted in different conclusions. You could potentially introduce seasonality issues by selecting two data points as comparison points instead of using all of the data or selecting two data points from the same time of year or point in the program.

It's true that the treatment group in both studies will be shaped by the same selection-in and selection-out processes. But there are a lot of choices you can make about how to use the data to address the issues that will impact the Campbell Score. For example, I believe the first team uses all public school students as the comparison set, while the second team uses only those that applied for school choice (I need to double-check that detail). The point being, one would remove the potential bias from selection into the program, whereas comparing all of the treated in the dataset to all of the untreated would not address selection.

So in general, the two studies should all have the same answers for question (1) above - does the problem potentially exist. Where they might differ is (2), did the authors do an adequate job at addressing the issue or did they make decisions about how to utilize that data that made things worse?

[castower]

Hello all,

To clarify, in addition to examining the Campbell Scores, are we also suppose to state why we think the outcomes were different?

-Courtney

[lecy]

It's sufficient to complete the Campbell Scores for each. In doing so, it should become evident why the outcomes are different. You don't have to write anything additional, the hints were to help you make sense of why the studies had different conclusions.

[castower]

Thank you!

-Courtney

On Thu, Feb 20, 2020, 4:26 PM Jesse Lecy notifications@github.com wrote:

It's sufficient to complete the Campbell Scores for each. In doing so, it should become evident why the outcomes are different. You don't have to write anything additional, the hints were to help you make sense of why the studies had different conclusions.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/DS4PS/cpp-524-spr-2020/issues/8?email_source=notifications&email_token=AM6K2WTSU7N7D5QOQRPHBCTRD4NUNA5CNFSM4KX7IBEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMRCCXQ#issuecomment-589439326, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM6K2WXOLTBI7Q6BJ23FHLDRD4NUNANCNFSM4KX7IBEA .