Choosing an Appropriate Comparison Group

[ellihammons21]

Hello,

I am working on Step 4 for the research design project and am having trouble with thinking through how to constuct an appropriate comparison group.

I have decided to use the Reflexive (Pre-Post) estimator for the evaluation of a vocatiol support program for adults with disabilities. The main indicator that I will be using to assess program impact is the number of new jobs obtained by consumers after one year in the program. I believe this estimator is appropriate because we would not expect to see a change in new job growth seperate from the program.

So, if I am understanding the reflexive estimator correctly, the counterfactual would be T1 (number of jobs consumers obtained in the 12 months before starting the program). But I'm having troubls figuring out what would be an appropriate comparison group to capture job gains seperate from the treatment. For a little background, the program is very small. We serve about 25 clients in the vocational support service. It would not be appropriate to compare these consumers (who are interested in job seeking) to those who are not enrolled in the VSS (and therefore are either ineligble or have not expressed interest in job seeking). I was thinking that an option might be to take a similar population from a county that does not provide vocational support, or perhaps have varying levels of treatment.

Does the reflexive eatimator seem appropriate for this research design? And if so, will either of the comparison groups that I have suggested be a strong comparison? Any feedback is much appreciated.

Thanks!

Bethany

[castower]

Hello Bethany,

Sorry I overlooked this post!

I think your suggestion to use a similar population from a county that does not provide vocational support would serve as a good control group (although recall for this paper you don't have to find actual data...just a hypothetical example).

@lecy any additional insights here?

Thanks! Courtney

[lecy]

Bethan - I somehow missed this post as well. It looks like it went to my spam folder, which is odd.

The challenge with your outcome is there will be no pre-treatment measure. If people are in an employment program it is because they need a job. If they have a job they will not join the program, I am assuming?

That fact basically eliminates the reflexive estimate since you won't know T1 in the (T2-T1) estimate. Or we can assume T1 is always zero, i.e. none are employed.

The caveat is that it depends on how you measure the group. You could use all of the people that file for unemployment in Jan of 2010, then T1 is what proportion are employed by December 2010. The next could be the same county, all people that file for unemployment in Jan of 2011 and also start the job training, then T2 is the proportion of that group that are employed by December 2011. Here the unit of analysis is the county, so the group mean for T1 is rate of re-employment before the program, and T2 is rate of re-employment with the treatment.

Not sure this is a great design since economic trends could impact rates in each year, so you would likely want a reference group of other counties to construct the (C2-C1) employment trends. Then you would be back to a diff-in-diff estimate.

For the post-test only you need to establish that your comparison group is identical to the treatment group, which means you are likely doing randomization into the training program similar to how the mental health program chapter created groups - select every other person on the unemployment list in the year of the treatment.

Unfortunately, for this assignment you can't use randomization unless it is a process independent of the study like a lottery system. Using an RCT simplifies things too much for the assignment. You would need another way to ensure your treatment and control groups are identical. For example, if you can only accommodate 50 people at a time in a class and 100 sign up, stagger the treatment so one group gets it 6 months earlier than the others. You could then observe how many people found employment on their own without the training as C2 (control group after 6 months) compared to how many succeeded with training (T2 - treatment group 6 months after starting the program). You can use randomization in this sort of staged roll-out.

You need to be careful, because if someone gets to start training 6 months later that might land a higher-paying job maybe they just wait and don't actively search, in which case are they a true counterfactual (how successful would someone be at finding a job without the job training, all else equal)? Also, if the training takes 2 months then your treatment group would only have four months of job search and your control group would have 6 months, so they are searching for 50% longer. These are the details you would want to manage.

I know I'm fudging some of your study details here, but hopefully these examples make sense.

[ellihammons21]

Thank you for the detailed response!

I see that there are some issues with using the reflexive estimator in the context of my study, as we would be limited in how we can measure rates of employment prior to T1.

My new plan is to use the Post-Test only estimator, combined with a deferred enrollment study model to create a comparison group.

Basically, based on staffing and program resources, the program can only effectively serve 10-15 clients per 12 month session, though total enrollment is usually somewhere around 25. Assuming that the groups are equivalent prior to treatment, T2 - C2 will capture any maturation/secular trends that might result in consumers securing employment seperate from the treatment. How does this sound?

Also, given the small sample size, would it be better to randomize particpation through some sort of lottery, or to perform some type of group matching to ensure that the groups are more or less equivalent at the start of the program? For example, we might match consumers based on age, employment experience, pre-placement assessment measures, etc.

[ellihammons21]

@lecy @castower Forgot to tag each of you in my above response, but wanted to make sure that you saw it. Thanks!

[lecy]

I would say for the sake of the exercise don't let yourself get entirely constrained by the reality of your program.

You might google some of the research on job training programs, and you will see that even when it works the effects are pretty low. So I would anticipate that you would need a decent sample.

Your framework could be a proposal to build an evidentiary basis for your program - how should you collect data over time to determine effectiveness? After 4-5 years you might have a decent dataset. Main issue would be whether it would have all of the info you need to analyze it without having to go back and collect anything else since your clients will be long gone.

Assuming that the groups are equivalent prior to treatment, T2 - C2 will capture any maturation/secular trends that might result in consumers securing employment separate from the treatment.

It's not a bad approach if you naturally have limits to the size of the training program. Your counterfactual will be comparing people that just finished 12 months of training with people that signed up to start the new course and just got waitlisted for the next 12 months? If so, I would want to know more about why you think they would be equivalent. Does motivation to search for jobs increase once you have new skills? Does knowing you are on a waitlist make you less likely to search for jobs because you want to wait for better jobs? If you end up on a wait list do you look for other training programs (contamination)? Do you expect someone that was laid off for 12 months and just starting the training to be the same in terms of motivation and need for money as someone that just got laid off? If they only complete half the program then find a job are they still in the treatment group?

Selection is definitely going to be the main issue you need to contend with, and you will need to think through your counterfactual construction carefully. You job is to argue that the process by which yo will construct the groups is likely to generate equivalent groups if you want to use the post-test only estimator since you won't have data to simply test that assumption.

With such a small group size matching would actually be better than randomization in this case. But what are the ethics? Is everyone equally entitled to the available spots in the program? What about someone that signed up 10 months ago vs someone that just signed up last week? Are there implications to denying certain people the treatment, and does that create constraints for assignment?

Are there any discontinuities you can leverage? What are the criteria for participation? Can you find people that just miss the criteria?

What about a placebo group, like a resume writing workshop? Would the same kind of people sign up for that as would for your job training? If so, could they serve as a reference point somehow? Or a different kind of job training?

The lack of a pre-treatment measure creates some headaches!