Cap # of participants and balance data collection among age ranges

[okaycj]

TL;DR

Cap # of participants and balance data collection among age ranges

Narrative

Right now, researchers have to track manually how many children participated in the study, how many children were assigned to a specific age range and how many participants were assigned to a specific list or condition. Ideally, the researcher would define this information in the study editor and Lookit would automatically cap the number of participants in each age range and make sure that the # of participants assignated to each list/condition is balanced among age ranges. For example, a researcher wants to test 24 4-5 month-olds and 24 6-7 month-olds in a study with 4 conditions. The researcher wants an equal number of participants of each age group assigned to each condition (i.e. 6 participants per condition in each age group). Right now, the researcher should check how many children have participated in each age range and remove the age range manually when it gets to 24 participants. Moreover, there is no way to make sure that from the 24 participants in each age group, 6 of them are assigned to condition 1, 6 to condition 2, and so on. For example, it may happen that condition 1 is only assigned to 4-5 month-olds. Thus, the researcher may need to remove manually the lists/conditions that are full and test additional participants to make sure that all conditions have enough number of participants. If the researcher was able to define this information in the study editor, Lookit could automatically track the participation, cap the number of participants and optimize the assignation of participants to balance data collection among age ranges.

To ensure that researchers do not test more participants than what they need and that the collected data is properly balanced. This feature is particularly critical for studies that test more than one age range in the same study.

Acceptance Criteria

• [ ] Researcher will need a UI to set number of responses for specific age range.
• [ ] Researcher might need a UI to have a response not be applied to the above age range response count. For example, the response is of mediocre quality, so they might want to allow more response for an age range.
• [ ] Researcher might want to adjust their cap on an age range after collecting some responses.
• [ ] Experiments would have to be unavailable to participants where their criteria has already received enough responses
• [ ] Tests for new feature
• [ ] QA on new feature

Note

This issue needs some work. There's a lot here and what is needed feels nebulous.

[kimberscott]

Closely related to https://github.com/lookit/lookit-api/issues/677

[bleonar5]

A concern related to this issue appeared in tech_support, and based on the reacts, it seems to have some support behind it. I think the nebulousness of this feature request could be fixed a bit by dividing it into stages. The first stage, which relates most closely to the tech_support inquiry I linked, would be to allow users to cap the number of responses they want to collect for a given study. The second stage would be to allow users to cap the number of responses for a given criteria expression, or a given stimuli condition.

With a little bit of effort, I think having the first stage available as an option would make it possible to manually accomplish the function that the second stage would provide (either by separating studies into sub-studies, or cycling through criteria/conditions in order, using response caps along the way).

If this ends up being something we want, I think I could handle it as a side-project. At my first job I developed a tool that handled this kind of dynamic list-balancing for mturk studies, so I have a good amount of experience and interest in the topic.