Balanced condition assignment based on what other sessions have been completed

[theomorfoisse]

The problem is this: I have 48 different videos corresponding to different conditions, and I would like each participant to see one of these 48 videos, but not randomly. I would like to be able to administer them in a particular order - the first participant would see the first video, the n-th participant, the n-th video... etc. So the idea is to create a kind of global variable for all the experiments that allows the right video to be seen by the right participant. An idea is to use this global variable to indicate the value of whichFrames in a select-randomizer-test frame, if that's make sense.

In summary, the idea is to do assignment based on what sessions other kids have done, by storing additional (non-identifiable) data on the study model itself - e.g. something like “conditions for the last probably-usable response.”

Théo Morfoisse

Possible general approaches:

• Store an integer on the study model; each time someone participates, increment it. Allow access to this number from the API & use from the frameplayer. Provide randomizers that cycle through conditions based on the counter, and allow researchers to supply a custom function that determines condition based on the number.
• Provide an API endpoint that fetches, for a given study, summary information about the condition data in previous responses. Access this from the frameplayer. Provide randomizer options to balance conditions based on this information and allow providing custom function to do condition assignment based on response.

[kimberscott]

Just copying over some notes from Slack -

The current workaround: If you essentially want to end up with N usable data points evenly balanced across each of 48 conditions, you could start out assigning kids randomly and (as you code the data and count up how many usable data points you have) stop testing in the conditions that are done. This is obviously a bit clunky (and will require stopping/resubmitting/restarting as you change conditions) but is what people have done so far, and works fine.

There are a few things that make “just do the next video for the next kid” a little more complicated than it sounds to set up as an option, though not impossible:

• Not all ‘response’ objects correspond to a finished session, and not all finished sessions represent usable data. But the criteria researchers would want to use to say “ok run the next condition we’re set on this one” vary quite a bit.
• The experiment player cannot fetch data from Lookit from anyone but the current kid participating in the current study (otherwise our participants would be able to access each other’s data!) To do longitudinal condition assignment, it currently relies on fetching previous responses, and then can decide what to do next based on those data. But to do assignment based on what sessions other kids have done, we’d need to store additional (non-identifiable) data on the study model itself - e.g. something like “conditions for the last probably-usable response” or even just "number of probably-usable responses" (and the experiment runner can then rotate through based on that number). That is doable but will involve both changing the study model on the Lookit app and adding use of that study model to the frameplayer code. Then a randomizer in ember-lookit-frameplayer could make use of that study data to select the correct current condition.
• The randomizer will need to be able to react gracefully (or at least predictably) when the condition structure changes (e.g., the last condition run is one that isn't being run at all anymore).
• It would be better for absolute counterbalancing to keep an object storing conditions: counts, with ideal total counts defined, but likely more complicated given the variety of (possibly nested) counterbalancing structures.

Questions for Théo and any other folks interested -

• Do you care about actual order (run B first, then A, then C) vs. just getting the correct total number of kids per condition?
• When would you want to move on to running the next condition? A family completes a consent video? The family completes the entire study (all frames)? The family completes the exit-survey (having possibly deliberately opted out)? The family gets through frame X? One of the above AND the child is eligible per current criteria? A researcher manually marks the session as included?

[theomorfoisse]

1. I don't think the order is essential here. What is more important is that each child be associated with a specific condition.

2. There's a compromise here for sure. On the one hand we would like to eliminate a condition if we are sure that the data is analyzable - child meets the criteria and the family completes the entire study. On the other hand, as you said, the algorithm has to be reactive, if several children are starting the experiment in the same time. But I would opt for safety, and I would wait until the child has done all the experiment, and checks some criteria set in the survey for example.

[kimberscott]

Merging this with lookit/ember-lookit-frameplayer#47, Allow randomization with a goal of maintaining a given ratio of condition assignments

• User specifies what counts as complete data record (e.g., can put "countAsCompleted": true on any frame and count responses that reach that point - could use e.g. the post-save hook that processes each frame type (CONSENT, DEFAULT, etc.) to check for this and increment if appropriate - might want though to have study-level default behavior that counts everyone who gets through consent for instance
• User specifies whether to count unique participants only

[kimberscott]

Note this is a frequent question/request on Slack (probably 5-10 people now?) as well as important for MCS - prioritize among other usability issues!