✅ Demo dispatch, live randomization and game creation

[JamesPHoughton]

demo

HIT funnel

We can think about the process of collecting data from turk workers as a funnel, in which potential participants enter, but through a number of "off ramps" fail to translate to data.

Some of the off ramps are things we can't really control: some people will not want to do the HIT regardless of what else we do, once they know they will need to use a webcam.

Other off-ramps are things that we can address by improving how we use turk - scheduling HITs, notifying workers, etc.

There are a few facets of recruitment which have to do with multiplayer games specifically.

• One is that the most prolific turkers generally have "scripts" that go out and fetch HITs for them, which automatically enter their queue, and don't get seen until they finish other things. This means that if we set a HIT duration to be long enough for the whole game, these participants tend to miss the start by the time they get to the HIT. At the same time, they are "squatting" a HIT, so it isn't available to someone else. We can instead work with these scripts by paying after they complete intro steps (5 mins?) and setting HIT availability so that this must be complete before the group stage starts.

• Another challenge is that if we have a batch with a large number of conditions, it is possible that there will be multiple people sitting in games waiting for other people to arrive. This is compounded if there is a lot of dropout in the intro steps (as there is in webcam experiments) If we can randomize people to groups at "launch" then we don't care so much about dropouts that happen during intro steps.

[markwhiting]

There are so many competing effects here. I wonder which ones matter most and which ones are worth trying to attend to most directly.

I know people grab hits like this but I'm not sure it consumes a majority of spaces (especially if we suitably constrain participants), or if it is just a small portion.

Do you have a strong sense of how critical these things are?

[amaatouq]

A developed version of this (both the challenges and potential solutions) would make an amazing post for the empirica blog https://empirica.ly/blog

Also, I think we can easily get SAGE publishing to post it on their MethodSpace https://www.methodspace.com/ (let me know if you are interested, and I will make the intro).

[JamesPHoughton]

@amaatouq Sure, would love to write up more, probably after we test a bunch... =)

[JamesPHoughton]

@markwhiting my sense is that it is a minority of workers overall, but may be a majority of HITs that get completed, as it's biased towards the most active users. I've definitely had problems with HIT squatting in experiments.

[JamesPHoughton]

Notes from discussion:

• Can we make this another "default" recruitment strategy next to "simple", "complex", etc.
• If people are paid early, they may get confused and think they are done.
• Make a map/ timeline / funnel of how much you get paid based on how much you participate. Put in the first page. (range, this much to that much).
• Put numbers on the funnel? (Leakage diagram?)
• Missing from the funnel: overflow players?

[markwhiting]

Another strategy I've used is around adding in players after the initial recruitment. This is tricky and perhaps a bit too case specific. It's especially useful if theres something like a rolling experiment start for repeated experiments, but again, rather specific to that setting.

[JamesPHoughton]

Possible HIT sequencing: