Automatically throttle worker recruitment

[jbmartin]

Ideally, we could provide psiTurk with N, and it could automatically create HITs based on the user's bandwidth until N is reached.

[gureckis]

I like the idea of this combined with a tool to plan studies with a pre-determined N. Currently psiTurk is flexible but encourages an approach where you just keep "running more people" when you feel like it instead of planning an experiment in advance, specifying the N you need, and letting it run semi-autonomously. We might be able to build a tool that does something like this and could also automatically throttle for the sake of the tunnel server as well as the local server.

[jodeleeuw]

This is slightly off-topic for the issue, but related to what @gureckis said (feel free to move).

I find truly balancing conditions to be difficult on Turk, since you are typically collecting all of the data in parallel, and drop out rates are higher than laboratory studies. If I could specify an N per condition, and psiTurk could somehow guarantee that N, there might be experiments where I would find that useful and be willing to wait longer for the data (I'm assuming that it would require a longer wait since collection would need to slow down as the number of subjects approached N).

[gureckis]

yes, I agree. there has been a long discussion about this offline with the guys behind PlanOut and we have started work on a general purpose balancing algorithm that might help with this (https://github.com/NYUCCL/counterbalancing), contributions and idea welcome! eventually something like this would allow specifying complex designs in psiTurk and let it do this type of planning for you. we made it stand-alone because it seems modular and technical enough that psiTurk might just import this library.

for now though psiTurk does have a very simple system which doesn't guarantee a fixed N, but deals with the returned HIT issue in a simplistic way. See documentation on Task Parameters (http://psiturk.readthedocs.org/en/latest/config/task_parameters.html).

One point of terminology that I learned... people who decide not to do the task are not dropouts. A dropout is only someone who has been exposed to your actual manipulation in an experiment. In the same way, people who decide not to take part in your lab study after reading the ad in the hallway at your university are not dropouts. Thus, your technical dropout rate is something you have to compute based on your data files and how far people got in the task before deciding to quit. it is usually pretty low because once a Turk worker reads your instructions, etc... they are typically motivated to finish the task. psiTurk does provide a feature to record in the database when people being the actual "experiment" as opposed to finishing the instructions (see http://psiturk.readthedocs.org/en/latest/api.html#psiturk-finishinstructions).

[jhamrick]

In case you all are interested, I wrote a little script for my own use that will monitor a HIT and extend it with more assignments as the old assignments are completed. This is nice if you need a lot of participants, but your experiment is heavy enough that having them all work on it at once would overwhelm the server. Here's the code:

https://gist.github.com/jhamrick/95de85ac010723c2ee4a

It probably wouldn't be too hard to do something like extend psiTurk to launch another thread and monitor the HIT in the background in a similar manner.