search

NYUCCL / psiTurk

An open platform for science on Amazon Mechanical Turk.
https://psiturk.org
MIT License
277 stars 140 forks source link

Suggestion: option to define your own randomization algorithm #295

Open markkho opened 6 years ago

markkho commented 6 years ago

I noticed that the get_random_condcount function doesn't take into account the mode of participants who have completed an experiment when assigning the next participant's condition. Was this intentional? I can see why you might want to be able test random assignment in debug or sandbox mode, but it can throw off condition assignment once the experiment is run live (and the database isn't cleared).

deargle commented 6 years ago

I have noticed that also. I don't think it's intentional, although with the balanced randomization, it all works itself out, doesn't it? A workaround is to increment the codeversion, which resets the random assignment count.

On Sun, Jan 21, 2018, 9:07 PM markkho notifications@github.com wrote:

I noticed that the get_random_condcount function doesn't take into account the mode of participants who have completed an experiment when assigning the next participant's condition. Was this intentional? I can see why you might want to be able test random assignment in debug or sandbox mode, but it can throw off condition assignment once the experiment is run live (and the database isn't cleared).

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/NYUCCL/psiTurk/issues/295, or mute the thread https://github.com/notifications/unsubscribe-auth/ABHsfa3gQkYjHTi9SgJ8jExC0XwPnPaXks5tNAmagaJpZM4RmJJ7 .

jacob-lee commented 6 years ago

Seems to me like it ought to distinguish mode. I guess I can look into adding it, if Dave agrees.

Actually, although the balanced randomization mechanism works pretty well, in some cases it does not do well (e.g. when workers submitted, but their data is not usable), or when the design is more complicated. It is hard to anticipate what users will require; I've spent quite a bit of time writing ugly work-arounds. What I would really like to see is a more general mechanism, in which the counterbalancing function can be over-riden with a user-defined one.

gureckis commented 6 years ago

yes, this is a long ago-recognized problem. we tried soliciting feedback about the best algorithm for this (long story short it apparently is simple random assignment ala PlanOut.

but yeah, ideally we could have a file akin to custom.py such as assignment.py that you can define your own randomization algorithm and have it travel along with your experiment code for replicability.

jacob-lee commented 6 years ago

PlanOut looks interesting; hadn't seen that project before.