presleyp
commented
9 years ago The plan is to send PsiTurk's counterbalance variable in like I send in condition, and use that as an index on the lexicographically sorted list of exchangeable blockIDs within any container with such a list.
The downside of this is that all lists of exchangeable blocks are treated the same. All must be the same length for there to not be errors or uneven sampling of versions of the experiment. And we can't do exchangeability classes, a feature I had thought about but hadn't implemented, which would look like this:
Experiment( blocks = [A, B, C, D], exchangeable = [[A, C], [B, D]] )
versions: A B C D, C B A D, A D C B, C D A B
Instead what's possible is:
Experiment (blocks = [A, B, C, D], exchangeable = [A, C])
versions: A B C D, C B A D
or Experiment(blocks = [A, B, C, D], exchangeable = [A, B, C, D])
versions: all 24 permutations
The reason for these limitations is that PsiTurk only provides one counterbalance variable (and reasonably so). So a workaround would be to add information to the experiment describing when to use random shuffling rather than the counterbalance variable. From the variable itself you can't tell what the config file says about its possible values - if it's set to 0, you don't know how many versions are possible. But we could add information to the experiment that says to ignore the variable for exchanging certain blocks. Perhaps "exchangeable" could mean randomly exchange and "counterbalance" could mean deterministically choose a permutation.
...That seems best. I'll add an issue to expand exchangeability to allow nested lists, and in the meantime add a variable to hold a list to be deterministically counterbalanced.
Other questions: should counterbalancing be done analogously to Latin Squaring? So that blocks get a treatment variable the way pages get a condition variable? I don't think so. It doesn't seem natural in the same way. I'm tempted to rename PsiTurk's variable inside Speriment again, like I renamed "condition" "version". I want to call "counterbalance" "permutation" and use "counterbalance" the way I use "exchangeable". It trades off one kind of confusion for another but I think it's worthwhile - the downside though is that the confusion does leak from developers out to users because the config file uses the PsiTurk terms. But users will just see it as the length of the the (unrepresented) list of conditions and the (represented) list of to-counterbalance blocks.
Use PsiTurk's counterbalance variable to inform block exchanging.