search

expfactory / expfactory-python

python module for managing experiment factory javascript experiment files, batteries to deploy them to (eg, psiturk), and virtual machines to host the compilation of those things.
http://expfactory.readthedocs.org/
MIT License
4 stars 11 forks source link

Experiments can't be run in non-random order #134

Closed earcanal closed 6 years ago

earcanal commented 7 years ago

To avoid order effects expfactory --run --experiments ... randomises experiment order. This behaviour is a little unclear (e.g. not mentioned in expfactory --help). There are also cases where specifying a specific order is useful, for example, where performance on one task should affect the outcome of a subsequent task.

Can I suggest the following additional expfactory option

--no-random don't randomise experiment order (default is to randomise which minimises order effects)

vsoch commented 6 years ago

hey @earcanal ! I have the early (alpha?) version of the experiments container ready to go! I've written up some documentation for you to get started:

https://expfactory.github.io/expfactory/

This is truly a reproducible experiments container, meaning you generate it once, and it's not going to run into a lot of these issues. I'd appreciate feedback on all things, and importantly, suggestions / requests for your use case. I'd like to get feedback from a handful of users, do a bit more work, and then port the rest of the library (experiment, games, surveys) over to the new version. Thanks for your help with this! You can close all your issues here and re-open them at the new repository.

vsoch commented 6 years ago

Issue moved to https://github.com/expfactory/expfactory/issues/35