ctreffe / alfred

Alfred - A library for rapid experiment development
MIT License
10 stars 1 forks source link

Possible abortions by ListRandomizer even though there are open slots #208

Closed jobrachem closed 2 years ago

jobrachem commented 2 years ago

Quoting @ctreffe (#205):

I think there is another problem with the process of determining the number of open slots: As described above, my randomization should assign 160 participants to two experimental conditions, 80 participants in each condition. I have set the _exp_sessiontimeout to 7200 seconds instead of 86400 seconds. Hence, I would expect the randomizer to assign slots to new sessions until all slots are finished or all slots have ongoing datasets with saving times not older than two hours. My dataset, however, shows that sessions are aborted well before these criteria are met. Here is an excerpt from my randomizer slot list for the experiment:

{
                "label": "sanitized",
                "session_groups": []
            },
            {
                "label": "natural",
                "session_groups": []
            },
            {
                "label": "sanitized",
                "session_groups": []
            },
            {
                "label": "sanitized",
                "session_groups": []
            },
            {
                "label": "natural",
                "session_groups": []
            },
            {
                "label": "natural",
                "session_groups": []
            },
            {
                "label": "natural",
                "session_groups": []
            },
            {
                "label": "natural",
                "session_groups": []
            },
            {
                "label": "natural",
                "session_groups": []
            },
            {
                "label": "natural",
                "session_groups": []
            },
            {
                "label": "natural",
                "session_groups": []
            },
            {
                "label": "natural",
                "session_groups": []
            },
            {
                "label": "natural",
                "session_groups": []
            },
            {
                "label": "natural",
                "session_groups": []
            }

I do not see how the randomizer could abort a session when there are slots in the randomization list to which no session was ever assinged (hence, no ongoing session for this slot). Nontheless, sessions were not assigned an experimental condition and aborted prematurely, according to the experimental dataset.

jobrachem commented 2 years ago

This turned out to be an artifact caused by apparently automatic requests to the experiment url (possibly some user agents) that never even actually visited the site, which would trigger a first complete save via JavaScript callback.