Players get 'no game found' if there is a batch (or batches) with different treatments

empiricaly / empirica

Open source project to tackle the problem of long development cycles required to produce software to conduct multi-participant and real-time human experiments online.

Apache License 2.0

43 stars 8 forks source link

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?

This can be recreated with the initial demo application (from empirica create test-app)

Create a batch with two games - one of each treatment. Or you can create two separate batches with one game each. The key here is that you're running two different treatments at the same time.

Join players into the games - you will see one of the treatments get 'overbooked' (yellow icons).

Complete the intro step for players - when the game starts, the overbooked players are dropped (no more games).

Clearing the users in the users page of the admin screen does fix this. But we don't want to have to do that every time.

Steps To Reproduce

empirica create test-app cd test-app empirica

Go to admin panel, create a batch with two games, each with a different treatment (one solo, one two player).

Create 3 player tabs and join the game (enter player identifier). You should see a player (or two) overbooked (yellow icon).

Complete the Intro step by clicking next on each user.

Players who were overbooked will not go to the other game, they will just be dropped with 'no more games'.

Empirica Version

Version: v1.10.0
SHA:     5467a49
Build:   208
Branch:  main
Time:    2024-03-31T02:18:22Z

Client:  1.10.0
Server:  1.10.0

What OS are you seeing the problem on?

macOS, Linux

What browser are you seeing the problem on?

Chrome

Relevant log output

No response

Anything else?

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I believe this is the expected behavior. Whether it's right behavior is up for debate. I am not an expert at statistics or experiment randomization. The 2 behavior at play are:

overbooking: I believe this is done to so that assignment is "more" random. If we were to distribute equally between games, we would potentially introduce some sort of bias. Again, I am not an expert here, and I am following recommendations, which I in turn might have gotten wrong. This issue has been brought up before and I think it would make sense to make this behavior configurable.
reassignment of overbooked players: when the game starts, the overbooked players should be reassigned automatically, but they will only be reassigned to game with the same treatment. We only reassign to games with the same treatment because we assume the intro steps are specific to the treatment. It would not make sense to reassign players to a treatment where the rules of the game have changed. For example, you might have a hasChat factor ; if you talk about the chat the chat in the intro steps, then switch the player to a game without chat, the player will be confused and could invalidate the experiment. This has also been a point of discussion. Not all experiment have significantly different intro steps depending on the treatment.

I am very much open to discussion on this. We should probably be able to introduce some configuration options to the default assignment logic. You can always override the default logic, but that is a non-trivial amount of effort, and common scenarios, such as the ones described in this ticket, might be better served by simply changing a setting. We can also discuss what the defaults should be.

CC @amaatouq @malsobay @JamesPHoughton

empiricaly / empirica