Race condition in onStageEnd callback?

[dharakyu]

Is there an existing issue for this?

• [X] I have searched the existing issues

What happened?

I ran into an issue within the onStageEnd callback in which a field of a player object is undefined when it shouldn't be. I have logic that triggers the game end if players do not select by the end of the stage, which I check for by checking if player.round.get('decision') is undefined for any of the players in the game. I can see in my tajriba file that this decision variable was properly captured for the round for both players during this round, yet the game ending logic is triggered (i.e. if (player1Choice === undefined || player2Choice === undefined) returns true). This makes me think there is some kind of race condition where a player's attributes are momentarily unable in the callback. However I am having difficulties replicating this (only observed during a pilot study) so I'm not completely sure.

Steps To Reproduce

No response

Empirica Version

Version: v1.9.8
SHA:     1eb6b17
Build:   198
Branch:  main
Time:    2024-03-17T13:26:31Z

Client:  1.9.8
Server:  1.9.8

What OS are you seeing the problem on?

macOS

What browser are you seeing the problem on?

Chrome

Relevant log output

No response

Anything else?

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[npaton]

I think I understand your situation, but I am not 100%, I might need to see some code. Sorry If I'm going off track here.

In pseudo code, your situation looks something like this?

Client:

const onDecisionClick = (decision) => {
    player.round.set("decision", decision);
    player.stage.set("submit", true);
}

Server:

Empirica.onStageEnd(({ stage }) => {
    const playerFail = stage.currentGame.players.find(p => !p.round.get("decision"));
    if (playerFail) {
        // end game
        return;
    }
});

Normally, all sets should be processes sequentially. "decision" should be processed before "submit", so "decision" should be set in the stage end callback. There could be a bug here, and I will look into it.

There is also natural race condition here with the stage timeout. If the last user to set their decision field does right so as the stage ends (there is always network lag between the client and the server), the stage timeout might get triggered on the backend before their decision comes in. It should be detectable from the data in the tajriba file, if the callback happens before the decision is recorded. Though it might be difficult to read that in the raw data. I'd be glad to take a look.

If this is the case, one solution might be to "stagger" the stage timeout between what the player experiences and the real timeout, by a few seconds. By that I mean the stage might last 12 seconds, but the player only sees the stage lasting 10s, and, for the last 2 seconds, the player sees the inter-stage loading. This way, there is no (or little) chance of their input to arrive after the timeout. To do that, you'd need to tweak the timer UI, and have logic to hide the stage when the "fake" timer runs out. I don't know if I'm clear, sorry.

This could be a feature that other users might need as well, so I'd consider this a potential Empirica feature to add. Though it can be implemented outside Empirica, and might a good idea to try out before we add it to the platform. It could even be considered something that we should always do, though with network lag, it's hard to decide what is the right default. It might still be something for the experimenter to decide to use or not.

That's a lot of speculation. Ideally, I'd like to see the code and the data to see if this could indeed be the problem. If you can share that with me (you can DM me in Slack), that would be very helpful. 🙏

[dharakyu]

Hi Nicolas, thanks so much for following up on this. That's pretty much the logic I have in my code - I've sent you the relevant code snippet and data over Slack. I can see from the exported data that the decision value for each round was populated, so I think that means it was successfully written to the database. But the round values that should be populated in onStageEnded are not there, so that indicates that the decision round variable was not accessible in the callback.

I was thinking of something similar to what you proposed - the idea was to try to read the decision round variable several times and if it's still null after a few attempts then assume that the value is in fact null, and end the game. I will investigate how to make this work with the UI.

[dharakyu]

Ok, I was able to replicate the problem. I confirmed that both player.round.get('decision') and stage.round.get('decision') sometimes return undefined in onStageEnded even though they have been set in the stage that has just ended. I was able to find a hacky workaround by checking both player.round.get('decision') and stage.round.get('decision') and only ending the game if both were undefined (since the baseline rate of undefined is pretty low, so the odds that both of those were undefined and it wasn't an actual timeout are also low). But it does seem to be an issue, which is odd because I only noticed this problem recently (coinciding with my update to 1.9.8).

[npaton]

Thank you for the update. I have looked into it again and I can't figure it out yet. Since attribute changes are processes in order, if an attribute is set before submit on the client, it should be available in the callback. With the last release, I did push a fix for an issue where attribute updates could potentially get run out of order on the backend. Though I am not sure this is the problem, but it is worth a try. Let me know if you see any changes. I will keep looking into this.

[npaton]

I have still not been able to reproduce this. Are you still seeing this?

[dharakyu]

On a small scale I also can't replicate the problem. I've only noticed it when running actual experiments (around 20 participants simultaneously online). I haven't run any experiments since my last update so I haven't had a chance to look for it again.

[dharakyu]

Hi @npaton, I was wondering if you had any thoughts about this. I continue to see this problem where the variable stored in the player.round and/or player.stage object is sometimes undefined. Is there any reason to believe that storing the variable in the game object might yield better results? It's still difficult to replicate this problem in my experimental setup but I definitely see it even when running relatively small-scale experiments (with 15ish participants online)