Some participants not writing to all databases in DynamoDB

[anfrimov]

We've noticed in the experiment we're running with WebGL and AWS that participants who use Safari have their data saved to all the databases correctly except TrialResults, where no data appears for these participants. We have also noticed a handful of participants whose data is correctly saved to TrialResults and OtherSessionData but who have no entries in ParticipantDetails. As a result, for these latter, we can't really guess what might be happening as we can't even see what browser they were using.

Do you have any idea what might be causing this or what we could try to do in order to look into it?

[jackbrookes]

Hi! I have not experienced this, the thing I can think of is that you may have the experiment to end the session on quit. UXF tries to prevent them from closing the tab and save the data first. Its possible that this method may not work on some browsers.

Instead, maybe make sure your experiment session has been ended properly (e.g. after the final trial) rather than waiting for the tab to be closed.

[anfrimov]

Thank you for the quick response!

Yes, I believe we have both of those things set up. The session ends after the final trial or failed attention checks, but it is also set to end the session if participants try to close the tab.

This could explain some of the behavior, for sure, if UXF waits to write the trial results until the session ends. Is that the case when using AWS? I thought it wrote the data after each trial.

[jackbrookes]

Trial data is written only at the end of the session. There are some good reasons for doing this. It seems like some of your participants are quitting mid way through the study, and maybe Safari doesn't allow hi-jacking the closing of the tab

You can always save data trial by trial using custom data saving. i.e. after every trial you can run trial.SaveJSONSerializableObject and pass in a dictionary of values, then you should end up with data saved on each trial.

[SHEscher]

Hi Jack. I work with @anfrimov on the project. Could you expand on the reasons why it is better to write data only at the end of the session? I also ask, because we encountered more issues, were data was only partially written to a table on DynamoDB.

[jackbrookes]

The main reason is simplicity, in some experiments, we retroactively set older trials' results. So during trial 3, we may go and fill in a value in trial 2. Writing on each trial makes this more tricky.

Regarding the original issue, are you sure these are not mobile users? Mobile users can exit their browser and close the browser app at any time, and there is no way to interrupt that and save all data.

[SHEscher]

Thanks for your reply. We invite participants via Prolific and exclude mobile users, so I think this is not the issue. At the same we have record whether participants finished the study providing a completion code only at the end of the experiment.

We noticed actually two sides of the problem:

• We have one table from which participants pull indices for stimulus data and then lock a subset of these indices on DynamoDB such that other participants do not use them. The query from and writing to DynamoDB should be done in less than 5 sec. However, we noticed that updating the table is not done accurately in all cases. The problem occurs more often when many participants do the study around the same time, but when looking into the logs of participants with overlapping indices the time difference at the start of their queries is usually > 1 min.

• the other side is that among the common tables provided by UXF, we observed that some participants write in one table but not into other(s). The TrialResults, that @anfrimov mentioned above, are the most painful for us, obviously, but we had also participants, for instance, writing in UXFData.*.OtherSessionData but not in UXFData.*.ParticipantDetails, while their trials were recorded. But again, we also have quite some cases, were no trials were written while we received other information.

So far I cannot see a clear pattern, where these issues stem from. My guess would be that it has to do with Internet latency issues in combination that some packages are not transmitted accurately – I wonder whether this might has something to do with underlying network protocols (TCP vs UDP or so). However, I observed the issue with the TrialResults table on my own computer using Safari, and then everything worked with Chrome. But also this is not consistent - I also asked some participants w/o trial data about their browser and they all didn't use Safari.