Incorrect number of entries in paired experiment

[anauta00]

Hi everyone,

I have designed a paired experiment in which 16 people have participated: https://quickeval.no/observer/260

According to the UI there are 16 completed experiments as well as a number of uncompleted ones. This experiment use a dataset of 10 images, which means there are 45 comparisons to be made for every experiment.

The total score matrix shows a cumulative score of 674, where there should be 16*45=720 entries. Looking at the exported data it is clear that some experiments marked finalized actually were not: Paired (anon) n10-results.xlsx

Another issue is that, somehow, the z-score for image '20.jpg' is no longer calculated/shown in the UI:

Could you please have a look at this?

Thanks and greetings, Auke Nauta

[mariuspedersen]

From what I can see in the data; observers 536 has done the experiment twice (one incomplete session and one complete session). So when the data had been exported both sessions are included. If you remove the incomplete session, then the number of comparison is correct for this observer. The other observer, 530, also has two different sessions. Looking at the results, if you add the two sessions, then this observer also has the correct amount of comparisons.

Looking at the data observer 529 has done the experiment twice as well. This brings the total to 15 observers, 15*45 is 675.

I have attached a correct CSV file. I would recommend that you calculate the z-scores/statistics based on the CSV if you can. Paired.anon.n10-results-fixed.xlsx

The z-score of image 20 is not calculated as there are 0 (zero) values in the matrix, leading to NaN values. And therefore it is not shown.

Hope this helps.

[mariuspedersen]

@disastersystem: not sure why observer 529 has gotten a new session ID. But maybe you can explain this?

[anauta00]

Thanks for the feedback. It does clarify some things.

Still I think something is wrong. The UI really should be correct in identifying finished or non-finished experiments. At the moment this is not the case.

We are now at 19 finished experiments, according to the UI, which should result in 19*45=855 comparison entries. The actual number of entries in the (finalized) export is 809. So still 46 entries too low. Even if something went wrong with a session, making the actual finished experiment 18 (iso 19), the numbers are not correct. I do not know what is causing this...

And as to image 20, it does have entries in its column:

And the UI did show the z-score initially in the graph. Here again I do not know why it does not show up anymore, although I guess it could be related to the incorrect number of comparison entries.

I would appreciate if you would a look at this.

And thanks for the wonderful program, it is a pleasure to use. Greetings, Auke

[disastersystem]

@mariuspedersen @anauta00 You're right. I've gone through the database and found that session 1213 only has 7 comparisons but is tagged as finished. And session 1205 has 37 comparisons but is tagged as finished as well.

The only way to get tagged as finished in the database is to reach the last step of the experiment. So the observers have reached the last step but after comparison 37 and 7 the answers stopped being saved. There could be a few reasons for this, but I could not tell you why at this time as I have not been able to reproduce something like this myself.

What I can do to fix the statistics is calculate whether the experiment was finished or not based on whether the number of comparisons for each session matches the experiments total (instead of marking the experiment as completed on the last step). Which would have been the best approach to begin with, but I chose to do it the lazy way.

[anauta00]

Hi Robin, Your suggested fix seems fine to me. Thanks and greetings, Auke