JOD Normalisation in example_running.m

[nicolasch96]

Hello,

I am sorry to keep spamming questions on this repo, but I am now using your algorithm for some experiments (it works great, thank you very much) and I needed to revisit some parts of the code to answer some questions.

My concern is about the JOD normalization that we do when inferring the final scores. I can't understand why we need to multiply by 1.4826? Do we suppose that our data follows a normal distribution N(0, 1.4826) that we usually use for JOD definition? I couldn't find any support for that. Also, with this normalization, the std of the data becomes equal to 1.4826, is there any mathematical background for such normalization?

Thank you very much :)

Nicolas

[mikhailiuk]

Hi Nicolas,

Thanks a lot for all your questions and suggestions on how to improve the code! Both are always very welcome! :)

So to bring some clarification on the use of JOD units. These stand for Just Objectionable Differences (JODs) and are meant to make the differences in scores interpretable. By re-scaling the scores to JOD units we ensure that a distance of 1 unit between the scores corresponds to 75% chance of one condition being better than another (i.e. normcdf(1,0,1.4826)). This normalisation is not necessary for the work of the algorithm, however, re-normalizing to JODs the scores provided by ASAP can aid in interpreting the final experimental results.

For more details you can refer to this paper: https://www.cl.cam.ac.uk/~rkm38/pdfs/perezortiz2017pwcmp.pdf, specifically page 8, Figure 6 and Section 5.3.

Let me know if there is anything else I can help you with :)

Thanks,

Aliaksei