gkiar
commented
5 years ago URL: https://link.springer.com/article/10.3758/s13428-017-0935-1 Figure 4 is something like what I should make for 1vox, characterizing the relative magnitude of sources of variability. In my case this will include: noise, session, participant (w/in dataset), participant (across dataset), and tools.
The Spearman disattenuation formula (True corr of A and B = observed corr / sqrt(reliability of A times reliability of B) ) could be used to reconcile differences between pipelines, and quantify the level of agreement. This could potentially be extended to not only apply to correlations of derivatives, but the agreement of claims and their relative significance(s). N.B.: the disattenuated correlation will always be higher than observed correlation, if measures are imperfectly reliable.
One point raised in the discussion is that paradigms with high robustness may in fact be very useful for group-level discrimination, even if (and partially because of) their inter individual reliability is low.
Additional reading
- "It is not intuitive, and rarely discussed, that such questions may be at odds with each other because one requires low and one requires high variability between individuals" Rogosa, 1988
URL: https://link.springer.com/article/10.3758/s13428-017-0935-1
This paper does...
In the context of experimental design, a reliable effect is one that nearly always replicates. In the context of correlational research, reliability means the ability to consistently "rank" individuals (i.e. TestReTest/ICC). Naturally, the more reliable the experiment (i.e. smaller between-participant variation), the less reliable the correlational analysis (i.e. less distance between participants and their peers).
(skipped task definitions)
In Results: task reliabilities: to confirm, they mean correlative/ICC reliability.