GoogleCodeExporter
commented
9 years ago First, we have a paper that was accepted for publication that describes the
math and statistics that are the foundation of this software. The introduction
addresses this issue. Please check it out as it may help you understand the
software better and further discussion below.
Chen, W.; Wunderlich, A.; Petrick, N. A. & Gallas, B. D. (Accepted 2014), 'A
general framework for MRMC reader studies with binary assessments: simulation
for validation and sizing.' J Med Img.
The iMRMC-binary software expects as input binary outcomes for each reader-case
evaluation. For example, "zero" may mean that the reader did not agree with the
reference and "one" may mean that the reader did agree with the reference. This
agreement could come from a qualitative comparison of a reader's free-text
report and the reference report (truth).
If the data being compared are quantitative (e.g. size), agreement with the
reference could be defined by some error tolerance. In other words, a "zero" is
given to an observation if the size reported by the study reader is outside the
tolerance region about the reference reader measurement, and a "one" is given
if the size is within the tolerance region. Alternatively, a threshold can be
introduced and a 2x2 table can be created for this kind of data.
Study reader
size < threshold Study reader size ≥ threshold
Reference
size < threshold A B
Reference
size ≥ threshold C D
Given a table like the one above, you could get a binary outcome that reflects
• Total agreement: Every case in squares A or D gets a “one”. Every case
in squares B or C gets a “zero”.
• Agreement given reference size < threshold: Every case in square A gets a
“one” and every case in square B gets a “zero”. Cases in squares C and
D are not included in the analysis.
• Agreement given reference size ≥ threshold: Every case in square D gets a
“one” and every case in square C gets a “zero”. Cases in squares A and
B are not included in the analysis.
It is possible to generalize to multiple thresholds by considering all squares
that are equivalent (total agreement) or all squares in a particular row
(agreement given a particular reference result).
If the data being compared is ordinal or qualitative, a similar table can be
constructed and binary outcomes can be determined. For example, qualitative
data may be one of three disease types. Total agreement could be determined by
assigning cases a "one" when the reader and the reference decide the same
disease type and "zero" when the reader and reference decide different disease
types.
The most important issue to consider when binarizing data is that a measure of
agreement and the rules for dichotomizing the data make sense to clinicians,
you, your collaborators, and your audience.
BTW, for ordinal data, we are investigating concordance measures for
implementation in iMRMC. These would allow you to have multi-level truth
instead of binary truth. AUC is a special case of a conditional concordance
measure. Other common concordance measures are Kendall’s tau-a and tau-b.
Please refer to the following for more information.
Kim, J.-O. (1971), 'Predictive Measures of Ordinal Association.' Am J Sociol,
76, (5), 891-907.
Smith, W. D.; Dutton, R. C. & Smith, N. T. (1996), 'A Measure of Association
for Assessing Prediction Accuracy That is a Generalization of Non-Parametric
ROC Area.' Stat Med, 15, (1), 1199-1215.
Kendall, M. G. (1938), 'A New Measure of Rank Correlation.' Biometrika, 30,
(1/2), pp. 81-93.
Original comment by Brandon.Gallas on 28 Nov 2014 at 4:01
- Changed state: Answered
- Added labels: Type-Question
- Removed labels: Type
Original issue reported on code.google.com by
cedric.m...@free.fron 13 Nov 2014 at 4:13