When to stop

[azhe825]

Goal

When stops, it should be:

• retrieved roughly 95% relevant docs
• number of docs reviewed as small as possible

Estimate

57 updated

Conclusions:

• our estimation (SEMI) is better than the state-of-the-art and is completely compatible with FASTREAD
• SEMI can have a good estimation in ealy stage
• Estimation from SEMI is close enough to the true prevalence that it might be able to be used as a stop rule

Stop rule with Estimate

Stop when #relevant found >= 0.95 * #estimate:

When it stops, x = #studies reviewed pos = #relevant found.

Hall:

• x: median = 490 [440, 440, 590, 540, 630, 470, 430, 560, 830, 490, 420, 450, 530, 430, 720, 690, 690, 470, 490, 480, 440, 540, 570, 470, 890, 490, 470, 700, 480, 880],
• pos: median = 102 = 96.2% (Total: 106) [102, 102, 102, 102, 102, 103, 102, 102, 102, 103, 101, 103, 102, 102, 102, 102, 103, 102, 102, 102, 102, 101, 102, 102, 102, 102, 101, 102, 103, 102]

Wahono:

• x: median = 1060 [980, 610, 1410, 1530, 1630, 740, 600, 1270, 1510, 510, 1590, 1450, 1170, 1700, 510, 850, 680, 950, 610, 1490, 990, 660, 1650, 1240, 1630, 1130, 680, 1410, 690, 580],
• pos: median = 59 = 95.2% (Total: 62) [59, 59, 59, 61, 59, 58, 57, 59, 61, 58, 59, 60, 60, 61, 58, 59, 58, 59, 59, 59, 60, 58, 61, 59, 59, 60, 59, 59, 57, 57]

Danijel:

• x: median = 860 [540, 910, 420, 880, 550, 970, 940, 800, 1070, 310, 340, 710, 970, 1040, 870, 1190, 770, 1040, 1000, 1120, 470, 850, 330, 1180, 1010, 1110, 390, 840, 560, 530],
• pos: median = 45 = 93.8% (Total: 48) [43, 46, 41, 45, 43, 45, 45, 46, 47, 38, 40, 45, 46, 43, 46, 45, 45, 46, 46, 45, 42, 46, 40, 46, 45, 45, 42, 44, 43, 43]

Kitchenham:

• x: median = 470 [520, 500, 410, 480, 510, 500, 650, 470, 450, 500, 500, 420, 470, 470, 470, 440, 500, 390, 440, 440, 450, 470, 480, 400, 410, 520, 480, 540, 460, 540],
• pos: median = 36 = 80.0% (Total: 45) [36, 33, 34, 36, 32, 36, 38, 38, 37, 38, 35, 35, 38, 35, 38, 37, 37, 34, 35, 34, 36, 37, 37, 38, 35, 37, 36, 37, 38, 36]

Conclusion:

• works fine on Hall, Wahono, Danijel. retrieved around 95% relevant studies.
• underestimate the number of relevant studies in Kitchenham dataset, retrieved 80%.

[azhe825]

Cormack Stopping: Engineering quality and reliability in technology-assisted review

The knee method:

a geometric stopping procedure, based on the shape of the gain (i.e., recall versus effort) curve, that augments BMI to achieve similar empirical reliability to the target method, with substantially less effort. The knee method, in contrast to the target method, is practical regardless of the number of relevant documents in the collection.