"Is the Dataset Used in Model Development Reflective of the Setting in Which the Model Will Be Applied?"

[mxochicale]

🚀 Feature

Questions to Consider for "github.com/vital-ultrasound":

• Was the Study Methodology Prespecified?
• Is the Model Being Evaluated in its Intended Stage in the Care Pathway?
• Do the Authors Provide Sufficient Clarity on How the Data Were Split?
• Are the Image Labels Likely to Reflect the True Disease State?
• How Is Diagnostic Accuracy Reported?
• Is the Dataset Used in Model Development Reflective of the Setting in Which the Model Will Be Applied?
• Is the Output of the Model Interpretable and Can it Be Interrogated? Are Differential Diagnoses and Estimates of Confidence Provided?
• Is the Performance Reproducible and Generalizable?

See reported answers

Livia Faes, Xiaoxuan Liu, Siegfried K. Wagner, Dun Jack Fu, Konstantinos Balaskas, Dawn A. Sim, Lucas M. Bachmann, Pearse A. Keane, Alastair K. Denniston; A Clinician's Guide to Artificial Intelligence: How to Critically Appraise Machine Learning Studies. Trans. Vis. Sci. Tech. 2020;9(2):7. doi: https://doi.org/10.1167/tvst.9.2.7.

Motivation

Pitch

Alternatives

Additional context

[mxochicale]

To add in references as few of the questions as future work https://github.com/vital-ultrasound/ML4H2022/issues/13#issuecomment-1230193866