Extended abstract to Machine Learning for Health 2022 (ML4H2022)

[mxochicale]

Machine Learning for Health 2022 (ML4H2022)

The Call for Participation for ML4H2022 is now out! Due to the growing ML4H community, ML4H will be a separate symposium. https://ml4health.github.io/2022/index.html

Important Dates

Sep 1st AoE: Submission Deadline Sep 30th : Author Response Period Starts Oct 5th : Author Response Period Ends Oct 21st: Final Decisions Released Nov 14th [tentative]: Camera Ready Deadline Nov 28th: Hybrid Event

Submission Instructions

Submissions (full papers and extended abstracts) are due on September 1st 11:59 PM AoE in the form of anonymized PDF files. There is no separate submission registration deadline. As part of the submission, authors are required to fill out a submission form that will be visible to reviewers to help them assess the work. Authors will also indicate whether they would like the submission to be in the proceedings track or the extended abstract track.

All submissions for ML4H 2022 will be managed through the OpenReview system. Submissions must be formatted using the ML4H 2022 LaTeX template; gross violations of formatting guidelines may be desk-rejected without review.

Submission Site: https://openreview.net/group?id=ML4H/2022/Symposium

ML4H 2022 LaTeX template: download link or Overleaf

Machine Learning for Health 2021 (ML4H2021): https://ml4health.github.io/2021/

• We have accepted 18 papers to be included in the Volume 158 of the Proceedings of Machine Learning Research: https://ml4health.github.io/2021/papers.html
• We have accepted 30 extended abstracts for presentation at the workshop, which are hosted on the ML4H 2021 arXiv index: https://arxiv.org/html/2112.00179

[mxochicale]

Writing a good ML4H Paper

Extended Abstract

An excellent extended abstract is one that leads to insight at the symposium through interaction with other attendees. This can be through presenting new ideas/ways of thinking, leading to insightful discussion and feedback, dissemination of new valuable resources, or enabling new opportunities for collaborations.

Extended abstract submissions should demonstrate that the work will produce fruitful discussion when presented at the symposium. Highlight opportunities for insightful discussion and demonstrate that your work will contribute to a creative, engaging, and constructive poster session. Reviewers will be explicitly asked to gauge how valuable they feel this work could be to other attendees of the event, as well as how valuable attending the event could be to the author of the abstract. This is not a license to submit low quality or barely begun work -- while these submissions may garner constructive comments during the review process, they will not likely generate useful discussions or insightful feedback during the event.

Reviewers will be asked to answer specific questions regarding the relevance to healthcare in the review form. It may be beneficial for authors to explicitly discuss these points in the abstract.

Exemplar abstracts These abstracts from a prior year’s event presented preliminary, but promising ideas that served as good discussion points between the authors and other attendees.

Drug Repurposing for Cancer: An NLP Approach to Identify Low-Cost Therapies Transfusion: Understanding Transfer Learning for Medical Imaging (This is the submitted, extended abstract version -- the work has also been published separately as a full paper, and interested readers should see that here)

https://ml4health.github.io/2022/writing_guidelines.html

[mxochicale]

Few comments to address:

• [x] "I struggle to review as I don't understand if it will contain original data (as it seems) or not. Why don't you structure it as a conventional abstract with data? At the end of the day you have an aim in mind and data" ~Luigi Pisani

• [x] "check with the workshop organisers that they consider papers that are not original research, but instead are an overview of the state of the art in some topic" ~Alberto Gomez

  Hello! Thank you for your email. At first glance, this definitely looks applicable and we encourage you to submit. Of course it remains difficult to assure whether this topic will be relevant and accepted before reviewers see the whole thing, but if you would like to get a sense of other extended abstract topics to see how yours might fit, you can look at last years here: https://ml4health.github.io/2021/papers.html Best wishes

• [x] "For this piece of work it is really important that the title, the introduction, and the paper as a whole are very clear of what the paper is about. Otherwise reviewers will look at it as a technical research paper and will most likely dismiss it." ~Alberto Gomez

• [x] Alberto Gomez: The title suggests that you propose a method for RT AI echo. Which you don't so the title should nbe changed, perhaps to something like: "A review on Real-time..." or "Unsolved problems in Real Time-AI..." Also please check Nhat's paper (white paper presnted at the MICCAI-FAIR workshop last year) on challenges of AI-enabled echo in low resource ICUs. Make sure what are the differences and synergies. I think yours is similar, but on the technical side of things Aug 16, 2022 4:28 PM

• [x] Alberto Gomez: There is something missng after this, but within this section. You have just identified a problem (there is little to no studies on RT AIE echo for 4ch views in ICU), so the reader expects that this is going to be the missing study. But it is not really. So you should say what this paper is. Aug 16, 2022 4:31 PM

• [x] Alberto Gomez: What this new content should include are brief answers to the following questions: - What is the main controbiution of this paper? - What is the main conclusion of this contribution? Aug 16, 2022 4:32 PM

• [x] Alberto Gomez: - How is this contribution assessed or validated? - How is the paper organised to describe this contribution? Aug 16, 2022 4:32 PM

[mxochicale]

• [ ] checking/reviewing accepted abstracts in previous years 2021: https://arxiv.org/html/2112.00179 2020: https://arxiv.org/html/2011.11554v1 (in which there is only one US-related work by J Galeotti in ML4H2020 https://arxiv.org/pdf/2011.07019.pdf) 2019: https://arxiv.org/html/2002.01584v1 2018: https://arxiv.org/html/1811.07216

[mxochicale]

• [x] nhat.phung: The title might be misunderstood. I don't see you say much about LMIC here. It rather a technical side of realtime AI echo Aug 19, 2022 11:00 AM
• [x] nhat.phung: do you mean "reduce" inter-observer variability Aug 19, 2022 11:09 AM
• [x] "\citet{hanson2001} also touched on data visualisation to demonstrate the hypothetical ICU for large number of patients (head injury, sepsis, acute respiratory distress syndrome, etc)." > nhat.phung: it's not related to the heading of the section Aug 19, 2022 11:35 AM
• [x] "\citet{Ghorbani-DigitalMedicineNature-JAN2020} reported the first deep learning model to predict age, sex, weight and height from echocardiogram images and make use of such models to understand how models predicts systematic phenotypes which are difficult for human interpreters. Authors trained CCN models with 2.6 million echocardiogram images from 2850 patients with the extraction of labels local structures and features (e.g. pacemaker lead, dilation of left atrium, hypertrophy for left ventricular) and labels from the physician-interpreted report (e.g, catheters, pacemaker, and defibrillator leads)." nhat.phung: same as above, I don't see it link with ICU. maybe move it to section 3. Real-time AI-empowered echo. or combine two section and touch a bit on ICU context, and you can say that very few research have been done in ICU. which you are gonna do in this paper. Aug 19, 2022 11:37 AM

[mxochicale]

• [x] " {\caption{Real-time AI clinical system (a) clinical system, (b) deep learning pipeline.}}" nhat.phung: is it your proposed pipeline? what is other's pipeline Aug 19, 2022 11:30 AM
• [x] "\subsection{Light neural networks to classify US images}" nhat.phung: why light neural networks? you have not described the benefit, and how it may help in AI-enabled echo Aug 19, 2022 11:28 AM
• [x] " {\caption{Neural Networks}}%" nhat.phung: why do you show the parameters of MobileNet? I assume it is part of your method Aug 19, 2022 11:39 AM

[mxochicale]

Feedback from our meeting on 23-Aug-2022

Andy:

• [x] Check that inference is for clips and not batches

  Inference analysis is no included in this work but perhaps I can add few results if abstract is accepted. ~MX

• [x] Perhaps try LSTM, frames as multiple channels or voting procedures (5frames, voting > majority)

  Great suggestions but will put it as future work due to bandwidth capacity  ~MX

• [x] Title should math the contributions of the paper, perhaps something like this "Thinner networks for view classification..."

Nhat:

• [x] Add further details for background and 4chamber view selection
• [x] Add deployment details of real-time application.

Leave real-time deployment for future work! So title has been amended:

%A Machine Learning Case Study for Real-time AI-empowered echocardiography of Intensive Care Unit Patients in low- and middle-income countries  %Wed 31 Aug 06:18:13 BST 2022
A Machine Learning Case Study for AI-empowered echocardiography of Intensive Care Unit Patients in low- and middle-income countries  %Fri  2 Sep 04:10:09 BST 2022

These are few words on RT AI for US

\subsection{Real-time AI-empowered echocardiography} %In terms of real-time analysis of echocardigraphy \subsubsection{State of the art} \label{subsec:State_of_the_art} \citet{woudenberg2018} trained an DenseNet-LSTM with 2000 clips of apical 4 chamber view in which the real-time system made use of 10 input frames and reported a latency of 352.91ms. \citet{toussaint2018-MICCAI} proposed ResNet18-SP trained with 85,000 frames of Fetal US imaging, reporting real-time performance at inference time of 40 m$s$ per image ($\sim$20Hz). \citet{ostvik2021-TMI} proposed Echo-PWC-Net trained with synthetic, simulated and clinical datasets, reporting real-time performance with 7 frames for the input. Recently, \citet{wu2022} applied baselines of UNET with temporal context-aware encoder (TCE) and bidirectional spatiotemporal semantics fusion (BSSF) modules to EchoDynamic %datasets %(10,030 video sequences with of 200 frames of 112x112 pixes) and CAMUS datasets %(450 video with 20 frames of 778x594 pixels) , reporting metrics of Dice score (DS), Hausdorff Distance (HD), and area under the curve (AUC). To ensure low latency and real-time performance, \citet{wu2022} presented a comparison of eight methods networks including FLOPS (G), number of parameters (M) and speed ($ms/f$) being their method with the lowest speed at 32 $ms/f$ and 56.359 $G$ FLOPS but network size was 74.79M parameters (join motion model with 237.592G FLOPS, 17.315M parameters and seep of 154 $ms/f$).

[mxochicale]

From: Sophie Yacoub <syacoub @ oucru.org> Sent: 26 August 2022 10:54

• [x] Not sure about the format you were aiming for - but it seems a bit heavy on the intro/literature search and past studies and minimal on your results. Can you expand the results a bit more?
• [x] And add the potential clinica application and clinical role in the conclusion?

[mxochicale]

Comments from Louise Thwaites <lthwaites @ oucru.org> Sent: 28 August 2022 18:37

• [x] explain basic protocol

  Echocardiography is an increasingly valuable clinical tool in the Intensive Care Units (ICU) due to its advantages of portability, low cost, low radiation and ability to provide real-time visualisation of dynamic cardiac anatomy \citep{Feigenbaum1996, Vieillard-Baron2008, singh2007, cambell2018}. [\textbf{I SUGGEST ANOTHER SENTANCE OR TWO EXPLAINING WHAT THE BASIC PROCEDURE INVOLVES AND WHAT THE STANDARD VIEWS ARE - MAYBE FOCUS ON THE 'FICE' PROTOCOL AND DISCUSS THAT VARIOUS COUNTRIES NOW MANDATE BASIC CARDIAC ECHO IN ICU TRAINING - Uk, australia etc}]

• [x] clarity on challenges

  \item Inter-view similarity of echocardiograms (similar views of valve motion, wall motion, left ventricle, etc) and transducer position during acquisition \citep{zhang2018}, [\textbf{\textit{I AM NOT SURE HOW THIS IS A CHALLENGE - IT JUST SEEMS TO BE OPPOSITE OF THE FIRST}}]

• [x] clarity on challenges

  \item Redundant information in the clinical echo system (icons, date, frame rate, etc) \citep{khamis2017} and variation of Ultrasound images from different clinical US systems \citep{brindise2020unsupervised}, and [**A\textbf{GAIN I AM NOT SURE HOW THIS IS A CHALLENGE}..}}.]

[mxochicale]

• [ ] Add the following reference and few of the questions to future work:

Questions to Consider for "github.com/vital-ultrasound":

• Was the Study Methodology Prespecified?
• Is the Model Being Evaluated in its Intended Stage in the Care Pathway?
• Do the Authors Provide Sufficient Clarity on How the Data Were Split?
• Are the Image Labels Likely to Reflect the True Disease State?
• How Is Diagnostic Accuracy Reported?
• Is the Dataset Used in Model Development Reflective of the Setting in Which the Model Will Be Applied?
• Is the Output of the Model Interpretable and Can it Be Interrogated? Are Differential Diagnoses and Estimates of Confidence Provided?
• Is the Performance Reproducible and Generalizable?

See reported answers

Livia Faes, Xiaoxuan Liu, Siegfried K. Wagner, Dun Jack Fu, Konstantinos Balaskas, Dawn A. Sim, Lucas M. Bachmann, Pearse A. Keane, Alastair K. Denniston; A Clinician's Guide to Artificial Intelligence: How to Critically Appraise Machine Learning Studies. Trans. Vis. Sci. Tech. 2020;9(2):7. doi: https://doi.org/10.1167/tvst.9.2.7.

[mxochicale]

Comments from alberto.gomez @ kcl.ac.uk Sent: 30 August 2022 09:38

• [x] 1. In the title, you state two important things: first, that the paper is about the challenges of AI, second that this paper is a machine learning case study. I think your paper is really about the second thing, and not so much about the first. The two challenges that you enumerate in the introduction have been identified elsewhere, hence nt your contrbution here. SO I would suggest to change the title to something like: "Machine Learning for Supporting Real-time echocardiography for Intensive Care Units in low- and middle-income countries: A Case Study". Then make sure that focus of the paper remains on this case study.
• [x] 2. In the intro you describe some challenges. I have added comments and suggestions in overleaf, but in summary you need to expand more on why each of the items is a challenge, why is it important to solve it, and what is the problem with leaving it unsolved.
• [x] 3. At the end of the introduction, when you describe the structure of the paper, you need to make a bit more emphasis of what the paper is about, ad I feel, what it is not about. After the list of challenges, a reader would expect that you propose a method to solve these challenges. So I suggest to write a paragraph where you say that in this paper, instead of focusing on how to solve these challenges, you focus on describing and framing these challenges based on the existing literature, first; and then you exemplify the impact of those challenges with a case study on a specific method for a specific application. Something like this.
• [x] 4. About the paper structure, in the body of the paper there are clearly two distinct parts: the literature review, and the case study. I suggest these correspond to two sections: so merge 2 and 3 (and use sub sections if appropriate)
• [ ] 5. Because the aim of this paper is to identify some challenges and limitations of current methods, I think the discussion and conclusions looks a bit thin. You should use this section to , first, put in correspondance the literature review and the case study, and second wrap up your findings, so I suggest including some reflections about the following: - A summary of the main unsolved challenges, in general and how specifically they map to your case study
• [ ] 6. - A reflection on priorities: of those challenges, which one is most important and why
• [ ] 7. - How can the challenges be solved, or where research should focus. For example: making datasets available to improve diversity, investigating ways of making AI methods insensityve to different manufacturers, make models lighter so they run on cheaper hardware, etc.
• [ ] 8. - A reflection of whythese problems are still unsolved. Why is there so little research on this if it is so important? Is there something missing in the clinician / scientist collaborations? Do we need to engage more with industry, or with regulatory bodies?
• [ ] 9. - FInally an outline of future directions specifically for your case study.

[mxochicale]

:fire: submitted abstract!

[mxochicale]

I close this one as abstract was successfully submitted, however feel free to look previous comments as those might lead to refine what was submitted on Fri 2 Sep 12:59:26 BST 2022 https://github.com/vital-ultrasound/ML4H2022/blob/ad4483101f60ea6b14aa1075811e435bd9140647/ml4h2022.pdf