search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
701 stars 36 forks source link

[PRE REVIEW]: HDMIA: An Architect for High Demand Medical Image 2 Analysis using Deep Learning #4087

Closed whedon closed 2 years ago

whedon commented 2 years ago

Submitting author: @magedhelmy1 (Maged Helmy) Repository: https://github.com/magedhelmy1/capillarydetection Version: 0.1 Editor: @Bisaloo Reviewer: Pending Managing EiC: Kyle Niemeyer

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/5508f9f11abc53f889b6547dc68dd6ef"><img src="https://joss.theoj.org/papers/5508f9f11abc53f889b6547dc68dd6ef/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/5508f9f11abc53f889b6547dc68dd6ef/status.svg)](https://joss.theoj.org/papers/5508f9f11abc53f889b6547dc68dd6ef)

Author instructions

Thanks for submitting your paper to JOSS @magedhelmy1. Currently, there isn't an JOSS editor assigned to your paper.

The author's suggestion for the handling editor is @Bisaloo.

@magedhelmy1 if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSS submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands
whedon commented 2 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 2 years ago

Wordcount for paper.md is 732

whedon commented 2 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.26 s (216.3 files/s, 76470.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
JSON                             3              0              0          16768
Python                          22            365            169            821
JavaScript                       8             84             15            579
YAML                             5             42            356            412
Markdown                         4             67              0            149
TeX                              1             11              0             89
CSS                              3              8              0             56
HTML                             2              3             20             37
Dockerfile                       5             14             10             32
Bourne Shell                     3             13              0             28
SVG                              1              0              0              7
-------------------------------------------------------------------------------
SUM:                            57            607            570          18978
-------------------------------------------------------------------------------

Statistical information for the repository '06e55491a54eb028252e02ad' was
gathered on 2022/01/21.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Jorgen Ader                     14           104             93            0.04
Jyrno Ader                       4            17              1            0.00
Maged Helmy                    323        257727         255721           99.96

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Jorgen Ader                  27           26.0          1.1               18.52
Jyrno Ader                   16           94.1          0.4                0.00
Maged Helmy                1990            0.8          1.5                7.14
whedon commented 2 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- 10.1007/s00134-018-5070-7 may be a valid DOI for title: Second consensus on the assessment of sublingual microcirculation in critically ill patients: results from a task force of the European Society of Intensive Care Medicine
- 10.1109/ms.2021.3089730 may be a valid DOI for title: Database of the Year: Postgres

INVALID DOIs

- None
whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

kyleniemeyer commented 2 years ago

Hello @magedhelmy1, thanks for your interest in JOSS.

I noticed that you described your submission as an "original research study". JOSS does not actually review or publish original research, but instead research software packages: https://joss.readthedocs.io/en/latest/submitting.html

Original research should be submitted to a domain journal for review; we would review software that was developed in support of or related to that research. Can you clarify how your submission meets that requirement?

magedhelmy1 commented 2 years ago

Hi @kyleniemeyer , thanks for your comment.

Indeed, this is not an original research study to be published here. But this is a research software package intended to address a research challenge: quantifying blood capillaries in patients. This software research package aims to support the functioning of research instruments (blood vessel quantification) and the execution of research experiments (quantifying the capillaries in the skin). My apologies if that was not clear.

Another small typo is in the title "HDMIA: An Architect for High Demand Medical Image 2 Analysis using Deep Learning", the number "2" should not exist there. Again, please accept my apologies for that.

magedhelmy1 commented 2 years ago

Hi @kyleniemeyer , I hope you had a good weekend.

In addition to the previous answer, I include a detailed answer where applicable below to the points in the link shared.

Submission requirements

I, the main author, have 99.96% contribution to the code as per the commits.

No new research has been accomplished in this paper. The proposed research software quantifies the capillaries to enable doctors and clinical researchers to develop their medical hypotheses with regard to specific diseases and capillary density alternation.

In addition, the software associated with your submission must:

What we mean by research software

This paper solves a problem in medicine, specifically the automated quantification of capillaries in Microcirculation images.

Substantial scholarly effort

First, we enable some new research challenges to be addressed by allowing the complete automation of quantifying the capillary density in Microcirculation images. On the second fold, the proposed software makes addressing research challenges significantly better by enabling doctors, researchers, and clinical personnel to develop their hypotheses with regards to the significance of capillary density changes in patients with specific diseases. More concretely, it allows them to assess if various diseases indeed alter the capillary density and by how much.

Almost daily commits for the past 120 days (with the exclusion of Christmas and new year, weekends and public holidays)

Me plus two professors in computer science and one associate professor in mathematics since we spent 1-year surveying state of the art and preparing the proposed architecture in this research package. This can be evident that the capillary quantification algorithm proposed in JOSS can be found in a time-stamped paper on Arxix from 23 Apr 2021

A different set of authors has informed me they are working on 2 separate papers that will use the proposed research software. The first paper will compare the alteration of capillary density between recovering COVID19 patients and dying COVID19 patients. The second paper will compare the capillary density between pigs with a mechanical heart and pigs with a normal heart. Another set of authors has informed me that they will explore the option of using the proposed software to compare the capillary density in healthy subjects and patients with diabetes and if capillary density can be used to predict amputation of a diabetic patient.

Both papers are still in writing with the authors.

Our proposed software is currently used by a company that provides microcirculation analysis software. Furthermore, the authors mentioned in the previous point have cited our software and those citations will be public if their papers have been accepted for publication. We are optimistic that more will cite and use our proposed research package since it automates a manual process with comparable accuracy to the manual researcher.

As part of the repo, we deployed our system here http://www.analysecapillary.space/ so it can be used out of the box.

Co-publication of science, methods, and software

As indicated in the README, the details of the capillary detection and quantification algorithm can be found here CapillaryNet: An Automated System to Quantify Skin Capillary Density and Red Blood Cell Velocity from Handheld Vital Microscopy. The paper is currently under review with the Artificial Intelligence in Medicine journal (AIIM). However, the CapillaryNet paper submitted to AIIM covers more algorithms than just capillary detection and capillary quantification (Capillary Velocity, Intra mediate flow and Capillary Flow Direction). The submitted research software to JOSS focuses on capillary quantification and in addition, its deployment, usage, software architecture, and the "High Demand Part" where the servers have an RPS of 3000+.

Your paper should include:

arfon commented 2 years ago

@whedon invite @Bisaloo as editor

:wave: @Bisaloo – would you be able to edit this submission for JOSS?

whedon commented 2 years ago

@Bisaloo has been invited to edit this submission.

Bisaloo commented 2 years ago

:wave: :wave: @arfon @magedhelmy1, yes, I can edit this submission

Bisaloo commented 2 years ago

@whedon assign me as editor

whedon commented 2 years ago

OK, the editor is @Bisaloo

Bisaloo commented 2 years ago

Hi @magedhelmy1 :wave:, I'll be your topic editor for this submission.

The first step in this pre-review is to find at least 2 reviewers. If you have any suggestions for potential reviewers, please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

magedhelmy1 commented 2 years ago

Hi @arfon, Thanks for assigning a reviewer to this. Hi @Bisaloo, glad you are on board to be the topic editor for this submission.

I went through the Google sheet and can suggest several names. Any 2 who are available of the following as you see fit.

sgk98 arn1291 gxlarson AnthonyOfSeattle Ykarpate nnadeau dirmeier vyasr lucaferranti rabdill tuelwer robbisg erik-whiting

Bisaloo commented 2 years ago

Thanks for the suggestions! I'm going to already request some changes, as to facilitate the life of potential reviewers and already clarify certain points about this submission. Could you please:

magedhelmy1 commented 2 years ago

Hi @Bisaloo, thank you for being proactive and listing the improvements! I appreciate it. I have incorporated all but one. Below are my comments on each point you stated and the corresponding change location. I am only stuck at one point, which is the second point. Please clarify it, and I will improve right away.

I have updated the references and added DOIs where applicable. The changes are reflected in paper.bib I am unsure how to re-run Whedon to see the report generated and fix the consequential issues.

Extended figures caption to include more details

I have now removed ter temp, jq.exe, ray and other similar files and folders that are obolsete.

I have added this, and the changes are reflected in license.md

I have now removed that file, SECURITY.md .

I have updated this and the changes are reflected in README.MD and contributing.md

I have now added that section, and the changes are reflected in project structure

Bisaloo commented 2 years ago

For the caption, I meant a more informative & more descriptive caption. At the moment, as a reader, I don't really understand what I'm looking at and what information I should get from these figures.

Bisaloo commented 2 years ago

Hi @tuelwer :wave:, I see that you volunteered to review for JOSS and this submission seems to fit perfectly your expertise, both in terms of topic and programming language.

Would you be available to review this submission?

Please note that this is a somewhat complex submission with several layers. If you want to review only one of these layers (e.g., the python backend but not the react frontend), this is perfectly fine :slightly_smiling_face:

magedhelmy1 commented 2 years ago

Thank you @Bisaloo for taking the time to clarify. I have now extended the captions of the figures to include your feedback. Please let me know if I can improve it further or if you have other questions.

Bisaloo commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

tuelwer commented 2 years ago

This is very interesting work!

@Bisaloo unfortunately, I am currently busy and cannot review this submission in reasonable time. Please, feel free to contact me again in a couple of weeks if you have not found a reviewer by then!

magedhelmy1 commented 2 years ago

@tuelwer thank you! and thanks for responding fast.

Bisaloo commented 2 years ago

Thanks for your quick answer @tuelwer, I'll ping you again in a couple of weeks if necessary :slightly_smiling_face:

magedhelmy1 commented 2 years ago

Hey @Bisaloo , happy to suggest other names if needed besides the 13 above.

Bisaloo commented 2 years ago

Hi @fepegar :wave:, I see you volunteered to review submissions for JOSS. This submission, with the associated pape, seems to fit quite well with your expertise. Would you be able to review it?

Please note that this is a somewhat complex submission with several layers. If you want to review only one of these layers (e.g., the python backend but not the react frontend), this is perfectly fine :slightly_smiling_face:

Bisaloo commented 2 years ago

Hi @michaelberks :wave:, I see you volunteered to review submissions for JOSS. This submission, with the associated paper, seems to fit quite well with your expertise. Would you be able to review it?

Please note that this is a somewhat complex submission with several layers. If you want to review only one of these layers (e.g., the python backend but not the react frontend), this is perfectly fine :slightly_smiling_face:

fepegar commented 2 years ago

Hi @fepegar 👋, I see you volunteered to review submissions for JOSS. This submission, with the associated pape, seems to fit quite well with your expertise. Would you be able to review it?

Please note that this is a somewhat complex submission with several layers. If you want to review only one of these layers (e.g., the python backend but not the react frontend), this is perfectly fine 🙂

Hi, @Bisaloo. Thanks for pinging me. Unfortunately, this topic is not really close to my areas of knowledge. Good luck finding someone!

Bisaloo commented 2 years ago

Hi @robbisg :wave:, the author has identified you as a potential good reviewer for this submission.

Would you be available to review this piece of software and associated paper for JOSS?

magedhelmy1 commented 2 years ago

Kind reminder to Bisaloo comments to check if you are available to review the paper submitted to JOSS :)

@michaelberks @robbisg

robbisg commented 2 years ago

Sorry for the delay! Yes I'm available!

Bisaloo commented 2 years ago

Thanks for your answer @robbisg! :pray: Please see my comment below.

@magedhelmy1, while I was looking for reviewers, some comments have been raised about whether this submission is really in scope for JOSS. In particular:

Please let me some time to gather the opinions of other editors, and I'll get back to you as soon as possible. Thanks

Bisaloo commented 2 years ago

@whedon query scope

whedon commented 2 years ago

Submission flagged for editorial review.

danielskatz commented 2 years ago

While useful for many areas of research, pre-trained machine learning models and notebooks are not in-scope for JOSS.

https://joss.readthedocs.io/en/latest/submitting.html

robbisg commented 2 years ago

Hi @Bisaloo ,

at a first sight it seemed to me that this software is a platform for uploading images and applying pre-trained methods for the identification of some diseases. Despite it can be useful for some research activity since it detects capillarity density from images, speeding up this process for clinical applications, it seems that is a bit out of the scope of JOSS, as far as I understood and as pointed out by @danielskatz .

magedhelmy1 commented 2 years ago

Hi @danielskatz , thanks for the hyperlink and highlight. This paper is not a pre-trained machine learning model. The focus of this paper is not the algorithm but rather the "Architecture for High Demand Medical Image Analysis..." built as a platform powered by deep learning models. The algorithm is added for sake of completion and usability of the proposed architecture.

In Figure 3 in the paper, you will see where the "pre-trained" model is in relation to the whole system. It barely occupies 5% of the total system workflow. In fact, it occupies around 2 lines of code in the whole project.

The pre-trained algorithm is submitted as a separate paper for publication in another journal including the data used for training it. This paper focuses on the deployment of this model by coupling several systems together. This way my aim is to meet the following requirement from JOSS The software must have an obvious research application. is @robbisg comment which is it detects capillarity density from images, speeding up this process for clinical applications, or else this is an architecture was no obvious research use.

@robbisg There are several JOSS papers that provide software platforms that do improve existing research activities as you highlighted yourself, for example:

https://joss.theoj.org/papers/10.21105/joss.00040

You can find a few more here if you search https://joss.theoj.org/papers/published

With regards to the JOSS requirements, I have a detailed list earlier https://github.com/openjournals/joss-reviews/issues/4087#issuecomment-1020590129 , I'd appreciate it if you highlight there which part is a bit out of the scope of JOSS and I am happy to address.

danielskatz commented 2 years ago

Thanks @magedhelmy1 - the editors are/will examine this and make a decision, and your comments will be helpful in that process

magedhelmy1 commented 2 years ago

Thank you @danielskatz for keeping me posted. I appreciate it.

Since the comments might be helpful, I will take this opportunity to also address @Bisaloo comments.

it seems to be more a clinical/medical tool than a research tool when our policy states: "The software must have an obvious research application."

This is a clinical tool, yes, but that does not mean it is not a research tool also. In fact, I did go the extra step and built the clinical tool on top of the architecture to demonstrate the usefulness of the proposed architecture as a research tool. In this case, analyze capillary density in under 2 seconds. This should also satisfy the sentence in that same document later which states

Whether or not the software is sufficiently useful that it is likely to be cited by your peer group.

This is a general-use tool that can be adapted to many similar types of research because it provides a general architecture for pushing images through a deep learning model to provide results in real-time.

Thus, I would argue there are 2 interlinked obvious research applications and they are

1) Researchers who are interested in Microcirculation analyses can simply build on this architecture to analyze many different kinds of Microcirculation images. Some expansions on this package are:

I will not state them all here, but if you have the time, please feel free to check table 9 in Second consensus on the assessment of sublingual microcirculation in critically ill patients: results from a task force of the European Society of Intensive Care Medicine where researchers will need such tools if they will to use microcirculation analysis in bedside inside hospitals.

2) As the title captures, this is an architect on how to analyze medical images using deep learning. So if one would strip away the "tool" part from the code, that is the microcirculation images and deep learning code. You can use this architecture to analyze any image in high demand using any deep learning model. In the performance testing, I show that you are able to get up to 3000 Response per Second which is the equivalent of 10.8million RPS per hour. Thus, the "High Demand" part is in the title.

it is not clear to me how a trained ML model without the training data fits in our scope

As highlighted in the earlier comment that this paper is not about the trained ML Model thus I did not provide the data. A paper with the model+data by the same authors is under consideration in another journal (AIIM). I am happy to modify the paper to reflect the above statements since it seems I did not do that adequately.

Happy to address any further comments or clarify myself further.

magedhelmy1 commented 2 years ago

hey @danielskatz, I hope all is well. Any news?

danielskatz commented 2 years ago

👋 @magedhelmy1 - I'm sorry to say that after discussion amongst the JOSS editors, we have decided that this submission has issues in both the JOSS research software and substantial scholarly effort criteria.

This does not mean that it is not software that is useful in research, but just that JOSS does not consider it in scope for review as a substantial contribution of research software. Please see https://joss.readthedocs.io/en/latest/submitting.html#other-venues-for-reviewing-and-publishing-software-packages for other suggestions for how you might receive credit for your work.

Our specific comments are that the submission seems mostly targeted to clinicians, as clearly framed in the paper. This is also highlighted by the fact that there is no way to analyse a folder of images in bulk, or even to export the results of the analysis in a data frame. Currently, the user has to copy the results manually to aggregate them. Additionally, it is not clear what this submission is really about. One major part seems to be a pre-trained ML model, which would make it outside of JOSS scope (+ this part is currently under consideration for publication in another journal according to the author). You defended your submission by saying that the pre-trained ML model is just here to illustrate their submission but that the actual object of interest here is the "architecture", i.e., the fact that you built a react frontend to this ML model and a system to handle the communication between the frontend and the backend. But if we ignore the ML model, we don't think this qualifies as a substantial scholarly effort, as illustrated by the very low number of lines of code.

danielskatz commented 2 years ago

@whedon reject

whedon commented 2 years ago

Paper rejected.