JOSS Review - Paper Comments

gchure commented 1 month ago

Hi! I'm reviewing your paper for JOSS and am opening issues as I make my way through the checklist. This issue pertains to the software paper.

Overall, I think the paper is good and meets the requirements for JOSS. None of my comments below are blocking, and you should take/leave them at your (and @ppxasjsm) discretion.

Clarity

I’m a practicing microbiologist, and even though I live and breathe quantitative plating assays, I had a hard time understanding precisely what AIgarMIC is used for. I think having a diagram showing the basic workflow of a MIC assay would help the reader understand what type of data AIgarMIC is expecting. Repurposing figure 1 from your ASM Spectrum paper I think would be very helpful (which should be fine, considering it’s CC-BY 4.0 licensed).
I think having an additional figure showing an API diagram for AIgarMIC would be very helpful. This is necessary beyond developers, so I think it’s worthwhile including in the paper, even though you have pretty detailed documentation.
As AIgarMIC is at heart a ML model, you should provide some information both in the paper and the documentation about the structure of the network and some of the design decisions you’ve made. You did this in your ASM spectrum paper, but I find its absence from the JOSS paper to be a bit confusing. I strongly think you should include a section outlining the structure of the network and information about how it has been trained, pointing to your ASM paper for more detail on the validation of its efficacy. Additionally, this will allow you to reference libraries such as Keras that you use extensively in the software.

Statement of Need

The statement of need is pretty sparse and could be more clear. My view is that most MIC is done qualitatively by visual inspection of plates or very rudimentary measurements. AIgarMIC provides a means by which to make this more robust and transferrable between labs and scientists. There should be a brief outline of how this emerges.

Line-by-line Comments

Here are some minor comments that I think would help the language of the paper. Again, take or leave these as you see fit.

Line 11: MIC is not a test, it’s a quantity that an assay is performed to infer.
Line 12: Clinic testing is one of the outcomes, not the only one. It’s broadly used in microbiology.
Line 17: “[…] bacterial growth on each position on solid […]”. It’s not clear from this sentence what “position” means, unless you are deeply familiar with how these assays are technically run. I would be more pedagogical and verbose with what you mean (or better yet, use a figure).
Line 18: “[…] 3-dimensional growth matrix.” What are the dimensions? Time, concentration, strain? I think you should be more clear with what types of data AIgarMIC is designed to process.
Line 21 - 22: “[…] Users can customise AIgarMIC to their workflow with bespoke models, or use the pre-trained models provide.” This is not clear or supported in the paper, though the documentation is detailed. Giving a small example or pointing to specific pages in the documentation would be really useful.
Line 23: “[…] minimizes measurement error.” This is very broad and does not go into what types of “measurement error” are considered. Minimizing implies an optimization procedure, which I do not think is warranted. I think this should be replaced with “reduces subjective manual assignment error” or something along those lines.

agerada commented 1 month ago

Thank you @gchure for the helpful comments. We will review the paper in light of these comments over the next few days, and produce a revised version and response.

agerada commented 3 weeks ago

Firstly, thank you, @gchure for your very constructive comments and contributions to this project. All points raised are reasonable; we have tried to incorporate them all into the revised manuscript (latest version in main branch). Comments below (line references are for the revised manuscript):

I’m a practicing microbiologist, and even though I live and breathe quantitative plating assays, I had a hard time understanding precisely what AIgarMIC is used for. I think having a diagram showing the basic workflow of a MIC assay would help the reader understand what type of data AIgarMIC is expecting. Repurposing figure 1 from your ASM Spectrum paper I think would be very helpful (which should be fine, considering it’s CC-BY 4.0 licensed).

Thank you for the feedback. We agree and have repurposed our figure from ASM Spectrum, with the following modification:

Focus towards how the software fits within the laboratory workflow.
Clear delineation of the steps that AIgarMIC can process/automate.
Formatting changes to fit better with JOSS style.

I think having an additional figure showing an API diagram for AIgarMIC would be very helpful. This is necessary beyond developers, so I think it’s worthwhile including in the paper, even though you have pretty detailed documentation.

We have made two major changes to the manuscript based on this comment. Firstly, a 'Software design' section was added, giving a broad overview of how users interact with AIgarMIC's API, including some minimal examples. Secondly, API diagrams were added, covering the Plate, PlateSet, and Model derivative classes, which are the main interfaces for user interaction with the software. We have included typing in the class diagrams, but happy to remove them if it is deemed too verbose.

As AIgarMIC is at heart a ML model, you should provide some information both in the paper and the documentation about the structure of the network and some of the design decisions you’ve made. You did this in your ASM spectrum paper, but I find its absence from the JOSS paper to be a bit confusing. I strongly think you should include a section outlining the structure of the network and information about how it has been trained, pointing to your ASM paper for more detail on the validation of its efficacy. Additionally, this will allow you to reference libraries such as Keras that you use extensively in the software.

We have given a lot of thought to this comment. The reasons for not including details on the neural architecture design in this manuscript were two-fold: firstly to avoid redundancy with the ASM spectrum paper, and secondly because we expect that a significant proportion of the user base will need to train their own models (e.g., if testing non-Enterobacterales strains, or if images are collected in significantly different lighting conditions). To support this, we have now included a paragraph in the software design section on how AIgarMIC facilitates the training of custom models (lines 51--55). Furthermore, we now specifically reference the ASM Spectrum paper for users to review the model architecture (lines 43--44).

The statement of need is pretty sparse and could be more clear. My view is that most MIC is done qualitatively by visual inspection of plates or very rudimentary measurements. AIgarMIC provides a means by which to make this more robust and transferrable between labs and scientists. There should be a brief outline of how this emerges.

Thank you, all taken on board and re-wrote the first two paragraphs of the statement of need section to make this point clear.

Line 11: MIC is not a test, it’s a quantity that an assay is performed to infer.

Thank you -- corrected. Changed to MIC assays (plural to reflect the point that there are other ways to test MIC).

Line 12: Clinic testing is one of the outcomes, not the only one. It’s broadly used in microbiology.

Fully agree, and have changed line to reflect this.

Line 17: “[…] bacterial growth on each position on solid […]”. It’s not clear from this sentence what “position” means, unless you are deeply familiar with how these assays are technically run. I would be more pedagogical and verbose with what you mean (or better yet, use a figure).

Agreed, the use of position has been removed, and instead we use Figure 1 to make the point visually.

Line 18: “[…] 3-dimensional growth matrix.” What are the dimensions? Time, concentration, strain? I think you should be more clear with what types of data AIgarMIC is designed to process.

This paragraph has been re-written to present a higher-level description in parallel to Figure 1 which demonstrates the discrete steps. We hope that the point we were making is clearer through the figure.

Line 21 - 22: “[…] Users can customise AIgarMIC to their workflow with bespoke models, or use the pre-trained models provide.” This is not clear or supported in the paper, though the documentation is detailed. Giving a small example or pointing to specific pages in the documentation would be really useful.

We have added two paragraphs that go in more detail on how users can use (lines 46--50) and train (lines 51--55) custom models. For this section, we avoided lengthy code examples but provided direct links to the documentation for more information.

Line 23: “[…] minimizes measurement error.” This is very broad and does not go into what types of “measurement error” are considered. Minimizing implies an optimization procedure, which I do not think is warranted. I think this should be replaced with “reduces subjective manual assignment error” or something along those lines.

Text updated to reflect the point, to: "[…] reduces human error, and reduces subjective operator variability." (line 29).

We are grateful for the feedback, and happy to make further changes if needed.

gchure commented 3 weeks ago

I read through the paper and my concerns are assuaged. Thanks for taking care of them and giving them thorough consideration!

I cannot build the pdf using Inara nor through the JOSS editorial bot, though the error messages aren't too illuminating. Were you able to build the PDF locally?

agerada commented 3 weeks ago

The pdf issue was quite strange as I haven't had any problems building the paper using the JOSS github action. Looking through the error, I wonder whether it was a temporary issue with Inara, as it failed prior to attempting to compile the pdf. I triggerred editorialbot just now and it seems to have compiled without issue.

gchure commented 3 weeks ago

Thanks! Looking through it now. I have a few more points that I think would really strengthen the paper.

Figure 2 -- I think there needs to be some plain-english expalanation of what is in the API for the Plate and PlateSet. Something as simple like "get_csv_data(): list[dict] -> Loads data from csv given column keys" for what's listed in the api diagram.
In "Software Design", the small script needs comments, especially where you're appending the growth matrices. You mention in the text that this is to demonstrate the utility of it, but that's not evident from the small script.
Could you also put in a short example of the command line usage? It doesn't have to be a true executeable example, but something that guides what the pipeline would look like for a user.

Other than that, I think that's all of my comments on the paper. Thanks again for entertaining them!

agerada commented 3 weeks ago

Thank you @gchure – happy to make all of those changes. I will work on them in the next few days.

agerada commented 2 weeks ago

Hi @gchure. Just a quick update on your last comment. I have made the proposed changes (see PR #30), look forward to see what you think.

As a side note, thank you for linking your recent paper and software (https://github.com/cremerlab/hplc-py). I have taken some more inspiration from your work, and integrated a workflow for publishing directly to pip (https://github.com/agerada/AIgarMIC/blob/main/.github/workflows/publish.yml). I'm very happy with the seamless testing and release workflow we have set up, which has made it easier to get releases out to users quicker.

Thanks again!

gchure commented 2 weeks ago

This looks great (and glad that my project helped with some gh-actions inspiration). I'm satisfied with the work overall, and can consider my review complete

agerada / AIgarMIC