Decision on article: revision

[Benjamin-Lee]

Good news! PLOS got back to us and our manuscript is likely to be accepted:

Dear Mr. Lee,

Thank you very much for submitting your manuscript "Ten Quick Tips for Deep Learning in Biology" for consideration at PLOS Computational Biology. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. The reviewers appreciated the attention to an important topic. Based on the reviews, we are likely to accept this manuscript for publication, providing that you modify the manuscript according to the review recommendations.

Both reviewers mention important issues that need to be adressed to satisfy the editors, and I ask you to address all points if you wish to be considered further.

Please prepare and submit your revised manuscript within 30 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to all review comments, and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Thank you again for your submission to our journal. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

@bffo

BF Francis Ouellette Education Editor PLOS Computational Biology

Patricia M Palagi Education Editor PLOS Computational Biology

---

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

Reviewer's Responses to Questions

Comments to the Authors: Please note here if the review is uploaded as an attachment.

Reviewer #1: The authors have specified the importance of using Deep Learning in Biological data and on what are the aspects they have to focus when the researchers are using the Deep Learning. Even though they have mentioned ten different areas they have to focus the authors did not mentioned enough suitable examples and some common practices for some areas. Further it is important to mention some of the difficulties and limitations they have to face when they are using these methods. Specially need to identify what sort of the data need to be there and how to react when there is not enough data to carry out Deep Learning technique. In the paper some area not contain precise information with relevant references to support the statements. Hence these areas need to improve it further.

Reviewer #2: The article titled “Ten Quick Tips for Deep Learning in Biology” provides a review of potential challenges to employing deep learning in biomedical research. This well written article is of importance at a time when deep learning is broadly viewed as a magic solution to many prediction and modelling problems. It cautions the inexperienced and reminds the experienced researcher that while DL has led to remarkable advances in certain applications, it is far from guaranteed to succeed in a broad range of biomedical applications for a gamut of reasons, can lead the researcher astray, be time consuming to develop, and result in unexpected ethical risks. While the article is well written, easy to follow and of interest to the readership, a few points for improvement are worth considering prior to publication: Major comments • In point 10, Distributed learning should be mentioned. An example reference: https://www.sciencedirect.com/science/article/pii/S0010482521005102 • I think it might be worth to further emphasis how results should be evaluated in the presence of class imbalance and risk/benefits of incorrect/correct predictions in the context of the specific application. On a similar note, it is mentioned that AUCROC may not be reliable, but consider recommending alternatives such as precision-recall curves. Minor comments: • Lines 153: those though • Line 183 is missing a period • Paragraph starting at line 271: This paragraph is cumbersome and contains jargon that is not introduced (CUDA/CuDNN). Is it implied that implementation (hardware and software) variations between platforms can hinder reproducibility? Or perhaps the implementation is reproducible, so long that random seeding is not used?

Reviewers who wish to reveal their identities to the authors and other reviewers should include their name here (optional). These names will not be published with the manuscript, should it be accepted.

Reviewer #1: (No Response)

Reviewer #2: Ran Klein

[Benjamin-Lee]

@agitter @cgreene @rasbt and other Deep Review coauthors, how did you manage generating the tracked changes in the past?

[cgreene]

Save word doc from submitted version. After revisions, get word doc from new version. Use compare versions feature in word :)

[Benjamin-Lee]

I hate it but I love it

[rasbt]

Awesome!

@agitter @cgreene @rasbt and other Deep Review coauthors, how did you manage generating the tracked changes in the past?

For everything we changed, we usually used red font color in LaTeX or Word to indicate what has changed. After that was completed, we made a separate version where all the font color was black.

These changes should then also be mentioned in the letter, ideally quoting the changed text (I had a reviewer complaining once that it is inconvenient to have to manually cross reference the responses with the changes in the manuscript; I can agree with that).

The suggestion by @cgreene

Save word doc from submitted version. After revisions, get word doc from new version. Use compare versions feature in word :)

Sounds like a good workaround.

[SiminaB]

Great job team! In terms of revisions - I would go with using something beyond just color changes due to possible colorblindness issues or even just reviewers printing it out greyscale to review it. For instance, you can do underline or boldface (I like to have that + the color changes.)

I 100% agree with having the changed text included in the response letter. If it's up to 1 paragraph, can just quote it in the letter, if it's a bigger reorg or a couple of new pages (doesn't seem like that will be the case though), should at least note the sections with substantial changes.

[agitter]

This is good news.

Reviewer 2 has good and specific suggestions. We'll be able to make these edits.

Reviewer 1's comments are more general. I've extracted these possible revisions:

• a few additional example applications and common practices
• be more specific about data required and what to do where there is insufficient data for deep learning (edit Tip 1?)
• be more precise and add relevant references

Anything else?

[tbrittoborges]

Would you like to "divide and conquer" the tasks for the revision?

[Benjamin-Lee]

Reviewer 2's points are straightforward to implement whereas Reviewer 1 is a little bit harder. I'm happy to take on reviewer 2's comments and make a PR but would appreciate some help on responding to Reviewer 1's comments. Specifically, suggestions of examples that we can add and ways to "contain precise information with relevant references to support the statements" (?) would be most appreciated.

I also completely missed the 30 day turnaround deadline. In my head, I thought there was more time left so I apologize for being distracted the last week.

[agitter]

One way someone could get started helping with that vague reviewer 1 comment would be to go through the full text and identify statements that are not currently referenced or well-supported. Then we could distribute adding appropriate references or removing the claims.

[rasbt]

I think we need to resubmit by the end of the week. (I will try to carve out some time this week to find and add some additional citations if possible)

[Benjamin-Lee]

Looking at the number of citations, it seems the baselines, hyperparameter, and interpretation tips are weakest. Adding 2-3 citations to each would bring them in line with the other sections and probably address the references issue.

@Benjamin-Lee ➜ /workspaces/deep-rules/content (master) $ grep -c -E "\[@"  *.md
00.front-matter.md:0
02.intro.md:6
03.when-to-use-dl.md:23
04.baselines.md:2
05.dl-complexities.md:4
06.know-your-problem.md:4
07.architecture-and-representation.md:11
08.hyperparameters.md:2
09.overfitting.md:10
10.blackbox.md:5
11.interpretation.md:2
12.privacy.md:11
13.conclusion.md:0
...

[agitter]

I also completely missed the 30 day turnaround deadline.

We can ask the editors for more time. Another few weeks should not be an issue for the journal. That will give us some time to finish edits and also circulate them among the co-authors so they can see the diff before resubmitting.

I'll have some time to work on this soon but am unlikely to get to it this week.

[rasbt]

That sounds like a good plan. Ben, can you write the Journal to request more time? Maybe 2-4 weeks?

[Benjamin-Lee]

That sounds like a good plan. Ben, can you write the Journal to request more time? Maybe 2-4 weeks?

Just sent the request, will keep you informed what they reply

[rasbt]

Did they reply to your request?

[Benjamin-Lee]

Apologies, forgot to add the reply. We have until the end of this month so we're GTG. Just polishing up the response now.

[rasbt]

No worries. Just reviewed the response letter. Thanks for putting that together, I think we are good to go :)

[agitter]

Thanks for getting the extension and working on the revisions and letter.

Does this mean all the reviewer comments have been addressed? If so, we could email all co-authors and give them ~1 week to review the revised version and then resubmit.

[Benjamin-Lee]

I think that the reviewer comments have been adequately addressed. I'll send out an email with the letter and revised copy for their review.

[Benjamin-Lee]

Revision submitted! Thanks again everyone for the help.

[rasbt]

Awesome, thanks for doing this!

[SiminaB]

We got editorial acceptance (in case anyone missed the email on January 4th)!

[rasbt]

Wohoo, congrats everyone! @Benjamin-Lee how are the next steps, do you need help with anything?

[rasbt]

@Benjamin-Lee do you need any help regarding the following?

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

[Benjamin-Lee]

Nope, it’s already been re-submitted! They just wanted me to convert the affiliations to numeric format and add cities/countries for all the authors. I think we’re good to go for now.

On Jan 19, 2022, at 2:50 AM, Sebastian Raschka @.***> wrote:

@Benjamin-Lee https://github.com/Benjamin-Lee need any help regarding

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

— Reply to this email directly, view it on GitHub https://github.com/Benjamin-Lee/deep-rules/issues/367#issuecomment-1016169616, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADH4YHPELUHM2ZT2FIZ6YILUWZUL7ANCNFSM5FELWWIQ. You are receiving this because you were mentioned.

[rasbt]

Awesome, thanks for taking care of it! Glad that went smoothly!