Reviews are in!

[cgreene]

We now have reviews from the journal. Below is the text of the reviews.

In terms of other elements:

• We have until January 15 to get a revised manuscript back to the journal.

• We need to review, revise, and merge or close each of the pull requests open from the community. We will need help with this from current authors: https://github.com/greenelab/deep-review/pulls

  • [x] #532
  • [x] #559
  • [x] #566
  • [x] #577
  • [x] #629
  • [x] #632
  • [x] #636
  • [x] #638
  • [x] #648

• We will stop accepting new contributions from those who are not authors of the manuscript or authors of any currently open pull requests as of the end of the day on Friday. This is for the purpose of getting the revised manuscript back by January 15.

I created sub-issues to track our responses to each individual reviewer comment.

• [x] Ref 1.1 #683
• [x] Ref 1.2 #684
• [x] Ref 1.3 #685
• [x] Ref 1.4 #686
• [x] Ref 1.5 #687
• [x] Ref 2.1 #688
• [x] Ref 2.2 #689
• [x] Ref 2.3 #690

When you address a review in a pull request, please also update response-to-reviewers.md.

---

Referee: 1

Comments to the Author The authors discussed opportunities and obstacles for applying deep learning to tackle biomedical problems. The reviewer thinks that this note would be a valuable contribution to the journal, since it provides with a coherent and comprehensive coverage of this interesting research field. However, the reviewer has the following concerns/suggestions.

The authors summarized over 400 literature references purely by narratives. The authors provided synopsis for each important reference, but lacks synthesis of related work. It would be better to synthesize related work into a table and analyze their characteristics.

The authors discussed deep learning models such as sDA, CNN, RNN etc. It would be better to have a figure illustrating their architectures. This way, the reader will have a concrete visualization that will aid the understanding of the discussion points in the manuscript.

The authors gave a case study of LADA to suggest that integrating multiple data sources may lead to breakthrough medical discoveries. However, it is unclear from the authors’ description that deep why learning models possess such integrating capability. In fact, the tensor model seems to be the widely acknowledged model that can easily integrating multiple data sources. The authors mentioned “One source of training examples with rich clinical annotations is electronic health records”. What does it mean by “rich clinical annotations”? Can the authors provide a definition and a few examples.

Some existing biomedical informatics systems are not cited. For example, please provide a citation to NegBio.

---

Referee: 2

Comments to the Author This is a very timely review of recent progress in applying machine learning or more precisely deep learning approaches to medical and biological data. Despite the relative recent surge in the application of deep learning algorithms in these areas the long list of literature references is a testament of the rapid growth of this field. Overall I found the review very well written and extremely interesting to read. It provides a very useful overview and a rich source of references to recent attempts of leveraging the power of deep learning algorithms for biomedical data analysis. All sections are highly informative, but I found the sections dealing with the problems around obstacles to data sharing, privacy issues, and data quality, as well as the sections on the difficulty of meeting the high standards of decision making in a clinical setting particularly thought stimulating.

Any review that tries to tackle such a complex and broad subject will have some shortcomings. The review in its current form represents certainly already a valuable resource and thought stimulating reading and would, in my view, be satisfactory and acceptable for publication. However, I want to point out a few suggestions that could make the review even more useful.

1) There is little explanation of key deep learning concepts: layers, autoencoders, RNNs, etc. It might be impossible to do that within the space limitations of the current review, I wonder whether a link to a dedicated website or supplementary material where the most often used deep learning concepts are explained in a way an uninitiated reader can quickly read through and understand would be a solution. There is certainly a large readership whose interest has been peaked by countless references to deep learning even in the popular press, but who are very confused when autoencoders, LSTMs and RNNs get thrown at them without any even brief explanation what they are. Just sending readers off to fend for themselves through internet searches or to study the excellent but still quite technical book [10] Goodfellow et al. is probably not the most satisfactory answer.

2) There is a slight imbalance in the presentation of various application areas. The section on drug development (p40ff), for example, is quite extensive and provides a lot of technical details which might be less relevant for a reader who tries to get a general overview of deep learning in biomedical research. An area which is little mentioned on the other hand are deep learning approaches to brain data, eg connectivity maps, and the area of learning from structured data, such as graphs.

3) The main issue with machine learning solutions in a medical, particularly clinical or public health setting is the lack of proper measures of uncertainty, as it is traditionally provided either in the framework of hypothesis testing or in the increasing acceptance of posterior Bayesian inference for public health decisions. Although this is mentioned throughout the review, this issue deserves a much more prominent place in the introduction and the discusssion, since it is one of the key obstacles for the acceptance of machine learning approaches outside exploratory analyses in basic biological research.

[cgreene]

@agitter : how do we want to parcel out the tasks related to revision and review?

[yfpeng]

I will address this for sure: "For example, please provide a citation to NegBio."

I can also help visualize deep learning models. I propose to put them in "supplementary material"

[SiminaB]

For comment 1 from reviewer 2, the table I created in https://github.com/greenelab/deep-review/pull/566 may help. I suggest that as our most urgent priority 😄

[michaelmhoffman]

If there are new figures added, better to have them in the actual manuscript to break up the text a bit. I don't understand why we would want to confine them to supplementary material.

[agitter]

@cgreene we could make individual issues for each of the specific requests from the reviewers and label them with a new revisions label. Then we can assign whoever volunteers to work on that issue.

@dhimmel we'll want to write a response to reviewers as well. Should we configure manubot to build two separate documents or is there a better way to proceed?

If there are new figures added,...

I agree, we should break up the wall of text when adding figures. We have some existing issues about adding figures that we can dig up and work from (e.g. #354 and #630). We can also work on merging @SiminaB's table.

[dhimmel]

we'll want to write a response to reviewers as well. Should we configure manubot to build two separate documents or is there a better way to proceed?

That may be overkill. If the response doesn't need a bibliography (we can still hyperlink text to sources), then I'd suggest we just make a markdown document like response-to-reviewers.md in this repository. Since the response to reviewers will likely just need to get converted to a PDF once, there's no need for automated builds beyond the GitHub markdown rendering.

[dhimmel]

@agitter I realize it was a large mistake to not update the repo the the latest manubot-rootstock right after the first submission when no PRs were open. While it could cause some (hopefully minor) hassles for the 10 open PRs, perhaps I should open a PR to upgrade now, before we do more revisions. This way the revisions would benefit from the improved design. What do you think?

[michaelmhoffman]

Should we continue to add issues for new studies?

[alxndrkalinin]

We started briefly discussing it, but didn't come to any decision yet. Would be nice to keep expanding, but add some structure (tags?).

[agitter]

@dhimmel what parts of manubot-rootstock would you like to merge? I agree that the current version is much better than this early version we used for deep review, but there will be some major incompatibilities. Things like the author, institution, and funding parsing and templating come to mind.

[agitter]

@michaelmhoffman @alxndrkalinin I plan to keep adding new issues for papers until we come up with a better plan for logging and discussing papers in the longer term. In the short term, we also may want to add a few new papers to the review during revision.

[dhimmel]

what parts of manubot-rootstock would you like to merge?

Everything.

there will be some major incompatibilities. Things like the author, institution, and funding parsing and templating come to mind.

Yes that will have to be reengineered under the updated system. Although in the longterm this is inevitable... deep review is already too far removed from manubot-rootstock and therefore hasn't been receiving any upstream manubot updates. I'm happy to help rework the aspects deep-review codebase that're incompatible.

[agitter]

@dhimmel maybe we should open a new issue to preview and discuss how much would be involved in the merge. I'm hesitant to create a lot of new work when the current version of deep review is running smoothly even if it lacks some (important) features.

[cgreene]

@agitter : I agree with you that a new issue for each reviewer point is probably the most convenient way to address things.

[cgreene]

I created sub issues. See the checkboxes at the top. Please sign up for the ones you are interested in. 👍

[dhimmel]

Before starting work on revisions, we should get #681 merged, which does major updates to the automation system. Please make new commits off of the latest master branch after #681 is merged to avoid any incompatibilities.

[agitter]

Because we have so many issues for paper discussion, I created a milestone with the issues and pull requests that we need to resolve before resubmitting.

[agitter]

We're getting close to the journal's deadline for revisions, January 15. The deadline was already extended once, so we'd like to hit this one. That means we only have about a week to make and review pull requests and write the responses to reviewers.

Please check the milestone I created for pointers to pull requests that need review and reviewer comments we need to address. It would be great to assign one or more of us to each referee comment so that we know which ones will be covered and where we need help.

[swamidass]

What journal was this submitted to (sorry I'm just getting back up to speed).

[agitter]

@swamidass Journal of the Royal Society Interface

[agitter]

Closed by #799