The future of deep review

[agitter]

We resubmitted version 0.10 to the Journal of the Royal Society Interface and bioRxiv. Thanks everyone for all the great work on the revisions!

I'd like input on where we want to go from here. Should we continue to accept edits to the manuscript even after a version is accepted at a journal? Should we accept only errata corrections but lock the main content?

I don't want to dissolve this amazing group of authors. However, there isn't much precedent for living manuscripts that continue to change after publication, and realistically we are all very busy with other projects. The activity dropped off considerably between the first submission and the reviewer feedback.

[stephenra]

@evancofer @nafizh Great. The priority labels are now all added.

[evancofer]

What can we do to drum up more discussion in the issues? Aggregating new articles on deep learning is essential, but I get the feeling that we will lose momentum without continued and consistent contributions in the form of discussion and writing as well. Perhaps we should set some very easily attainable goals for activity/discussion/editing contributions? I am currently reviewing and drafting some bits on the various genomics sections (e.g. splicing, variant calling, sequencing), but I realize that there are many other sections of the review that may need attention.

[cgreene]

From my experience the path is to start writing + get some small wins in (adjustments to specific sections, etc). Then we can start tweeting about those to build more momentum. If the community is active and the topic remains of interest (as I suspect this one does), I think that's what it'll take. Right now it's unclear if the project is really alive or not, which may make it hard to draw in contributors.

[agitter]

I completely agree with @cgreene. If momentum is restored, it could also help to use issues to recruit contributors to work on specific small sections that need updates. (Though I tried this with #847 and it didn't go anywhere.)

A minor idea would be to rebrand the review. We could adopt the style used in database papers (e.g. The UCSC Genome Browser database: 2018 update) and add : 2019 update or something similar to the title. That may help contributors feel like they are contributing to an active project instead of maintaining something that has already been published.

[cgreene]

Major +1 for adding : 2019 update to the title!

[jmschrei]

I have been following this for a while and I'd like to contribute but I'm not sure what the best way to is. I'm more than happy to link my papers (or those I stumble upon) and discuss them. However, I'm not sure what the goal of discussion in the issues are. Is it to talk about our thoughts / critiques on the methods or to discuss how to best incorporate it into the paper?

[agitter]

@jmschrei both are goals of the issues. @evancofer has some recent examples (e.g. #886) of discussing and critiquing methods. The intent is that this helps us decide what we want to say about a paper if/when we add it to the review.

I'm also proposing that using issues could help restart the writing effort by opening a discussion topic, discussing what should be written, and then making a pull request. For example, there have been several new methods about autoencoders for single-cell RNA-seq data. I could open an issue that notes we only reference two of these in the current review. Then we could re-assesses the state of the subarea with an updated assessment of what has been done well and what challenges remain. Ideally other contributors would help provide relevant papers and form a consensus opinion. We haven't had many issues of this type yet, but I'm hoping it could help re-engage our contributors (past and future).

[evancofer]

@agitter I agree that this is probably the optimal next step.

[baoilleach]

I think you've already made your plans but just checking that you are aware of http://www.livecomsjournal.org/, the Living Journal of Computational Molecular Science (c.f. @davidlmobley).

[agitter]

Thanks for the pointer @baoilleach. We did see the Living Journal of Computational Molecular Science and reference it in our manuscript on the Manubot system for collaborative writing https://greenelab.github.io/meta-review/

We'd be happy to discuss that platform versus Manubot more with you and @davidlmobley, but I suggest taking that conversation to a new issue in https://github.com/greenelab/meta-review

[bachev]

Just in case: the 2019 edition shouldn't be published without mentioning alphafold for structure prediction by Google/Alphabet. They simply crushed everyone else as a first time entrant to the CASP protein structure prediction competition.

[agitter]

@bachev it's definitely relevant. Please open a new issue in this repo if you'd like to discuss AlphaFold. It may be hard for us to write too much about it until they have a complete description of the method. For now, this blog post and comments from Jinbo are the most informative.

[vd4mmind]

Hi all,

A phenomenal read, and thanks for all the amazing work. Just in case if this work is still possible to enrich/enhance? If so, can we include knowledge from some of the publications in Section of Single Cell concerning Deep Learning published in 2019? Some papers I feel I did not see there (in-case can be added) that have already laid some new additional information with DL and single cell:

1. Eraslan, G., Avsec, Ž., Gagneur, J. et al. Deep learning: new computational modelling techniques for genomics. Nat Rev Genet 20, 389–403 (2019) doi:10.1038/s41576-019-0122-6

2. Deep learning does not outperform classical machine learning for cell-type annotation Niklas D. Köhler, Maren Büttner, Fabian J. Theis bioRxiv 653907; doi: https://doi.org/10.1101/653907 ( I feel to an extent anchoring in Seurat via transfer learning did bring in a lot of value, however, this space of annotation has still more scope of development)

3. Eraslan, G., Simon, L.M., Mircea, M. et al. Single-cell RNA-seq denoising using a deep count autoencoder. Nat Commun 10, 390 (2019) doi:10.1038/s41467-018-07931-2

4. Interpretable factor models of single-cell RNA-seq via variational autoencoders Valentine Svensson, Lior Pachter bioRxiv 737601; doi: https://doi.org/10.1101/737601

Some of my personal notes are here but I have not developed them past few months in single-cell space and some mere ML/DL info:

1. Blog link1

2. Blog link 2

3. Blog Link 3

For now, these are the above that came to my mind to enrich the single-cell space in the current review if possible. I did see mention of Autoencoders as well by @agitter . Will be happy to contribute if possible and if all agree to what I have proposed for now. (PS: let me know if the above information fits in the scope here).

Kind regards,

Vivek

[agitter]

Thanks for the suggestions @vd4mmind. Those topics are certainly in scope, and there has been a lot of recent work in the area that is not covered in the current version of the review. However, it's unclear how much actual updating there will be to this review. We've found that we need to have good editors/reviewers lined up if we're going to make major additions or revisions so that pull requests don't languish.

My latest thoughts are that we should drop the "2019 update" part of the title and say something instead about this being the living version or post-publication update. Ideally we would also have a better way to show a rich diff of what has changed since publication (https://github.com/manubot/rootstock/issues/54), a more dynamic way of adding authors frequently (#959), and a preferred way for readers to refer to the specific version they read or cited.

Finally, to help keep track of papers, we've been opening one issue per paper. The format in #940 shows an example with the title in the issue title, abstract quoted, and DOI link.