Paper examples for discussion

[cgreene]

Share the papers that you've identified as relevant to our reproducibility discussion here. You do not need to note in this issue which ones fall into each category.

[linzho]

http://www.biorxiv.org/content/early/2015/08/28/025767

http://www.cell.com/cell-stem-cell/pdf/S1934-5909(16)30047-9.pdf

http://biorxiv.org/content/biorxiv/early/2017/01/17/100958.full.pdf

[gwaybio]

Find one that you think exemplifies reproducibility and transparency.

Pooled CRISPR screening with single-cell transcriptome readout

• Data is available on GEO
• Exact molecular protocol available in supplement
• Computational analysis pipeline available on Github
  • Although they don't specify a licence :frowning_face:
• Specify package version requirements

I also think this reanalysis of a high profile paper is pretty cool.

Find one that you think exhibits poor reproducibility, transparency, or both.

Xenome -- a tool for classifying reads from xenograft samples

Find one that you are a coauthor on. If you haven't written a paper yet, find one from your current lab.

Comprehensive Cross-Population Analysis of High-Grade Serous Ovarian Cancer Supports No More Than Three Subtypes

[oryoruk]

Good 👍

• Eren et al. 2015
  • Data is available
  • Analysis pipeline is on GitHub
  • Plot generation steps are available on lab website

Bad 👎

• Stam et al. 2009
  • I have reproduced part of the results in this paper.
  • Data was not available, analysis scripts were not available, authors were not helpful over email.
  • Data filtering steps were vague, I approximated/recreated as well as I could.
  • Even some of the publicly available data was hard to found. (Jim Watson's SNPs)
    • I am sure at the time everyone was nuts about this dataset, but it is not publicly available anymore, I got them from a random old source.

Ugly 😄 👌

• Aggarwala et al. 2016
  • not particularly great in reproducibility. I recently recreated some of the fairly straightforward results on this paper with only 0.9993 correlation with my analysis :face_with_thermometer: even though I used the same input files.

[bemert]

http://www.nature.com/nmeth/journal/vaop/ncurrent/full/nmeth.4155.html (http://biorxiv.org/content/early/2016/07/23/065052)

http://www.nature.com/nature/journal/v499/n7459/full/nature12228.html

http://genome.cshlp.org/content/25/10/1558.long

[apexamodi]

Reproduced by Center for Open Science: BET bromodomain inhibition as a therapeutic strategy to target c-Myc doi:10.1016/j.cell.2011.08.017

Unable to be reproduced by Center for Open Science: Melanoma genome sequencing reveals frequent PREX2 mutations doi:10.1038/nature11071

Repression of the Chromatin-Tethering Domain of Murine Leukemia Virus p12. doi:10.1128/JVI.01084-16

[sklasfeld]

Good

Lu, Z., Hofmeister, B. T., Vollmers, C., DuBois, R. M., & Schmitz, R. J. (2016). Combining ATAC-seq with nuclei sorting for discovery of cis-regulatory regions in plant genomes. Nucleic acids research, gkw1179.

• You can access both the processed and raw data from this experiment on GEO. The files are clearly named and contain descriptions that match the paper. While I found that one of the data files may be corrupt I was able to track it back to the author for a possible explanation.
• The methods of this paper extensively explains both the wet lab and dry lab techniques. Unlike other papers that study plants, they listed the exact parameters used for each computational tool and to evaluate the statistics.
• The only thing they are missing is the versions of the software they were using.

Bad

Li, C., Gu, L., Gao, L., Chen, C., Wei, C. Q., Qiu, Q., ... & Chen, C. Y. (2016). Concerted genomic targeting of H3K27 demethylase REF6 and chromatin-remodeling ATPase BRM in Arabidopsis. Nature genetics. BRM-REF6_RELATIONSHIP.pdf

• This paper points to data accessions on NCBI where labels do not match the paper
• In the paper they say they use Bowtie but on the NCBI site they say they use Bowtie2 for mapping
• Their mapping results in the supplemental data does not match the data given on NCBI

Wagner Lab

Winter, CM, Austin RS, Blanvillain-Baufume S, Reback MA, Monniaux M, Wu MF, Sang Y, Yamaguchi A, Yamaguchi N, Parker JE, Parcy F, Jensen ST, Li H, Wagner D. 2011. LEAFY Target Genes Reveal Floral Regulatory Logic, cis Motifs, and a Link to Biotic Stimulus Response, Apr 19. Dev Cell. 20:430-43., Number 4 LEAFY.pdf

• Lists parameters used for computational pipeline
• Supplemental methods are more specific about wet lab methods
• Does not contain versions of tools
• Uses "custom" script and explains what it does, but there is no where to find it

[zz327]

I feel like Yuchao's paper is quite helpful with reproducibility. The code is on github and I myself have used his package many times. https://academic.oup.com/nar/article/43/6/e39/2453417/CODEX-a-normalization-and-copy-number-variation On the other hand I am trying to reproduce some of the result from this paper, but couldn't make it: http://science.sciencemag.org/content/352/6282/189 The gene selection gives me a really hard time.