search

jimhester / knitrBootstrap

A framework to create bootstrap styled HTML reports from knitr Rmarkdown.
Other
274 stars 61 forks source link

Pandoc step hanging with knitrBoostrap 1.0.0 (jimhester/knitrBootstrap@76c41f0) #72

Closed lcolladotor closed 9 years ago

lcolladotor commented 10 years ago

Hi,

I'm asking here because you might know what the problem is. I am re-running an analysis I did before and for some reason it's taking much longer: basically pandoc hangs.

In particular, one case used to run in 5 hours and 32 mins (5 hrs 19 mins were spent on R analysis, the rest on the pandoc step). It now has hanged for 5 days (killed it), hanged a 2nd time (killed the process again), and is hanging for a third time (24 hrs already).

Currently, I'm using:

> library(knitrBootstrap)
> library(rmarkdown)
> library(knitcitations)
> devtools::session_info()
Session info-------------------------------------------------------------------
 setting  value
 version  R version 3.1.1 Patched (2014-10-16 r66782)
 system   x86_64, linux-gnu
 ui       X11
 language (EN)
 collate  en_US.UTF-8
 tz       <NA>

Packages-----------------------------------------------------------------------
 package        * version  date       source
 bibtex           0.3.6    2013-07-29 CRAN (R 3.1.1)
 colorout       * 1.0.2    2014-05-30 local
 devtools         1.6.1    2014-10-07 CRAN (R 3.1.1)
 digest           0.6.4    2013-12-03 CRAN (R 3.1.0)
 evaluate         0.5.5    2014-04-29 CRAN (R 3.1.0)
 formatR          1.0      2014-08-25 CRAN (R 3.1.1)
 htmltools        0.2.6    2014-09-08 CRAN (R 3.1.1)
 httr             0.5      2014-09-02 CRAN (R 3.1.1)
 knitcitations  * 1.0.2    2014-10-15 Github (cboettig/knitcitations@c7c5909)
 knitr            1.7      2014-10-13 CRAN (R 3.1.1)
 knitrBootstrap * 1.0.0    2014-10-15 Github (jimhester/knitrBootstrap@76c41f0)
 lubridate        1.3.3    2013-12-31 CRAN (R 3.1.1)
 markdown         0.7.4    2014-08-24 CRAN (R 3.1.1)
 memoise          0.2.1    2014-04-22 CRAN (R 3.1.0)
 plyr             1.8.1    2014-02-26 CRAN (R 3.1.0)
 Rcpp             0.11.3   2014-09-29 CRAN (R 3.1.1)
 RCurl            1.95.4.3 2014-07-29 CRAN (R 3.1.1)
 RefManageR       0.8.34   2014-08-19 CRAN (R 3.1.1)
 RJSONIO          1.3.0    2014-07-28 CRAN (R 3.1.1)
 rmarkdown      * 0.3.3    2014-09-17 CRAN (R 3.1.1)
 rstudioapi       0.1      2014-03-27 CRAN (R 3.1.1)
 stringr          0.6.2    2012-12-06 CRAN (R 3.1.0)
 XML              3.98.1.1 2013-06-20 CRAN (R 3.1.0)
$ pandoc --version
pandoc 1.12.3.1
Compiled with texmath 0.6.6, highlighting-kate 0.5.6.
Syntax highlighting is supported for the following languages:
    actionscript, ada, apache, asn1, asp, awk, bash, bibtex, boo, c, changelog,
    clojure, cmake, coffee, coldfusion, commonlisp, cpp, cs, css, curry, d,
    diff, djangotemplate, doxygen, doxygenlua, dtd, eiffel, email, erlang,
    fortran, fsharp, gnuassembler, go, haskell, haxe, html, ini, java, javadoc,
    javascript, json, jsp, julia, latex, lex, literatecurry, literatehaskell,
    lua, makefile, mandoc, markdown, matlab, maxima, metafont, mips, modelines,
    modula2, modula3, monobasic, nasm, noweb, objectivec, objectivecpp, ocaml,
    octave, pascal, perl, php, pike, postscript, prolog, python, r,
    relaxngcompact, restructuredtext, rhtml, roff, ruby, rust, scala, scheme,
    sci, sed, sgml, sql, sqlmysql, sqlpostgresql, tcl, texinfo, verilog, vhdl,
    xml, xorg, xslt, xul, yacc, yaml
Default user data directory: /home/bst/student/lcollado/.pandoc
Copyright (C) 2006-2013 John MacFarlane
Web:  http://johnmacfarlane.net/pandoc
This is free software; see the source for copying conditions.  There is no
warranty, not even for merchantability or fitness for a particular purpose.

The process has been stuck at:

/jhpce/shared/jhpce/core/JHPCE_tools/1.0/bin/pandoc basicExploration.utf8.md --to html --from markdown-hard_line_breaks+superscript+tex_math_dollars+raw_html+markdown_in_html_blocks-implicit_figures --output basicExploration.html -H /scratch/temp/2732570.1.shared.q/RtmpWjnxOa/knitr_bootstrap_full.html

I suspect that updating pandoc can help, but well, this used to work with the same version. Previously something similar got resolved by fixing a piece of rmarkdown https://github.com/jimhester/knitrBootstrap/issues/47 and I don't know if it has any relation to https://github.com/rstudio/rmarkdown/issues/31.

The previous output of devtools::sessionInfo() is:

##  setting  value                                      
##  version  R version 3.1.1 Patched (2014-10-05 r66720)
##  system   x86_64, linux-gnu                          
##  ui       X11                                        
##  language (EN)                                       
##  collate  en_US.UTF-8                                
##  tz  
##  package                           * version  date      
##  acepack                             1.3.3.3  2013-05-03
##  AnnotationDbi                     * 1.27.19  2014-10-07
##  base64enc                           0.1.2    2014-06-26
##  BatchJobs                           1.4      2014-09-24
##  BBmisc                              1.7      2014-06-21
##  bibtex                              0.3.6    2013-07-29
##  Biobase                           * 2.25.0   2014-04-26
##  BiocGenerics                      * 0.11.5   2014-09-17
##  BiocParallel                        0.99.25  2014-10-03
##  biomaRt                             2.21.5   2014-10-08
##  Biostrings                          2.33.14  2014-09-09
##  biovizBase                        * 1.13.11  2014-09-17
##  bitops                              1.0.6    2013-08-17
##  brew                                1.0.6    2011-04-13
##  BSgenome                            1.33.9   2014-09-13
##  bumphunter                          1.5.5    2014-09-09
##  Cairo                               1.5.6    2014-06-26
##  checkmate                           1.4      2014-09-03
##  cluster                             1.15.3   2014-09-04
##  codetools                           0.2.9    2014-08-21
##  colorspace                          1.2.4    2013-09-30
##  DBI                                 0.3.1    2014-09-24
##  derfinder                         * 0.99.5   2014-10-07
##  derfinderHelper                     0.99.2   2014-10-07
##  derfinderPlot                       0.99.5   2014-10-07
##  devtools                            1.6      2014-09-19
##  dichromat                           2.0.0    2013-01-24
##  digest                              0.6.4    2013-12-03
##  doRNG                               1.6      2014-03-07
##  evaluate                            0.5.5    2014-04-29
##  fail                                1.2      2013-09-19
##  foreach                             1.4.2    2014-04-11
##  foreign                             0.8.61   2014-03-28
##  formatR                             1.0      2014-08-25
##  Formula                             1.1.2    2014-07-13
##  futile.logger                       1.3.7    2014-01-25
##  futile.options                      1.0.0    2010-04-06
##  GenomeInfoDb                      * 1.1.25   2014-10-07
##  GenomicAlignments                   1.1.30   2014-10-07
##  GenomicFeatures                   * 1.17.19  2014-10-07
##  GenomicFiles                        1.1.19   2014-09-02
##  GenomicRanges                     * 1.17.43  2014-10-07
##  GGally                              0.4.8    2014-08-26
##  ggbio                               1.13.13  2014-09-17
##  ggplot2                           * 1.0.0    2014-05-21
##  graph                               1.43.0   2014-04-26
##  gridExtra                         * 0.9.1    2012-08-09
##  gtable                              0.1.2    2012-12-05
##  Hmisc                               3.14.5   2014-09-12
##  htmltools                           0.2.6    2014-09-08
##  httr                                0.5      2014-09-02
##  IRanges                           * 1.99.30  2014-10-07
##  iterators                           1.0.7    2014-04-11
##  knitcitations                       1.0.2    2014-10-07
##  knitr                             * 1.6.18   2014-09-19
##  knitrBootstrap                      0.9.0    2013-10-17
##  labeling                            0.3      2014-08-23
##  lambda.r                            1.1.6    2014-01-23
##  lattice                             0.20.29  2014-04-04
##  latticeExtra                        0.6.26   2013-08-15
##  locfit                              1.5.9.1  2013-04-20
##  lubridate                           1.3.3    2013-12-31
##  markdown                            0.7.4    2014-08-24
##  MASS                                7.3.35   2014-09-30
##  Matrix                              1.1.4    2014-06-15
##  matrixStats                         0.10.0   2014-06-08
##  memoise                             0.2.1    2014-04-22
##  mgcv                              * 1.8.3    2014-08-29
##  munsell                             0.4.2    2013-07-11
##  nlme                              * 3.1.118  2014-10-07
##  nnet                                7.3.8    2014-03-28
##  OrganismDbi                         1.7.5    2014-09-09
##  pkgmaker                            0.22     2014-05-14
##  plyr                                1.8.1    2014-02-26
##  proto                               0.3.10   2012-12-22
##  qvalue                              1.39.1   2014-05-30
##  RBGL                                1.41.1   2014-08-19
##  RColorBrewer                      * 1.0.5    2011-06-17
##  Rcpp                                0.11.3   2014-09-29
##  RCurl                               1.95.4.3 2014-07-29
##  RefManageR                          0.8.34   2014-08-19
##  regionReport                      * 0.99.3   2014-10-07
##  registry                            0.2      2012-01-24
##  reshape                             0.8.5    2014-04-23
##  reshape2                            1.4      2014-04-23
##  RJSONIO                             1.3.0    2014-07-28
##  rmarkdown                           0.3.3    2014-09-19
##  R.methodsS3                         1.6.1    2014-01-05
##  rngtools                            1.2.4    2014-03-06
##  rpart                               4.1.8    2014-03-28
##  Rsamtools                           1.17.34  2014-10-07
##  RSQLite                             0.11.4   2013-05-26
##  rstudioapi                          0.1      2014-03-27
##  rtracklayer                         1.25.19  2014-10-07
##  S4Vectors                         * 0.2.4    2014-09-15
##  scales                              0.2.4    2014-04-22
##  sendmailR                           1.2.1    2014-09-21
##  stringr                             0.6.2    2012-12-06
##  survival                            2.37.7   2014-01-22
##  TxDb.Hsapiens.UCSC.hg19.knownGene * 3.0.0    2014-10-07
##  VariantAnnotation                   1.11.35  2014-10-07
##  XML                                 3.98.1.1 2013-06-20
##  xtable                              1.7.4    2014-09-12
##  XVector                           * 0.5.8    2014-09-07
##  zlibbioc                            1.11.1   2014-04-26
##  source                                      
##  CRAN (R 3.1.1)                              
##  Bioconductor                                
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  Bioconductor                                
##  Bioconductor                                
##  Bioconductor                                
##  Bioconductor                                
##  Bioconductor                                
##  Bioconductor                                
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  Bioconductor                                
##  Bioconductor                                
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  Github (lcolladotor/derfinder@da289c4)      
##  Github (lcolladotor/derfinderHelper@54a9721)
##  Github (lcolladotor/derfinderPlot@08d7f07)  
##  Github (hadley/devtools@45a6ac0)            
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  Bioconductor                                
##  Bioconductor                                
##  Bioconductor                                
##  Bioconductor                                
##  Bioconductor                                
##  CRAN (R 3.1.1)                              
##  Bioconductor                                
##  CRAN (R 3.1.0)                              
##  Bioconductor                                
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  Bioconductor                                
##  CRAN (R 3.1.0)                              
##  Github (cboettig/knitcitations@c7c5909)     
##  local                                       
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  Bioconductor                                
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  Bioconductor                                
##  Bioconductor                                
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.1)                              
##  Github (lcolladotor/regionReport@204bf40)   
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  Github (rstudio/rmarkdown@fd8dafd)          
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  Bioconductor                                
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  Bioconductor                                
##  Bioconductor                                
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  Bioconductor                                
##  Bioconductor                                
##  CRAN (R 3.1.0)                              
##  CRAN (R 3.1.1)                              
##  Bioconductor                                
##  Bioconductor

Notice that rmarkdown GitHub version fd8dafd is the same as version 0.3.3 (https://github.com/rstudio/rmarkdown/commit/fd8dafdea0b4cc68b9dc3da0557a506e60e77ee2). The main difference is the knitrBootstrap version: basically I am re-running the analysis because I want to use the newer knitrBootstrap (not yet using the newest from https://github.com/jimhester/knitrBootstrap/pull/68)

You can find the Rmd file I'm using here which is called from regionReport::derfinderReport() defined here.

Cheers, Leo

jimhester commented 10 years ago

I don't know for sure what the problem is, although if I were to guess I would say that pandoc is incorrectly parsing some portion of the markdown because of a missing newline after one of the html blocks. In order to track down the problem I would inspect the .md file produced and try to produce a minimum working example, your report is too large (and takes too long to run) to figure out what is going wrong in a reasonable time frame.

lcolladotor commented 9 years ago

I could reproduce this until recently when:

My guess is that either of the first 2 things helped. I'll close it for now. Hopefully I won't run into this problem again.

lcolladotor commented 9 years ago

I should add that pandoc step still took 20 mins. But that's reasonable.