Packages failing to build on moscato2

[b-long]

As per @dtenenba 's email :

...on moscato2 as pkgbuild, went to e:\packagebuilder\workers\jobs\kimod_20151211194650

Went to R-libs and set R_LIBS_USER to the current directory. Went back up one level and ran

E:\packagebuilder\workers\jobs\kimod_20151211194650>\packagebuilder\r\bin\R CMD build kimod

and it worked fine.

But it doesn't like to run in the context of the spb, programmatically, it gives (as can be seen at > http://bioconductor.org/spb_reports/kimod_0.99.3_buildreport_20151211175404.html#moscato2_buildsrc_anchor):

* installing the package to build vignettes
* creating vignettes ...Warning: running command '"E:/packagebuilder/R/bin/x64/Rscript" --vanilla --default-packages= -e "tools::buildVignettes(dir = '.', tangle = TRUE)"' had status 1
 ERROR
Warning:The calculations were performed using the scalar product between the tables
Warning in DiStatis(NCI60Selec_ESet) :
  The Projection of all variables will be made when the number of variables is less than 50
Warning: running command '"C:\progra~2\MiKTeX~1.9\miktex\bin\texify.exe" --quiet --pdf "kimod-vignette.tex" --max-iterations=20 -I "E:/packagebuilder/R/share/texmf/tex/latex" -I "E:/packagebuilder/R/share/texmf/bibtex/bst"' had status 255
Error in find_vignette_product(name, by = "texi2pdf", engine = engine) :
  Failed to locate the 'texi2pdf' output file (by engine 'utils::Sweave') for vignette with name 'kimod-vignette'. The following files exist in directory '.': 'Rplots.pdf', 'authblk.sty', 'cites.bib', 'kimod-vignette-011.pdf', 'kimod-vignette-015.pdf', 'kimod-vignette-017.pdf', 'kimod-vignette-019.pdf', 'kimod-vignette-020.pdf', 'kimod-vignette-concordance.tex', 'kimod-vignette.Rnw', 'kimod-vignette.tex'
Calls: <Anonymous> -> find_vignette_product
Execution halted

[b-long]

I'm trying to understand what's going on here still. @dtenenba @jimhester Mind reviewing this one change in case there was a "gotcha" ? https://github.com/Bioconductor/packagebuilder/commit/008b17d19e1854b4009d16fd924b6d2048781810

We were setting win_multiarch to True in more than one place, and then using it in various conditionals. I removed code that effectively did nothing, but wouldn't mind a second set of eyes.

[b-long]

I've just run a build of cellTree again, using a test issue on the tracker. It's failed, with a similar error to Friday evening (referring to find_vignette_product). Here's the full build report, with a relevant snippet below:

* checking for file 'cellTree/DESCRIPTION' ... OK
* preparing 'cellTree':
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
* creating vignettes ...Warning: running command '"E:/packagebuilder/R/bin/x64/Rscript" --vanilla --default-packages= -e "tools::buildVignettes(dir = '.', tangle = TRUE)"' had status 1
 ERROR
Warning: running command 'kpsewhich framed.sty' had status 255
Warning in test_latex_pkg("framed", system.file("misc", "framed.sty", package = "knitr")) :
  unable to find LaTeX package 'framed'; will use a copy from knitr
Loading required package: topGO
Loading required package: BiocGenerics
Loading required package: parallel

Attaching package: 'BiocGenerics'

The following objects are masked from 'package:parallel':

    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
    clusterExport, clusterMap, parApply, parCapply, parLapply,
    parLapplyLB, parRapply, parSapply, parSapplyLB

The following objects are masked from 'package:stats':

    IQR, mad, xtabs

The following objects are masked from 'package:base':

    Filter, Find, Map, Position, Reduce, anyDuplicated, append,
    as.data.frame, as.vector, cbind, colnames, do.call, duplicated,
    eval, evalq, get, grep, grepl, intersect, is.unsorted, lapply,
    lengths, mapply, match, mget, order, paste, pmax, pmax.int, pmin,
    pmin.int, rank, rbind, rownames, sapply, setdiff, sort, table,
    tapply, union, unique, unlist, unsplit

Loading required package: graph
Loading required package: Biobase
Welcome to Bioconductor

    Vignettes contain introductory material; view with
    'browseVignettes()'. To cite Bioconductor, see
    'citation("Biobase")', and for packages 'citation("pkgname")'.

Loading required package: GO.db
Loading required package: AnnotationDbi
Loading required package: stats4
Loading required package: IRanges
Loading required package: S4Vectors

Attaching package: 'S4Vectors'

The following object is masked from 'package:base':

    expand.grid

Loading required package: DBI

Loading required package: SparseM

Attaching package: 'SparseM'

The following object is masked from 'package:base':

    backsolve

Attaching package: 'topGO'

The following object is masked from 'package:IRanges':

    members

Loading required namespace: maptpx

Warning: running command '"C:\progra~2\MiKTeX~1.9\miktex\bin\texify.exe" --quiet --pdf "cellTree-vignette.tex" --max-iterations=20 -I "E:/packagebuilder/R/share/texmf/tex/latex" -I "E:/packagebuilder/R/share/texmf/bibtex/bst"' had status 255
Error in find_vignette_product(name, by = "texi2pdf", engine = engine) : 
  Failed to locate the 'texi2pdf' output file (by engine 'knitr::knitr') for vignette with name 'cellTree-vignette'. The following files exist in directory '.': 'cellTree-vignette.R', 'cellTree-vignette.Rnw', 'cellTree-vignette.tex', 'framed.sty'
Calls: <Anonymous> -> find_vignette_product
Execution halted

I'm still investigating, but if anyone has insight, please share. I'm wondering if the output above is all expected, or if perhaps some portion of it hints at the underlying problem (e.g. unable to find LaTeX package 'framed'; will use a copy from knitr ). I'll continue to report here with findings.

cc/ @jimhester @dtenenba @mtmorgan

[jimhester]

What is the error output from

"C:\progra~2\MiKTeX~1.9\miktex\bin\texify.exe" --quiet --pdf "cellTree-vignette.tex" --max-iterations=20 -I "E:/packagebuilder/R/share/texmf/tex/latex" -I "E:/packagebuilder/R/share/texmf/bibtex/bst"

That will probably tell you what is going wrong.

[dtenenba]

It'll be hard to track that down since the problem only occurs in the context of the spb; if I were to log in and run the command above, presumably it would not cause an error. @mtmorgan Do you remember the outcome the last time we looked at this?

[b-long]

There isn't any "cellTree-vignette.tex" file in "C:\Users\pkgbuild\AppData\Local" or "C:\Program Files (x86)\MiKTeX 2.9" . However, Googling "texify.exe tmp directory", I came upon this :

http://tex.stackexchange.com/questions/51799/windows-api-error-5-access-is-denied-when-trying-to-compile-tikz-picture

Which seems to have a few interesting points among the discussion:

1. "You have to give yourself the permissions to read and write in C:\Program Files (x86)\MiKTeX 2.9"
2. "make sure that your TMP environment variable points to a writable source"

As the Administrator, I can see that TEMP and TMP are set to %USERPROFILE%\AppData\Local\Temp\. I'm not sure if I can bring up the "Environment Variables" screen as the pkgbuild user, but I'll try.

Interestingly, there's a "3" directory appended to the end when I print these as pkgbuild :

C:\Windows\system32>echo %TEMP%
C:\Users\pkgbuild\AppData\Local\Temp\3

C:\Windows\system32>echo %TMP%
C:\Users\pkgbuild\AppData\Local\Temp\3

[b-long]

Alright, I see in the the "Environment Variables" screen (as the pkgbuild user) TEMP and TMP are set to %USERPROFILE%\AppData\Local\Temp\ . The 3 is apparently part of an automatic process which updates TMP, I doubt it's related.

[b-long]

I've yet to make much progress in understanding this, it looks like it could be similar to this r-devel mailing list post from April 2013 .

Henrik's earlier post had me focusing on the texi2pdf portion of the logic :

Immediately after calling tools::texi2pdf("OrganismDbi.tex"), the code tries to locate the texi2pdf output file, that is, 'OrganismDbi.pdf', which it cannot find. This indicates that tools::texi2pdf() gave an error.

I went about researching, trying to focus on that section and found this item on the bioc-devel mailing list from October 2014. In that mail, there's quoted text from Sander Bollen (I can't find the original mail), which contains this :

I have had this issue as well when submitting my package. In my case it turned out that the Windows build apparently can't handle bibtex/biblatex very well. Looks like your build also stops at bibtex. If you have just a few references, you might just put them in the old-fashioned way (i.e. by hand).

I took a look at the cellTree source with grep, and it seems that this package does in fact use BibTeX. Perhaps this is our problem? Here's content from cellTree :

blong@work:~/Downloads/cellTree$ grep -r bib *
inst/doc/cellTree-vignette.Rnw:\begin{thebibliography}{1}
inst/doc/cellTree-vignette.Rnw:\bibitem{blei2003latent}
inst/doc/cellTree-vignette.Rnw:\bibitem{hornik2011topicmodels}
inst/doc/cellTree-vignette.Rnw:\bibitem{taddy2011estimation}
inst/doc/cellTree-vignette.Rnw:\bibitem{trapnell2014pseudo}
inst/doc/cellTree-vignette.Rnw:\bibitem{ashburner2000gene}
inst/doc/cellTree-vignette.Rnw:\end{thebibliography}
inst/doc/cellTree-vignette.Rnw:% \bibliography{cellTree/REFERENCES}
REFERENCES.bib:%% This BibTeX bibliography file was created using BibDesk.
REFERENCES.bib:%% http://bibdesk.sourceforge.net/
vignettes/cellTree-vignette.Rnw:\begin{thebibliography}{1}
vignettes/cellTree-vignette.Rnw:\bibitem{blei2003latent}
vignettes/cellTree-vignette.Rnw:\bibitem{hornik2011topicmodels}
vignettes/cellTree-vignette.Rnw:\bibitem{taddy2011estimation}
vignettes/cellTree-vignette.Rnw:\bibitem{trapnell2014pseudo}
vignettes/cellTree-vignette.Rnw:\bibitem{ashburner2000gene}
vignettes/cellTree-vignette.Rnw:\end{thebibliography}
vignettes/cellTree-vignette.Rnw:% \bibliography{cellTree/REFERENCES}

[b-long]

@dtenenba is able to build cellTree successfully when invoking R CMD build manually on moscato2. As such, we need to keep investigating. I've made a small change to get more debug information (https://github.com/Bioconductor/packagebuilder/commit/f3e1a2f5e7639ef6f03afea4fc363dfd505e24d4). The change has given us the current sessionInfo() after installPkgDeps.R is run is :

zin2

> sessionInfo()
R Under development (unstable) (2015-11-08 r69614)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.3 LTS

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] BiocInstaller_1.21.2

loaded via a namespace (and not attached):
 [1] httr_1.0.0     R6_2.1.1       magrittr_1.5   tools_3.3.0    curl_0.9.4    
 [6] memoise_0.2.1  stringi_1.0-1  knitr_1.11     stringr_1.0.0  digest_0.6.8  
[11] devtools_1.9.1

moscato2:

> sessionInfo()
R Under development (unstable) (2015-11-08 r69614)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows Server 2008 R2 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] BiocInstaller_1.21.2

loaded via a namespace (and not attached):
 [1] httr_1.0.0     R6_2.1.1       magrittr_1.5   tools_3.3.0    curl_0.9.4
 [6] memoise_0.2.1  stringi_1.0-1  knitr_1.11     stringr_1.0.0  digest_0.6.8
[11] devtools_1.9.1
>

[b-long]

As Martin points out, we should not only attempt to build on moscato2 by hand (R CMD build...), but we should also ensure that all SPB environment variables are set (manually), just as they are within the context of SPB running.

[b-long]

Fixed via https://trello.com/c/CTaaRiWu/42-moscato2-spb-triage .