installPackages not found - namespace issue (R 3.2/devel)

[bryanhanson]

Henrik, I am trying to get my package ChemoSpec running under R 3.2 (the devel version). In my vignette, I have

 %\VignetteDepends{chemometrics, robustbase, RColorBrewer, plyr, pcaPP, mvtnorm, mvoutlier, pls, lattice, grid, rgl, R.utils, mclust, mvbutils, sna, knitr}

And this seems to raise the following issue during R --as-cran CMD build ChemoSpec

* checking for file ‘ChemoSpec/DESCRIPTION’ ... OK
* preparing ‘ChemoSpec’:
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
* creating vignettes ... ERROR
Error : object 'installPackages' not found whilst loading namespace 'R.utils'
Quitting from lines 36-59 (ChemoSpec.Rnw) 
Error: processing vignette 'ChemoSpec.Rnw' failed with diagnostics:
package or namespace load failed for 'ChemoSpec'
Execution halted

I looked over the current .tar.gz for R.utils and it looks correct to me. Do you see any cause for this? Thanks, Bryan

[HenrikBengtsson]

Q1. Is this only happending on R devel, or also R 3.1.2 (patched or not patched)? Q2. Could you point me to the exact revision of ChemoSpec so I can try to reproduce this myself?

[HenrikBengtsson]

BTW, see if you also get the error if you build the vignette in an interactive R session (use a fresh one), e.g.

> tools::buildVignette("ChemoSpec.Rnw", dir="vignettes/")

That way you can also get a traceback() and a sessionInfo().

[bryanhanson]

Using buildVignette I run into a different problem which I think is a real problem with R 3.2, since I've seen it when trying to fix another vignette:

tools::buildVignette("ChemoSpec.Rnw")
Quitting from lines 170-175 (/Users/bryanhanson/Documents/Research/MetabolomicsProjects/ChemometricsStuff/BAHpackages/ChemoSpec/vignettes/ChemoSpec.Rnw) 
Error in stri_split_regex(string, pattern, n = n, simplify = FALSE, opts_regex = attr(pattern,  : 
  Missing closing bracket on a bracket expression. (U_REGEX_MISSING_CLOSE_BRACKET)

So, I'm going to search around to see if anyone else has seen this.

The current version of ChemoSpec can be had at Github I'll try to get to testing under R 3.1.2 a little later tonight. Thanks for your quick response!

[bryanhanson]

The missing bracket issue is a knitr problem and I reported it there.

[bryanhanson]

The problem does not occur with R 3.1.2 and is not related to the knitr problem, which was solved with the devel version of knitr.

[HenrikBengtsson]

MY TROUBLESHOOTING THIS FAR

1. I've checkout the 'devel' branch of ChemoSpec:

git clone git@github.com:bryanhanson/ChemoSpec.git
git checkout devel

2. I've installed all of its dependendant packages.

3. On Windows 7 64-bit, it all works fine:

% R --version
R Under development (unstable) (2015-01-12 r67440) -- "Unsuffered Consequences"
Platform: x86_64-w64-mingw32/x64 (64-bit)
...
% R CMD build ChemoSpec
* checking for file 'ChemoSpec/DESCRIPTION' ... OK
* preparing 'ChemoSpec':
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
* creating vignettes ... OK
* checking for LF line-endings in source and make files
* checking for empty or unneeded directories
* looking to see if a 'data/datalist' file should be added
* building 'ChemoSpec_3.0-0.tar.gz'

PS. Note --as-cran only exists as R CMD check --as-cran, but nowhere else.

4. I next tried on Linux, but I'm stuck trying to install rrcov, because I keep getting:

Error in dyn.load(file, DLLpath = DLLpath, ...) :
  unable to load shared object '/compbio/henrik/R/x86_64-unknown-linux-gnu-library/3.2/rrcov/libs/rrcov.so':
  /compbio/henrik/R/x86_64-unknown-linux-gnu-library/3.2/rrcov/libs/rrcov.so: undefined symbol: dopmtr_
Error : package 'rrcov' could not be loaded
ERROR: lazy loading failed for package 'robCompositions'

I've spent quite a bit of time trying to figure what this problem is about, but I don't have more time to troubleshoot that part now.

QUESTIONS Q1. Is the problem fully reproducible for you, or does it happen randomly or with difference error messages from time to time? Q2. Can you reproduce it using tools::buildVignette("ChemoSpec.Rnw")? FYI, you can build the package without the vignette using R CMD build --no-build-vignettes ChemoSpec.

Your answers could reveal that it has to do with finalizer functions being called by the garbage collector while R.utils or other packages has already been unloaded. Such problems are tricky to troubleshoot and sometimes hard to reproduce, but I've seen them happen before. If we can conclude that's the issue, I'll see if I can add more protection against this.

[HenrikBengtsson]

4. I managed to install 'rrcov' on Linux (using GCC 4.9.2 gfortran didn't work but 4.8.4 did). However, I can still not reproduce your build error using R CMD build ChemoSpec. The vignette is "weaved" successfully (I only get a LaTeX compilation error due to a missing sty file, but that's unrelated to your problem).

[bryanhanson]

Thanks so much Henrik for taking the time to work on this. Much appreciated. I tried R --as-cran CMD build --no-build-vignettes ChemoSpec and there's no problem. Then R --as-cran CMD check --no-vignettes ChemoSpec_3.0-0.tar.gz and the problem appears outside the vignette, which probably narrows things down quite a bit. Specifically:

* checking whether the package can be loaded ... ERROR
Loading this package had a fatal error status code 1
Loading log:
Error : object 'installPackages' not found whilst loading namespace 'R.utils'
Error: package or namespace load failed for ‘ChemoSpec’
Execution halted
Status: 1 ERROR
See
  ‘/Users/bryanhanson/Documents/Research/MetabolomicsProjects/ChemometricsStuff/BAHpackages/ChemoSpec.Rcheck/00check.log’
for details.

The file mentioned contains the same info. I think this answers your Q2: I can build but not check. As to Q1, yes, the message has been consistently the same when I tried to build with the vignette.

[bryanhanson]

So we know the problem is with R 3.2, not 3.1.2, and the problem is Mac specific, and occurs when trying to load/install the package during checking.

[HenrikBengtsson]

Now I've tried R CMD check --as-cran ChemoSpec_3.0-0.tar.gz on Linux (and latest R devel) and it works as expected.

BTW, I'm pretty sure R --as-cran CMD check is incorrect, because you pass --as-cran to R rather than the INSTALL script.

What does

Rscript -e "options('encoding')"

on the machine/setup where you get the error?

On Tue, Jan 13, 2015 at 4:59 PM, Bryan Hanson notifications@github.com wrote:

So we know the problem is with R 3.2, not 3.1.2, and the problem is Mac specific, and occurs when trying to load/install the package during checking.

— Reply to this email directly or view it on GitHub https://github.com/HenrikBengtsson/R.utils/issues/7#issuecomment-69851870 .

[bryanhanson]

Sorry... now as I think about it, yesterday I was able to build and check and it only failed when working on the vignette, but tonight it fails on check, so maybe there is an element of inconsistency?

[bryanhanson]

Alright, sorry I missed your comment about the position of --as-cran I've been doing that way for a long time and it doesn't cause an error. Doing it this way R CMD check --as-cran --no-vignettes ChemoSpec_3.0-0.tar.gz gives:

* checking whether package ‘ChemoSpec’ can be installed ... ERROR
Installation failed.
See ‘/Users/bryanhanson/Documents/Research/MetabolomicsProjects/ChemometricsStuff/BAHpackages/ChemoSpec.Rcheck/00install.out’ for details.
Status: 1 ERROR, 2 NOTEs
See
  ‘/Users/bryanhanson/Documents/Research/MetabolomicsProjects/ChemometricsStuff/BAHpackages/ChemoSpec.Rcheck/00check.log’
for details.

Looking at that file:

* installing *source* package ‘ChemoSpec’ ...
** R
** data
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error : object 'installPackages' not found whilst loading namespace 'R.utils'
Error: loading failed
Execution halted
ERROR: loading failed

Which is the same info one gets with the --as-cran in the other position. And here's the other item you asked about:

 Rscript -e "options('encoding')"
$encoding
[1] "native.enc"

[bryanhanson]

Sent you a small package that gives the same error, to henrikb@braju.com

[HenrikBengtsson]

Great that you've found what looks like a minimal example. Now we can rule out confounding factors from other packages.

I took the freedom to create a (temporary) repository containing the miniPac package you've sent me. Let's work with that one to narrow this problem.

It's really odd that you get this given that R.utils is used in so many places. My bets are now definitely on "stray" finalizers trying to access R.utils after it's been unloaded.

Q. If you R CMD build miniPac and then R CMD INSTALL --no-test-load miniPac_1.0.tar.gz, can you then load it in R as:

> library("miniPac")

Q. What is your sessionInfo() after loading the package. If you can't load it, what is before?

---

PS. Sorry, my comment on --as-cran was just FYI; I didn't think it made a difference for this problem. Compare the output of:

R --help CMD check
R CMD check --help

and you'll see my point.

[bryanhanson]

After building and loading the package as you asked, I get:

R > sessionInfo()
R Under development (unstable) (2015-01-11 r67421)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.10.1 (Yosemite)

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] miniPac_1.0 sos_1.3-8   brew_1.0-6 

loaded via a namespace (and not attached):
[1] R.methodsS3_1.6.1 R.oo_1.18.0       R.utils_1.34.0   

My .Rprofile loads sos.

[HenrikBengtsson]

Q. To rule out everything else, does the error occur also when you don't attach sos? Q. To be 100% sure, I think you can also do:

R --vanilla CMD check miniPac_1.0.tar.gz

[bryanhanson]

Hmm... getting a bit stranger. R --vanilla CMD build miniPac goes OK, but R --vanilla CMD INSTALL --no-test-load miniPac_1.0.tar.gz gives the error:

 > R --vanilla CMD INSTALL --no-test-load miniPac_1.0.tar.gz
* installing to library ‘/Users/bryanhanson/Library/R/3.2/library’
* installing *source* package ‘miniPac’ ...
** R
** preparing package for lazy loading
Error : object 'installPackages' not found whilst loading namespace 'R.utils'
ERROR: lazy loading failed for package ‘miniPac’
* removing ‘/Users/bryanhanson/Library/R/3.2/library/miniPac’
* restoring previous ‘/Users/bryanhanson/Library/R/3.2/library/miniPac’

and I double-checked, leaving off vanilla it works! And, R --vanilla CMD check miniPac_1.0.tar.gz gives the error, identical to the one above.

NOTE: just checked, sos brings in `brew. The above results suggests not having these loaded causes the problem???

[HenrikBengtsson]

Just a FYI, I've now also checked on the below system and cannot reproduce it there either:

> sessionInfo()
R Under development (unstable) (2015-01-13 r67443)
Platform: x86_64-apple-darwin10.8.0 (64-bit)
Running under: OS X 10.6.8 (Snow Leopard)

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] miniPac_1.0

loaded via a namespace (and not attached):
[1] R.methodsS3_1.6.2 R.oo_1.18.2       R.utils_1.34.0

PS. I'm getting the OS X R binaries from http://r.research.att.com/.

[HenrikBengtsson]

Now to a silly but important suggestion, have you tried to reinstall the R.utils package?

> remove.packages("R.utils")
> system.file(package="R.utils")
[1] ""
> quit("no")

Then reinstall it and try again.

[bryanhanson]

Not silly, it gives a clue (by the way, today I could get a binary off of CRAN, two days ago I had to build from sources for R 3.2):

R > install.packages("R.utils")
Installing package into ‘/Users/bryanhanson/Library/R/3.2/library’
(as ‘lib’ is unspecified)
installing the source package ‘R.utils’

trying URL 'http://cran.at.r-project.org/src/contrib/R.utils_1.34.0.tar.gz'
Content type 'application/x-gzip' length 354260 bytes (345 KB)
opened URL
==================================================
downloaded 345 KB

********
********  Loading Bryan's .Rprofile
********
Loading required package: brew
********
********  Completed Bryan's .Rprofile GET TO WORK!
********
* installing *source* package ‘R.utils’ ...
** package ‘R.utils’ successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
Warning in setGenericS3.default(name, export = exportGeneric, envir = envir,  :
  Renamed the preexisting function warnings to warnings.default, which was defined in environment base.
Warning in setGenericS3.default(name, export = exportGeneric, envir = envir,  :
  Could not create generic function. There is already a non-generic function named installPackages() in package base with the same name as an existing default function installPackages.default() in packagebase.
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
********
********  Loading Bryan's .Rprofile
********
Loading required package: brew
********
********  Completed Bryan's .Rprofile GET TO WORK!
********
* DONE (R.utils)

[HenrikBengtsson]

Ah...

Could not create generic function. There is already a non-generic function named installPackages() in package base with the same name as an existing default function installPackages.default() in package base.

Now, the message is slightly incorrect when it claims that there exists a installPackages() in the base package. However, there is one in the sos package, i.e.

> args(sos::installPackages)
function (x, minCount = sqrt(x[1, "Count"]), ...)
NULL

Looks familiar? ;) I think the less here is to avoid loading packages in .Rprofile as far as ever possible.

Personally, I think this is an issue of R itself, almost to the point where Iconsidered it a bug. I, Hadley Wickham and others have raised this concern previously, e.g.

• https://stat.ethz.ch/pipermail/r-devel/2012-February/063365.html
• https://stat.ethz.ch/pipermail/r-devel/2012-July/064427.html
• https://stat.ethz.ch/pipermail/r-devel/2013-January/065649.html

So, remove R.utils again. Remove all your library() calls in ~/.Rprofile and restart. I'm willing to bet this resolves the issue.

[bryanhanson]

Ugh... Yes, removing sos from .Rprofile uninstalling and re-installing, then going back to build and check (on ChemoSpec for good measure) w/ and w/o vignette, it all works. I'll read those Rd e-mails tomorrow (a bit tired now). Interesting that this just showed up, interesting too that sos has an installPackages at all. Anyway, thanks for your attention to detail and sorry to have wasted your time...

[HenrikBengtsson]

Great that it works now. No worries; pay it forward. Good night.

[HenrikBengtsson]

For the record, I can reproduce the R.utils installation warnings on the installPackages() clash, i.e.

Warning in setGenericS3.default(name, export = exportGeneric, envir = envir,  :
  Could not create generic function. There is already a non-generic function named installPackages() in package base with the same name as an existing default function installPackages.default() in packagebase.

However, this eventually also causes the package not to be installed at all, i.e.

** testing if installed package can be loaded
Error : object 'installPackages' not found whilst loading namespace 'R.utils'
Error: loading failed
Execution halted
ERROR: loading failed
* removing '/Users/hb/Library/R/3.1/library/R.utils'

EXAMPLE:

$ rm .Rprofile
$ R
> options(repos="http://cran.rstudio.com")
> install.packages("sos")
> cat("library(sos)\n", file=".Rprofile")
> install.packages("R.utils", type="source")
Installing package into '/Users/hb/Library/R/3.1/library'
(as 'lib' is unspecified)
trying URL 'http://cran.rstudio.com/src/contrib/R.utils_1.34.0.tar.gz'
Content type 'application/x-gzip' length 354260 bytes (345 KB)
opened URL
==================================================
downloaded 345 KB

Loading required package: brew
* installing *source* package 'R.utils' ...
** package 'R.utils' successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
Warning in setGenericS3.default(name, export = exportGeneric, envir = envir,  :
  Renamed the preexisting function warnings to warnings.default, which was defined in environment base.
Warning in setGenericS3.default(name, export = exportGeneric, envir = envir,  :
  Could not create generic function. There is already a non-generic function named installPackages() in package base with the same name as an existing default function installPackages.default() in packagebase.
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
Error : object 'installPackages' not found whilst loading namespace 'R.utils'
Error: loading failed
Execution halted
ERROR: loading failed
* removing '/Users/hb/Library/R/3.1/library/R.utils'

The downloaded source packages are in
        '/private/tmp/Rtmpx2KqHs/downloaded_packages'
Warning message:
In install.packages("R.utils", type = "source") :
  installation of package 'R.utils' had non-zero exit status
> library(R.utils)
Error in library(R.utils) : there is no package called 'R.utils'

I've tried this on R patched on OS X, and R devel on Linux and Windows.

[bryanhanson]

Just used sos to determine that there is an installPackages in JGR as well. People probably wouldn't run into that as often as the combination of sos and R.utils.