R package pkgbuild crashes with *** buffer overflow detected ***

[jwnki]

Describe the bug This is a known issue which was noticed first by rstan users. It seems that when $R_LIBS_SITE is too long calling pkgbuild::check_build_tools(debug = TRUE) results in:

> pkgbuild::check_build_tools(debug = TRUE)
Trying to compile a simple C file
Running /nix/store/sbchp611dkfbfcrgbm2wh7hinrk4yy2w-R-4.0.4/lib/R/bin/R CMD \
  SHLIB foo.c
*** buffer overflow detected ***: terminated
/nix/store/sbchp611dkfbfcrgbm2wh7hinrk4yy2w-R-4.0.4/lib/R/bin/SHLIB: line 4:  5891 Done                    echo 'tools:::.SHLIB()'
      5892 Aborted                 (core dumped) | R_DEFAULT_PACKAGES=NULL "${R_HOME}/bin/R" --no-restore --no-echo --no-site-file --no-init-file --args $@
Error: Could not find tools necessary to compile a package
Call `pkgbuild::check_build_tools(debug = TRUE)` to diagnose the problem.

pkgbuild maintainer(s) are aware of the issue but seem to think it should be dealt with by nixpkgs. (I don't know, I couldn't find any specification concerning the length of $R_LIBS_SITE but it's probably fair to assume that it can't be arbitrarily long).

I am interested in this because of the brms package which depends on rstan. Until now the solution proposed by mollermara here worked but it does not work on unstable. I believe this is because the brms from unstable populates $R_LIBS_SITE with around 160 paths. This is also the easiest way to reproduce the bug.

This doesn't affect RStudio.

To Reproduce Steps to reproduce the behavior:

1. nix-shell -p 'rWrapper.override{ packages = [ rPackages.brms ]; }' --pure -I nixpkgs=channel:nixpkgs-unstable
2. Rscript -e 'pkgbuild::check_build_tools(debug = TRUE)'

Expected behavior Running Rscript -e 'pkgbuild::check_build_tools(debug = TRUE)' should return:

Trying to compile a simple C file
Running /nix/store/sbchp611dkfbfcrgbm2wh7hinrk4yy2w-R-4.0.4/lib/R/bin/R CMD \
  SHLIB foo.c
/nix/store/zzvq5qwlm2xikawfqxb0q8gl2bw391a9-gcc-wrapper-10.2.0/bin/cc -I"/nix/store/sbchp611dkfbfcrgbm2wh7hinrk4yy2w-R-4.0.4/lib/R/include" -DNDEBUG      -fpic  -g -O2  -c foo.c -o foo.o
/nix/store/zzvq5qwlm2xikawfqxb0q8gl2bw391a9-gcc-wrapper-10.2.0/bin/cc -shared -L/nix/store/sbchp611dkfbfcrgbm2wh7hinrk4yy2w-R-4.0.4/lib/R/lib -o foo.so foo.o -L/nix/store/sbchp611dkfbfcrgbm2wh7hinrk4yy2w-R-4.0.4/lib/R/lib -lR

Your system is ready to build packages!

Notify maintainers rWrapper maintainers: @peti @timokau @omasanori @7c6f434c @collares

Metadata

> nix-shell -p nix-info --run "nix-info -m"                                        
 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.114, NixOS, 20.09.4038.d90df566caf (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.10`
 - channels(root): `"nixos-20.09.4038.d90df566caf"`
 - channels(jas): `"nixpkgs-unstable-21.05pre289180.1cdf5f44364"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: 
  - rWrapper
# a list of nixos modules affected by the problem
module:

[jwnki]

UPDATE 2021-06-14: 21.05 breaks this for some reason. If you find this and don't know what to do I suggest using R-4.0.2 revision (find it here: lazamar.co.uk). I'll post another update if I find out what's going on (but please comment if you know, maybe it's trivial).

EDIT this works, I don't know why I thought it doesn't. The brms not working is, I believe, related to some problems in rstan.

~I tried accommodating RStudio wrapper and it partially works, but only when recommended packages is set to empty list. As if there was a number of packages after which pkgbuild can't handle them anymore. Unfortunately brms still doesn't work but now I can't tell whether this is because of the wrapper~

Solution (uses overlay proposed here and RStudio wrapper from nixpkgs).

r-wrapper2.nix:

{ runCommand, R, makeWrapper, recommendedPackages, packages }:

runCommand (R.name + "-wrapper") {
  preferLocalBuild = true;
  allowSubstitutes = false;

  buildInputs = [R] ++ recommendedPackages ++ packages;

  nativeBuildInputs = [makeWrapper];
  fixLibsR = "fix_libs.R";
  # Make the list of recommended R packages accessible to other packages such as rpy2
  # (Same as in the original rWrapper)
  passthru = { inherit recommendedPackages; };
}
''
mkdir $out

echo "# Autogenerated by r-wrapper2.nix from R_LIBS_SITE" > $out/$fixLibsR
echo -n ".libPaths(c(.libPaths(), \"" >> $out/$fixLibsR
echo -n $R_LIBS_SITE | sed -e 's/:/", "/g' >> $out/$fixLibsR
echo -n "\"))" >> $out/$fixLibsR
echo >> $out/$fixLibsR

mkdir -p $out/bin
cd ${R}/bin
for exe in *; do
  makeWrapper ${R}/bin/$exe $out/bin/$exe \
  --set R_PROFILE_USER $out/$fixLibsR
done
''

rWrapper2.nix:

self: super: {
  rWrapper2 = super.callPackage ./r-wrapper2.nix {
    inherit (self.rWrapper) recommendedPackages;
    packages = [ ];
  };
}

default.nix:

with import <nixpkgs-unstable> { overlays = [ (import ./rWrapper2.nix) ]; }; {

  rEnv = rWrapper2.override {
    packages = with rPackages; [ pkgbuild brms ];
  };
}

[PhDyellow]

I had a related problem, I was pulling in a very large number of packages into my development environment. After a bit of debugging, I figured out the $R_LIBS_SITE was getting truncated.

The typical R installation has a handful of library paths, (something like system, admin provided global, and user installed local), and R looks in each folder for subfolders to find packages. NixOS is an unusual host system for an R installation, because NixOS gives every package an entry in $R_LIBS_SITE. You go from ~4 to 5 entries in $R_LIBS_SITE to an entry per package.

I put together a wrapper for R that symlinks all the packages into a "system library", so $R_LIBS_SITE can contain a single entry. I am managing ~400 packages this way.

{
  pkgs,
  R,
  makeWrapper,
  rPa,
  stdenv,
  util-linux
}:

#takes a list of rPackages, and copies each into it's own out
let

  in stdenv.mkDerivation{

    name = R.name + "-wrapper-compact-lib";

    buildInputs = [makeWrapper R] ++ rPa;

    unpackPhase = ":";

    installPhase = ''
               mkdir -p $out/library
while IFS=':' read -r -a RP; do
  for rpp in ''${RP[@]}; do
    for dir in $rpp/*/; do
      ln -sr $dir $out/library/
    done
  done
done <<< $R_LIBS_SITE
mkdir -p $out/bin
cd ${R}/bin
for exe in *;do
makeWrapper ${R}/bin/$exe $out/bin/$exe \
  --set "R_LIBS_SITE" "$out/library"
done
'';

}

I named the file nix_r_compact_libs.nix, then included the following line to include the R wrapper in the environment packages:

(pkgs.callPackage ./nix_r_compact_libs.nix {rPa = rpackages;})

where rpackages is just a list of R packages.

Code currently lives in my github repo because there is a backlog of PR's for the R language in nixpkgs at the moment. https://github.com/PhDyellow/nix_r_dev_shell

[collares]

This is a combination of two problems, a small buffer and buggy error reporting. Upstream fixed the error reporting and increased the buffer size from 10,000 bytes to 100,000 bytes, see https://bugs.r-project.org/bugzilla/show_bug.cgi?id=18001 for details. This is included in R 4.1.0 (released May 2021), so we just need to update R to get the fix.

[PhDyellow]

So R 4.1.0 should allow ~1000 packages in nix. That's good.

[jwnki]

@collares thanks for explaining this. I'll close the issue when R 4.1.0 will be available on stable.

[collares]

Fixed by #127798.