Build failure: rPackages.arrow

[PmUUK]

Steps To Reproduce

Steps to reproduce the behavior:

I am trying to install R arrow package, system wide. I add arrow as any other R library I use.

Build log

installing
* installing *source* package 'arrow' ...
file 'configure' has the wrong MD5 checksum
** using staged installation
*** Arrow C++ libraries found via pkg-config at /nix/store/mwavk5kg8rjjcjjz2b4nnhmchbyybrg3-arrow-cpp-12.0.0/lib
**** Warning: library version mismatch
**** C++ is 12.0.0 but R is 11.0.0.3
**** If installation fails, upgrade the C++ library to match
**** or retry with ARROW_USE_PKG_CONFIG=false
PKG_CFLAGS=-I/nix/store/mwavk5kg8rjjcjjz2b4nnhmchbyybrg3-arrow-cpp-12.0.0/include 
PKG_LIBS=-L/nix/store/mwavk5kg8rjjcjjz2b4nnhmchbyybrg3-arrow-cpp-12.0.0/lib -larrow
** libs
/nix/store/d9fndiing52fkalp5knfalrvlb3isi6w-gcc-wrapper-12.2.0/bin/c++ -std=gnu++17 -I"/nix/store/d0i117srx5jgqf0c33mw0p6s3jam04vp-R-4.2.3/lib/R/include" -DNDEBUG -I/nix/store/mwavk5kg8rjjcjjz2b4nnhmchbyybrg3-arrow-cpp-12.0.0/include  -I'/nix/store/wdhxn7lklildc2n0vd6c9sds4f6fz7bs-r-cpp11-0.4.3/library/cpp11/include'    -fpic  -g -O2  -c RTasks.cpp -o RTasks.o
/nix/store/d9fndiing52fkalp5knfalrvlb3isi6w-gcc-wrapper-12.2.0/bin/c++ -std=gnu++17 -I"/nix/store/d0i117srx5jgqf0c33mw0p6s3jam04vp-R-4.2.3/lib/R/include" -DNDEBUG -I/nix/store/mwavk5kg8rjjcjjz2b4nnhmchbyybrg3-arrow-cpp-12.0.0/include  -I'/nix/store/wdhxn7lklildc2n0vd6c9sds4f6fz7bs-r-cpp11-0.4.3/library/cpp11/include'    -fpic  -g -O2  -c altrep.cpp -o altrep.o
In file included from altrep.cpp:18:
./arrow_types.h:41:10: fatal error: arrow/compute/exec/options.h: No such file or directory
   41 | #include <arrow/compute/exec/options.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [/nix/store/d0i117srx5jgqf0c33mw0p6s3jam04vp-R-4.2.3/lib/R/etc/Makeconf:178: altrep.o] Error 1
ERROR: compilation failed for package 'arrow'
* removing '/nix/store/46qp7h3cl6l9lddl950b3dy5c8gjdw43-r-arrow-11.0.0.3/library/arrow'
error: builder for '/nix/store/z446iqac67kjc0869x4fi2ai50vkrnw0-r-arrow-11.0.0.3.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/8asvbibp48gmqg6vr190519qp7q608fc-R-4.2.3-wrapper.drv' failed to build
building '/nix/store/blsyjcwnlgjz8wkl4msjbinmbjlj5l51-system-generators.drv'...
error: 1 dependencies of derivation '/nix/store/5kjjq4z9fmflxb8sfh7h2frf80d56jjd-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/bh9dir11w7p7sslw7g5il9ac50wadz2h-nixos-system-nixVM-23.05.1471.b72aa95f7f0.drv' failed to build 

Additional context

I want to switch to nixos, so I am trying to recreate my Debian working environment on a VM and an old laptop. For now, I hardly understand the nix language, but all other R packages I add were build without problems.

Notify maintainers

I think there are no maintainers listed for the R arrow.

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.35, NixOS, 23.05 (Stoat), 23.05.1353.35130d4b4f0`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.3`
 - channels(root): `"nixos-23.05"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[thanasisn]

The same problem is also with

nix-shell -p rPackages.arrow

and

nix-env -iA nixos.rPackages.arrow

[tjni]

I can help explain what's happening but don't know the best solution in this case. TLDR: This issue is not present in the nixpkgs-unstable channel, because rPackages.arrow was updated to 12.0.0 in https://github.com/NixOS/nixpkgs/pull/234958. A way to fix this is to track nixpkgs-unstable instead of nixos-23.05. More information about channels here.

This happened when arrow-cpp was updated to 12.0.0 in https://github.com/NixOS/nixpkgs/pull/229356. It includes a breaking change in https://github.com/apache/arrow/issues/15280 which moved the arrow/compute/exec/ directory. I guess this was only fixed when rPackages.arrow was updated as well a month later. I suppose the branch off point for NixOS 23.05 was between these two events.

I don't know what the backporting policy is for R packages.

/cc R team @bcdarwin @jbedo

[thanasisn]

Thank you very much for the insight! I had a feeling that was something wrong with the package configuration. I am trying to move to nixos, and R arrow is essential to me, at this point.

Can this be fixed by updating the rPackages.arrow build process in NixOS 23.05, or should I try unstable?

[tjni]

The easiest thing is to try unstable. The nice thing about Nix is that you can revert that change if it's too unstable.

The other possibility that is entirely in your control is to update the package yourself in your own configuration, such as by overriding a single package (in "Pinning an unstable service" section); caveat that I have never tried this myself.

It still could be possible to backport to the 23.05 release branch.

[thanasisn]

The easiest thing is to try unstable. The nice thing about Nix is that you can revert that change if it's too unstable.

You mean to try just the unstable package or the whole unstable NixOS? I love the revert feature, If I manage to make all of my system work, I may leave Debian as a desktop.

The other possibility that is entirely in your control is to update the package yourself in your own configuration, such as by overriding a single package (in "Pinning an unstable service" section); caveat that I have never tried this myself.

I just starting playing with the nix way, I would like to try it, and learn, but it took me some days just to understand the proper way to setup R, so I am not very keen to go that way.

It still could be possible to backport to the 23.05 release branch.

If I accomplish the private package config, I probably can send a fix to the repo.

[tjni]

I meant trying the unstable channel as a whole, but it's a personal preference.

Here's something else I found that you can try: https://discourse.nixos.org/t/selecting-a-channel-with-nix-shell/4247

[jbedo]

Thanks for noting this issue, rPackages.arrow is unfortunately quite sensitive to minor version changes and currently even unstable is broken due to the latest package bump of rPackages. I have bumped arrow to correct this at https://github.com/NixOS/nixpkgs/pull/241068 but it hasn't been checked properly yet.

In this case I think we should backport an update for rPackages.arrow as it is a fairly critical package.

[thanasisn]

I meant trying the unstable channel as a whole, but it's a personal preference.

Here's something else I found that you can try: https://discourse.nixos.org/t/selecting-a-channel-with-nix-shell/4247

Thanks, I had in mind that is an option, and I was going to try it.

[jbedo]

This should be fixed with the backport.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/trouble-installing-rpackages-arrow/46196/2