Build failure: rPackages.terra on darwin

[b-rodrigues]

Steps To Reproduce

I noticed that rPackages.terra doesn’t build successfully on darwin while trying to build packages on Github Actions. Here’s the default.nix I tried:

let
 pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/refs/heads/master.tar.gz") {};
 rpkgs = builtins.attrValues {
   inherit (pkgs.rPackages) terra;
};
 system_packages = builtins.attrValues {
   inherit (pkgs) R;
};
in
 pkgs.mkShell {
   buildInputs = [ rpkgs system_packages ];
}

builds successfully on Ubuntu, not on macOS.

As I don’t have the hardware, I figured I’d open this issue.

Build log

building '/nix/store/jijyk08kamqm9yp0nqsc887if8wx9jlj-r-terra-1.7-71.drv'...
Running phase: unpackPhase
unpacking source archive /nix/store/kqq1zxx9ancmx14pyk2w5w1w363aky6a-terra_1.7-71.tar.gz
source root is terra
setting SOURCE_DATE_EPOCH to timestamp 1706718002 of file terra/MD5
Running phase: patchPhase
Running phase: configurePhase
Running phase: buildPhase
Running phase: checkPhase
Running phase: installPhase
* installing *source* package ‘terra’ ...
** package ‘terra’ successfully unpacked and MD5 sums checked
** using staged installation
configure: CC: clang
configure: CXX: /nix/store/vk5sglv5pjg949v50bk7l5wl48j528pp-clang-wrapper-16.0.6/bin/c++ -std=gnu++17
checking for gdal-config... /nix/store/fxcm05ffff86lrbv8vpc81csi0im35b3-gdal-3.8.5/bin/gdal-config
checking gdal-config usability... yes
configure: GDAL: 3.8.5
checking GDAL version >= 2.0.1... yes
checking for gcc... clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether clang accepts -g... yes
checking for clang option to enable C11 features... none needed
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for gdal.h... yes
checking GDAL: linking with --libs only... yes
checking GDAL: /nix/store/fxcm05ffff86lrbv8vpc81csi0im35b3-gdal-3.8.5/share/gdal/pcs.csv readable... no
checking GDAL: checking whether PROJ is available for linking:... yes
checking GDAL: checking whether PROJ is available for running:... yes
configure: GDAL: 3.8.5
configure: pkg-config proj exists, will use it
configure: using proj.h.
Package libtiff-4 was not found in the pkg-config search path.
Perhaps you should add the directory containing `libtiff-4.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libtiff-4', required by 'proj', not found
configure: PROJ: 9.4.0
checking PROJ: checking whether PROJ and sqlite3 are available for linking:... no
configure: error: libproj or sqlite3 not found in standard or given locations.
ERROR: configuration failed for package ‘terra’
* removing ‘/nix/store/7z6lhpy93bvbgidcwdz4clqb0bsfq58p-r-terra-1.7-71/library/terra’
error: builder for '/nix/store/jijyk08kamqm9yp0nqsc887if8wx9jlj-r-terra-1.7-71.drv' failed with exit code 1;
       last 25 log lines:
       > checking for stdlib.h... yes
       > checking for string.h... yes
       > checking for inttypes.h... yes
       > checking for stdint.h... yes
       > checking for strings.h... yes
       > checking for sys/stat.h... yes
       > checking for sys/types.h... yes
       > checking for unistd.h... yes
       > checking for gdal.h... yes
       > checking GDAL: linking with --libs only... yes
       > checking GDAL: /nix/store/fxcm05ffff86lrbv8vpc81csi0im35b3-gdal-3.8.5/share/gdal/pcs.csv readable... no
       > checking GDAL: checking whether PROJ is available for linking:... yes
       > checking GDAL: checking whether PROJ is available for running:... yes
       > configure: GDAL: 3.8.5
       > configure: pkg-config proj exists, will use it
       > configure: using proj.h.
       > Package libtiff-4 was not found in the pkg-config search path.
       > Perhaps you should add the directory containing `libtiff-4.pc'
       > to the PKG_CONFIG_PATH environment variable
       > Package 'libtiff-4', required by 'proj', not found
       > configure: PROJ: 9.4.0
       > checking PROJ: checking whether PROJ and sqlite3 are available for linking:... no
       > configure: error: libproj or sqlite3 not found in standard or given locations.
       > ERROR: configuration failed for package ‘terra’
       > * removing ‘/nix/store/7z6lhpy93bvbgidcwdz4clqb0bsfq58p-r-terra-1.7-71/library/terra’
       For full logs, run 'nix log /nix/store/jijyk08kamqm9yp0nqsc887if8wx9jlj-r-terra-1.7-71.drv'.
error: 1 dependencies of derivation '/nix/store/w0d8jvppdrf628m9kckydxpvrmph6fcp-nix-shell.drv' failed to build
Error: Process completed with exit code 1.

Additional context

Add any other context about the problem here.

Notify maintainers

@jbedo

I’m also notifying R users that are on darwin (afaik) who might be able to help: @philipp-baumann @nviets

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"

- system: `"x86_64-darwin"`
 - host os: `Darwin 21.6.0, macOS 10.16`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.21.2`
 - nixpkgs: `/nix/store/va49ys5mj9nb766ln1ng4q5s4r7v0vpg-source`

---

Add a :+1: reaction to issues you find important.

[b-rodrigues]

Adding libtiff results in following error:

last 25 log lines:
       > checking for stdlib.h... yes
       > checking for string.h... yes
       > checking for inttypes.h... yes
       > checking for stdint.h... yes
       > checking for strings.h... yes
       > checking for sys/stat.h... yes
       > checking for sys/types.h... yes
       > checking for unistd.h... yes
       > checking for gdal.h... yes
       > checking GDAL: linking with --libs only... yes
       > checking GDAL: /nix/store/fxcm05ffff86lrbv8vpc81csi0im35b3-gdal-3.8.5/share/gdal/pcs.csv readable... no
       > checking GDAL: checking whether PROJ is available for linking:... yes
       > checking GDAL: checking whether PROJ is available for running:... yes
       > configure: GDAL: 3.8.5
       > configure: pkg-config proj exists, will use it
       > configure: using proj.h.
       > Package libcurl was not found in the pkg-config search path.
       > Perhaps you should add the directory containing `libcurl.pc'
       > to the PKG_CONFIG_PATH environment variable
       > Package 'libcurl', required by 'proj', not found
       > configure: PROJ: 9.4.0
       > checking PROJ: checking whether PROJ and sqlite3 are available for linking:... no
       > configure: error: libproj or sqlite3 not found in standard or given locations.
       > ERROR: configuration failed for package ‘terra’
       > * removing ‘/nix/store/9gf7qax1bpwznzlrwkw9bibj66ba4zhh-r-terra-1.7-71/library/terra’
       For full logs, run 'nix log /nix/store/753smqyz9x2r879wmbbpj3jz2bfpds49-r-terra-1.7-71.drv'.
error: 1 dependencies of derivation '/nix/store/s2xl3gnxjas8ma3n8dwy7gxpfi13avql-nix-shell.drv' failed to build
Error: Process completed with exit code 1.

now also trying by adding pkgs.curl.dev.

[b-rodrigues]

Trying now yields

checking PROJ: checking whether PROJ and sqlite3 are available for linking:... no
configure: error: libproj or sqlite3 not found in standard or given locations.
ERROR: configuration failed for package ‘terra’
* removing ‘/nix/store/j5rdhs58x8g45wr2wmm8pirijn2cgrir-r-terra-1.7-71/library/terra’
error: builder for '/nix/store/i2mjzzyqx158w24n1kjmmlvj0jm0lyai-r-terra-1.7-71.drv' failed with exit code 1;
       last 25 log lines:
       > checking for stdlib.h... yes
       > checking for string.h... yes
       > checking for inttypes.h... yes
       > checking for stdint.h... yes
       > checking for strings.h... yes
       > checking for sys/stat.h... yes
       > checking for sys/types.h... yes
       > checking for unistd.h... yes
       > checking for gdal.h... yes
       > checking GDAL: linking with --libs only... yes
       > checking GDAL: /nix/store/fxcm05ffff86lrbv8vpc81csi0im35b3-gdal-3.8.5/share/gdal/pcs.csv readable... no
       > checking GDAL: checking whether PROJ is available for linking:... yes
       > checking GDAL: checking whether PROJ is available for running:... yes
       > configure: GDAL: 3.8.5
       > configure: pkg-config proj exists, will use it
       > configure: using proj.h.
       > Package libtiff-4 was not found in the pkg-config search path.
       > Perhaps you should add the directory containing `libtiff-4.pc'
       > to the PKG_CONFIG_PATH environment variable
       > Package 'libtiff-4', required by 'proj', not found
       > configure: PROJ: 9.4.0
       > checking PROJ: checking whether PROJ and sqlite3 are available for linking:... no
       > configure: error: libproj or sqlite3 not found in standard or given locations.
       > ERROR: configuration failed for package ‘terra’
       > * removing ‘/nix/store/j5rdhs58x8g45wr2wmm8pirijn2cgrir-r-terra-1.7-71/library/terra’
       For full logs, run 'nix log /nix/store/i2mjzzyqx158w24n1kjmmlvj0jm0lyai-r-terra-1.7-71.drv'.
error: 1 dependencies of derivation '/nix/store/slbk9azn08jkbgz2665gksn4wyr10w94-nix-shell.drv' failed to build
Error: Process completed with exit code 1. 

looks like libtiff can't be found?

[jbedo]

The logs suggest libtiff is needed to build PROJ, which I guess is vendored. Perhaps you could provide proj (and sqlite since it's complainign about that too) out of nixpkgs instead?

[b-rodrigues]

The logs suggest libtiff is needed to build PROJ, which I guess is vendored. Perhaps you could provide proj (and sqlite since it's complainign about that too) out of nixpkgs instead?

but isn’t that the case already? See https://github.com/NixOS/nixpkgs/blob/5672bc9dbf9d88246ddab5ac454e82318d094bb8/pkgs/development/r-modules/default.nix#L451

[jbedo]

In that case the problem is that it's not detected.

[b-rodrigues]

Something similar happens with rPackages.sf

ERROR: configuration failed for package ‘sf’
* removing ‘/nix/store/b8fyr26jkqgn39drgii5rbs08496kxy6-r-sf-1.0-16/library/sf’
error: builder for '/nix/store/aal02wn0i69yl75r1xlsirchaqhv9sk7-r-sf-1.0-16.drv' failed with exit code 1;
       last 25 log lines:
       > checking whether the compiler supports GNU C... yes
       > checking whether clang accepts -g... yes
       > checking for clang option to enable C11 features... none needed
       > checking for stdio.h... yes
       > checking for stdlib.h... yes
       > checking for string.h... yes
       > checking for inttypes.h... yes
       > checking for stdint.h... yes
       > checking for strings.h... yes
       > checking for sys/stat.h... yes
       > checking for sys/types.h... yes
       > checking for unistd.h... yes
       > checking for gdal.h... yes
       > checking GDAL: linking with --libs only... yes
       > checking GDAL: /nix/store/fxcm05ffff86lrbv8vpc81csi0im35b3-gdal-3.8.5/share/gdal/pcs.csv readable... no
       > checking GDAL: checking whether PROJ is available for linking:... yes
       > checking GDAL: checking whether PROJ is available for running:... yes
       > configure: GDAL: 3.8.5
       > configure: pkg-config proj exists, will use it
       > configure: using proj.h.
       > configure: PROJ: 9.4.0
       > checking PROJ: checking whether PROJ and sqlite3 are available for linking:... no
       > configure: error: libproj or sqlite3 not found in standard or given locations.
       > ERROR: configuration failed for package ‘sf’
       > * removing ‘/nix/store/b8fyr26jkqgn39drgii5rbs08496kxy6-r-sf-1.0-16/library/sf’
       For full logs, run 'nix log /nix/store/aal02wn0i69yl75r1xlsirchaqhv9sk7-r-sf-1.0-16.drv'.
error: 1 dependencies of derivation '/nix/store/8w4bdyifz96wp54jld564qcfkzpdzfyx-nix-shell.drv' failed to build
Error: Process completed with exit code 1.

[b-rodrigues]

Ok, so overriding terra’s definition to add a configure flag seems to work:

let
 pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/refs/heads/master.tar.gz") {};
 terra = pkgs.rPackages.terra.overrideAttrs (attrs: {
   buildInputs = attrs.buildInputs ++ pkgs.lib.optional pkgs.stdenv.isDarwin [ pkgs.curl.dev pkgs.libtiff ];
   configureFlags = [
        "--with-proj-lib=${pkgs.lib.getLib pkgs.proj}/lib"
      ];
 });
 system_packages = builtins.attrValues {
   inherit (pkgs) R;
};
in
 pkgs.mkShell {
   buildInputs = [ terra system_packages ];
}

This also works for Linux, but is unnecessary. How to add this flag only for darwin? I guess this could then go into the pkgs/development/r-modules/default.nix file, but I’d prefer if someone with the hardware could test this and open the PR.

[b-rodrigues]

So it turns out that adding the optional build inputs is not even needed, this builds successfully on darwin:

let
 pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/refs/heads/master.tar.gz") {};
 terra = pkgs.rPackages.terra.overrideAttrs (attrs: {
   configureFlags = [
        "--with-proj-lib=${pkgs.lib.getLib pkgs.proj}/lib"
      ];
 });
 system_packages = builtins.attrValues {
   inherit (pkgs) R;
};
in
 pkgs.mkShell {
   buildInputs = [ terra system_packages ];
}

[b-rodrigues]

Noticed the same issues with sf and vapour. Here is what I did to get it to build on Github Actions:

 terra = pkgs.rPackages.terra.overrideAttrs (attrs: {
   configureFlags = [
        "--with-proj-lib=${pkgs.lib.getLib pkgs.proj}/lib"
      ];
 });
 sf = pkgs.rPackages.sf.overrideAttrs (attrs: {
   configureFlags = [
        "--with-proj-lib=${pkgs.lib.getLib pkgs.proj}/lib"
      ];
 });
 vapour = pkgs.rPackages.vapour.overrideAttrs (attrs: {
   configureFlags = [
        "--with-proj-lib=${pkgs.lib.getLib pkgs.proj}/lib"
      ];
 });

Again, if someone with darwin hardware can try, that'd be great :smile:

These flags are not needed on linux though, and I don't know how to apply these flags on darwin only. They don't prevent building on linux though.

[Kupac]

lib.optionals?

[philipp-baumann]

Noticed the same issues with sf and vapour. Here is what I did to get it to build on Github Actions:

 terra = pkgs.rPackages.terra.overrideAttrs (attrs: {
   configureFlags = [
        "--with-proj-lib=${pkgs.lib.getLib pkgs.proj}/lib"
      ];
 });
 sf = pkgs.rPackages.sf.overrideAttrs (attrs: {
   configureFlags = [
        "--with-proj-lib=${pkgs.lib.getLib pkgs.proj}/lib"
      ];
 });
 vapour = pkgs.rPackages.vapour.overrideAttrs (attrs: {
   configureFlags = [
        "--with-proj-lib=${pkgs.lib.getLib pkgs.proj}/lib"
      ];
 });

Again, if someone with darwin hardware can try, that'd be great 😄

These flags are not needed on linux though, and I don't know how to apply these flags on darwin only. They don't prevent building on linux though.

> .libPaths(.libPaths()[-1])
> .libPaths()
 [1] "/nix/store/3wycd2immlxwkzky0vxcfaf36qq2gzmd-r-Rcpp-1.0.12/library"                        
 [2] "/nix/store/fjzgjswwzqlxkbyaiksw19zra5pl3a5a-r-sf-1.0-16/library"                          
 [3] "/nix/store/ijy5m5nx9vzbi1d3gg8vj39cknppzv7h-r-classInt-0.4-10/library"                    
 [4] "/nix/store/94zaagaw6rg9jqca6gz3bpzgzfgfxvih-r-class-7.3-22/library"                       
 [5] "/nix/store/4b18ha68y10wgpr87v82y62kpzhr43mz-r-MASS-7.3-60.0.1/library"                    
 [6] "/nix/store/7kax9rvm8plnlm61lf3w7gq3i7d8cw40-r-e1071-1.7-14/library"                       
 [7] "/nix/store/5lw0qlrg1mfkbklbxwvwclryjw9pj6pw-r-proxy-0.4-27/library"                       
 [8] "/nix/store/7la1ckzdzm3r1c2q087asawdj5952gpn-r-KernSmooth-2.23-22/library"                 
 [9] "/nix/store/0a415lrf6vn4h0vjyr2ipkg1y6yzlwqd-r-DBI-1.2.2/library"                          
[10] "/nix/store/nxnqc027342i8gviy5c98rvmpr3y685p-r-magrittr-2.0.3/library"                     
[11] "/nix/store/daana6ym4906fqpq5izjwk4967aiab3q-r-s2-1.1.6/library"                           
[12] "/nix/store/gxcl5c3dl01xnb3bj79bxdvg5zxglqbk-r-wk-0.9.1/library"                           
[13] "/nix/store/h202xdmrw9azwchyqf2b5gqd52x2zbdq-r-units-0.8-5/library"                        
[14] "/nix/store/1hc7g2wc00qazjwbxb2kbnw7jkpihfg2-r-vapour-0.9.5/library"                       
[15] "/nix/store/m5vfkb972bmjihv4gqfasi5z8nqjarnj-r-jsonlite-1.8.8/library"                     
[16] "/nix/store/1431dnq218bz3v9m6gk94fa4m8aa894i-r-nanoarrow-0.4.0.1/library"                  
[17] "/nix/store/5ylipgzwqx1xs6cadrv5wxrjsial7wlr-apple-framework-CoreFoundation-11.0.0/Library"
[18] "/nix/store/qyg9jzkvx6w3zwpd6hifzjm1871jfd7k-R-4.3.3/lib/R/library"                        
[19] "/nix/store/lqbn21h94fws9gbypxdcavim6jsnwlfd-r-terra-1.7-71/library"                       
> library(sf)
Linking to GEOS 3.12.1, GDAL 3.8.5, PROJ 9.4.0; sf_use_s2() is TRUE
> library(terra)
code for methods in class “Rcpp_SpatCategories” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatCategories” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatDataFrame” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatDataFrame” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatExtent” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatExtent” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatFactor” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatFactor” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatGraph” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatGraph” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatMessages” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatMessages” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatOptions” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatOptions” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatRaster” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatRaster” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatRasterCollection” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatRasterCollection” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatRasterStack” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatRasterStack” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatSRS” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatSRS” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatTime_v” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatTime_v” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatVector” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatVector” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatVectorCollection” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatVectorCollection” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatVectorProxy” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
code for methods in class “Rcpp_SpatVectorProxy” was not checked for suspicious field assignments (recommended package ‘codetools’ not available?)
terra 1.7.71
> library("vapour")

[b-rodrigues]

I don’t understand, did it build successfully without the fix?

[Kupac]

It looks like some kind of warning, bcz a recommended package is missing. If you create a shell with codetools included, it should go away, no?

[philipp-baumann]

It looks like some kind of warning, bcz a recommended package is missing. If you create a shell with codetools included, it should go away, no?

indeed the warning goes away with rPackages.codetools in nix-shell buildInputs. But its super weird. Sounds a bit similar as this one: https://www.mail-archive.com/debian-bugs-rc@lists.debian.org/msg606635.html However, terra does now not depend on raster anymore. So if then would it be somehow related to the order of installation? Or we just put codetools into the runtime to fix the weird warnings on macOS?

[philipp-baumann]

fix addressed in https://github.com/NixOS/nixpkgs/pull/344498