"In CPL_read_ogr(...): GDAL Error 1: PROJ: proj_identify: Cannot find proj.db" and "Error: node stack overflow"

[maxl-amberg]

Hi all,

I seem to have a problem related to proj.db and the setting of PROJ_LIB.

Description: When using st_read(), I am getting the following warning:

Warning message:
In CPL_read_ogr(dsn, layer, query, as.character(options), quiet,  :
  GDAL Error 1: PROJ: proj_identify: Cannot find proj.db

As a consequence, using geom_sf() yields the following error:

Error: node stack overflow
In addition: There were 50 or more warnings (use warnings() to see the first 50)
Error during wrapup: node stack overflow
Error: no more error handlers available (recursive errors?); invoking 'abort' restart

Repex The warning and error occur no matter which shapefile I load and regradless of the shapefile I want to plot. (For instance, I encountered it when loading Eurostat's NUTS 3 shapefile).

Additional context I have other software installed that might set the environment variable PROJ_LIB: Since I have last used st_read() and geom_sf() in my R project, I installed Anaconda, Python and VS Code to create a conda environment (incl. geopandans, etc.) to run Python via VS Code. However, I do not use conda for R. Instead, I use the renv package in my R project. Apart from that, I am not sure whether the upgrade from sf_1.0-14 to sf_1.0-15 or the installation of tmap might cause the issue. Lastly, running Sys.unsetenv("PROJ_LIB") at the beginning of the script does not do the trick.

> utils::sessionInfo() R version 4.2.2 (2022-10-31) Platform: aarch64-apple-darwin20 (64-bit) Running under: macOS 14.3 Matrix products: default LAPACK: /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices datasets utils methods base loaded via a namespace (and not attached): [1] Rcpp_1.0.12 lattice_0.20-45 tidyr_1.3.0 class_7.3-20 snow_0.4-4 zoo_1.8-12 gtools_3.9.4 [8] lmtest_0.9-40 foreach_1.5.2 utf8_1.2.4 gmp_0.6-10 R6_2.5.1 plyr_1.8.8 MatrixModels_0.5-3 [15] e1071_1.7-14 ggplot2_3.4.4 pillar_1.9.0 rlang_1.1.3 cubature_2.1.0 rstudioapi_0.14 SparseM_1.81 [22] Matrix_1.6-5 glmx_0.1-3 splines_4.2.2 stringr_1.5.0 fastDummies_1.6.3 bit_4.0.5 munsell_0.5.0 [29] proxy_0.4-27 compiler_4.2.2 numDeriv_2016.8-1.1 pkgconfig_2.0.3 CVXR_1.0-11 doSNOW_1.0.20 tidyselect_1.2.0 [36] tibble_3.2.1 quadprog_1.5-8 codetools_0.2-18 np_0.60-17 fansi_1.0.6 dplyr_1.1.4 sf_1.0-15 [43] MASS_7.3-58.1 grid_4.2.2 gtable_0.3.4 lifecycle_1.0.4 DBI_1.2.2 magrittr_2.0.3 units_0.8-2 [50] scales_1.3.0 KernSmooth_2.23-20 cli_3.6.2 scpi_2.2.5 stringi_1.7.12 renv_1.0.3 Rmpfr_0.9-0 [57] reshape2_1.4.4 generics_0.1.3 vctrs_0.6.5 boot_1.3-28 sandwich_3.1-0 Formula_1.2-4 iterators_1.0.14 [64] tools_4.2.2 bit64_4.0.5 glue_1.7.0 purrr_1.0.1 Qtools_1.5.6 abind_1.4-5 parallel_4.2.2 [71] survival_3.4-0 yaml_2.3.7 colorspace_2.1-0 classInt_0.4-9 ECOSolveR_0.5.4 quantreg_5.97 > sf::sf_extSoftVersion() GEOS GDAL proj.4 GDAL_with_GEOS USE_PROJ_H PROJ "3.11.0" "3.5.3" "9.1.0" "true" "true" "9.1.0"

Many thanks in advance!

[edzer]

We need your output of

sessionInfo()
Sys.getenv("PROJ_LIB")
Sys.getenv("PROJ_DATA")
Sys.getenv("GDAL_DATA")

[rsbivand]

How did you install sf? CRAN binary, CRAN source with CRAN recipes static libraries, or otherwise?

[maxl-amberg]

Thanks for the quick answers!

I already provided the session info under "details" (see above). Apart from that, the remaining outputs are as follows:

> Sys.getenv("PROJ_LIB")
[1] ""
> Sys.getenv("PROJ_DATA")
[1] ""
> Sys.getenv("GDAL_DATA")
[1] ""

I simply installed the sf package via install.packages("sf") and also tried renv:install("sf").

[rsbivand]

Yes, but did you install the binary package from a CRAN mirror (which mirror?) or did you install the source package? Use for example type="binary" because if you got a source package, you might get a failed install if you also have PROJ, GDAL and GEOS also installed on your platform by something else.

[maxl-amberg]

I see.

Regarding the CRAN mirror currently used:

> getOption("repos")["CRAN"]
                                         CRAN 
"https://packagemanager.posit.co/cran/latest"

My default is the type is the following:

> getOption("pkgType") 
[1] "both"

I have just removed the sf package again and tried install::packages("sf", type = "binary"), but this did not help – probably because "both" already implies ‘use binary if available and current, otherwise try source’.

[edzer]

That still does not tell us whether your sf came from a binary or a source install. What does it tell you when you install it? What's the extension of the downloaded file?

[maxl-amberg]

Sorry about that.

I am getting the following information: install.packages("sf") yields:

The following package(s) will be installed:
- sf [1.0-15]
These packages will be installed into "~/workdir/repos/github_private/transport_project/renv/library/R-4.2/aarch64-apple-darwin20".

Do you want to proceed? [Y/n]: Y

# Installing packages --------------------------------------------------------
- Installing sf ...                             OK [linked from cache]
Successfully installed 1 package in 8.3 milliseconds.

install.packages("sf", type = "both") and install.packages("sf", type = "binary") both yield:

Installing package into ‘/Users/maxl-amberg/workdir/repos/github_private/transport_project/renv/library/R-4.2/aarch64-apple-darwin20’
(as ‘lib’ is unspecified)
trying URL 'https://packagemanager.posit.co/cran/latest/bin/macosx/big-sur-arm64/contrib/4.2/sf_1.0-15.tgz'
Content type 'binary/octet-stream' length 19277046 bytes (18.4 MB)
==================================================
downloaded 18.4 MB

The downloaded binary packages are in
    /var/folders/tn/psl2yx_50ml1lr1zm43m11080000gn/T//RtmpTcuxs1/downloaded_packages

Lastly, install.packages("sf", type = "source") yields:

Installing package into ‘/Users/maxl-amberg/workdir/repos/github_private/transport_project/renv/library/R-4.2/aarch64-apple-darwin20’
(as ‘lib’ is unspecified)
trying URL 'https://packagemanager.posit.co/cran/latest/src/contrib/sf_1.0-15.tar.gz'
Content type 'binary/octet-stream' length 3544759 bytes (3.4 MB)
==================================================
downloaded 3.4 MB

- One or more packages recorded in the lockfile are not installed.
- Use `renv::status()` for more details.
* installing *source* package ‘sf’ ...
** package ‘sf’ successfully unpacked and MD5 sums checked
** using staged installation
configure: CC: clang -arch arm64
configure: CXX: clang++ -arch arm64 -std=gnu++14
checking for gdal-config... no
no
configure: error: gdal-config not found or not executable.
ERROR: configuration failed for package ‘sf’
* removing ‘/Users/maxl-amberg/workdir/repos/github_private/transport_project/renv/library/R-4.2/aarch64-apple-darwin20/sf’

The downloaded source packages are in
    ‘/private/var/folders/tn/psl2yx_50ml1lr1zm43m11080000gn/T/RtmpPJK3FK/downloaded_packages’
Warning message:
In utils::install.packages("sf", type = "source") :
  installation of package ‘sf’ had non-zero exit status

I hope this is what you need. Please let me know if not.

[rsbivand]

You are apparently using R 4.2. Avoid at all costs the Posit mirror and renv, they are the joint suspect until proven innocent. Download https://cran.r-project.org/bin/macosx/big-sur-arm64/contrib/4.2/sf_1.0-15.tgz manually and install manually from the command line (shell prompt) in regular R outside RStudio. If sf loads correctly in your (outdated) R 4.2, and example(st_crs) runs, you have located the problem (Posit mirror, Posit package systems, possibly Posit-built binary ignoring the need to build binary with extra configure arguments on Windows and macOS static builds).

[rsbivand]

Why avoid Posit? If you use CRAN builds, they are tested intensively very often. Posit does not test, or not often, and support for 4.2 may be patchy. Anyway better to upgrade now to R 4.3.3 just released, and to R 4.4 when it is released, most likely next month. Your macOS is probably a good deal younger than R 4.2.2.

[maxl-amberg]

Thanks. I have just upgraded to R 4.3.3.

However, renv::install("sf", type = "binary") now yields:

# Downloading packages -------------------------------------------------------
- Downloading sf from CRAN ...                      ERROR [error code 22]
Error: error downloading 'https://packagemanager.posit.co/cran/latest/src/contrib/sf_1.0-15.tgz' [error code 22]

Also renv::install("sf", type = "source") ends with an error:

# Downloading packages -------------------------------------------------------
- Downloading sf from CRAN ...                  OK [file is up to date]
- Downloading units from CRAN ...               OK [file is up to date]
Successfully downloaded 2 packages in 0.17 seconds.

The following package(s) will be installed:
- sf    [1.0-15]
- units [0.8-5]
These packages will be installed into "~/workdir/repos/github_private/transport_project/renv/library/R-4.3/aarch64-apple-darwin23.2.0".

Do you want to proceed? [Y/n]: Y

# Installing packages --------------------------------------------------------
- Installing units ...                          FAILED
Error: Error installing package 'units':
=================================

* installing *source* package ‘units’ ...
** package ‘units’ successfully unpacked and MD5 sums checked
** using staged installation
configure: units: 0.8-5
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C++... yes
checking whether clang++ -std=gnu++17 accepts -g... yes
checking for clang++ -std=gnu++17 option to enable C++11 features... none needed
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for _Bool... no
checking for stdbool.h that conforms to C99... yes
checking for error_at_line... no
checking for gcc... clang
checking whether the compiler supports GNU C... yes
checking whether clang accepts -g... yes
checking for clang option to enable C11 features... none needed
checking for XML_ParserCreate in -lexpat... yes
checking for udunits2.h... no
checking for udunits2/udunits2.h... no
checking for ut_read_xml in -ludunits2... no
configure: error: in `/private/var/folders/tn/psl2yx_50ml1lr1zm43m11080000gn/T/RtmpTn4tT4/R.INSTALL1013b283316a7/units':
configure: error: 
--------------------------------------------------------------------------------
  Configuration failed because libudunits2.so was not found. Try installing:
    * deb: libudunits2-dev (Debian, Ubuntu, ...)
    * rpm: udunits2-devel (Fedora, EPEL, ...)
    * brew: udunits (OSX)
  If udunits2 is already installed in a non-standard location, use:
    --configure-args='--with-udunits2-lib=/usr/local/lib'
  if the library was not found, and/or:
    --configure-args='--with-udunits2-include=/usr/include/udunits2'
  if the header was not found, replacing paths with appropriate values.
  You can alternatively set UDUNITS2_INCLUDE and UDUNITS2_LIBS manually.
--------------------------------------------------------------------------------
See `config.log' for more details
ERROR: configuration failed for package ‘units’
* removing ‘/Users/maxl-amberg/workdir/repos/github_private/transport_project/renv/staging/1/units’
install of package 'units' failed [error code 1]

Same if install.packages() is used with type = "binary" or type = "source".

[edzer]

My advice: try another CRAN mirror, try not using renv, try install binaries.

[maxl-amberg]

Thanks for your advice! Unfortunately, install.packages("sf", type = "binary", repos = "https://cran.r-project.org") also yields:

# Downloading packages -------------------------------------------------------
- Downloading sf from CRAN ...                      ERROR [error code 22]
Error: error downloading 'https://cran.r-project.org/src/contrib/sf_1.0-15.tgz' [error code 22]

[edzer]

No idea how you get there: that file does not exist. Looking at the landing page of sf on CRAN, the file you should be downloading is https://cran.r-project.org/bin/macosx/big-sur-arm64/contrib/4.3/sf_1.0-15.tgz (for r-release).

[maxl-amberg]

Not sure whether homebrew has a problem with newer versions of R (see here). This is why I uninstalled R via homebrew and installed R 4.3.3 for MacOS the usual way again. In my fresh setup, getOption("repos")["CRAN"] still gives me

                                         CRAN 
"https://packagemanager.posit.co/cran/latest" 

while getOption("pkgType") yields "both".

Using install.packages("sf", type = "binary", repos = "https://cran.r-project.org") allows me to download/install the file you are referring to:

Installing package into ‘/Users/maxl-amberg/workdir/repos/github_private/transport_project/renv/library/R-4.3/aarch64-apple-darwin20’
(as ‘lib’ is unspecified)
trying URL 'https://cran.r-project.org/bin/macosx/big-sur-arm64/contrib/4.3/sf_1.0-15.tgz'
Content type 'application/x-gzip' length 86909904 bytes (82.9 MB)
==================================================
downloaded 82.9 MB

The downloaded binary packages are in
    /var/folders/tn/psl2yx_50ml1lr1zm43m11080000gn/T//Rtmpv0V214/downloaded_packages

However, I am still getting the initially posted warning and error messages when using st_read() to load and geom_sf() to plot the shapefile.

[maxl-amberg]

@rsbivand: Could the problem be associated with the use of tmap (as you suspected here some time ago)? As mentioned initially, the warning and error occurred after I installed and used tmap for the first time.

By the way, example(st_crs) yields:

st_crs> sfc = st_sfc(st_point(c(0,0)), st_point(c(1,1)))

st_crs> sf = st_sf(a = 1:2, geom = sfc)

st_crs> st_crs(sf) = 4326

st_crs> st_geometry(sf)
Geometry set for 2 features 
Geometry type: POINT
Dimension:     XY
Bounding box:  xmin: 0 ymin: 0 xmax: 1 ymax: 1
CRS:           NA
POINT (0 0)
POINT (1 1)

st_crs> sfc = st_sfc(st_point(c(0,0)), st_point(c(1,1)))

st_crs> st_crs(sfc) = 4326

st_crs> sfc
Geometry set for 2 features 
Geometry type: POINT
Dimension:     XY
Bounding box:  xmin: 0 ymin: 0 xmax: 1 ymax: 1
CRS:           NA
POINT (0 0)
POINT (1 1)

st_crs> sfc = st_sfc(st_point(c(0,0)), st_point(c(1,1)))

st_crs> sfc %>% st_set_crs(4326) %>% st_transform(3857)
Error in st_transform.sfc(., 3857) : 
  cannot transform sfc object with missing crs
In addition: Warning messages:
1: In CPL_crs_from_input(x) :
  GDAL Error 1: PROJ: proj_create_from_database: Cannot find proj.db
2: In CPL_crs_from_input(x) :
  GDAL Error 1: PROJ: proj_create_from_database: Cannot find proj.db
3: In CPL_crs_from_input(x) :
  GDAL Error 1: PROJ: proj_create_from_database: Cannot find proj.db

[maxl-amberg]

@edzer: The proj.db file is located in /Users/maxl-amberg/workdir/repos/github_private/transport_project/renv/library/R-4.3/aarch64-apple-darwin20/sf/proj.

Since Sys.getenv("PROJ_LIB") yields "", do I need to set it via Sys.setenv("PROJ_LIB") manually? Unfortunately, Sys.setenv(PROJ_LIB = "/Users/maxl-amberg/workdir/repos/github_private/transport_project/renv/library/R-4.3/aarch64-apple-darwin20/sf/proj") does not change anything.

[edzer]

Have you tried doing all this without renv, as I suggested above?

[maxl-amberg]

Yes, I have tried it (but literally 10 min. ago). It worked out outside of my project in which I am using renv.

Outside of the renv-managed project, the setup is as follows:

> getOption("repos")["CRAN"]
                       CRAN 
"https://cran.rstudio.com/" 
> getOption("pkgType")
[1] "both"
> Sys.getenv("PROJ_LIB")
[1] ""
> Sys.getenv("PROJ_DATA")
[1] ""
> Sys.getenv("GDAL_DATA")
[1] "/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/sf/gdal"

So Sys.getenv("GDAL_DATA") is not empty. Does this mean that within renv-managed projects Sys.setenv(GDAL_DATA = "...") has to be set manually?

[edzer]

Does this mean that within renv-managed projects Sys.setenv(GDAL_DATA = "...") has to be set manually?

You tell me. To me it tells that renv is the cause of your problem. See also this thread: https://mastodon.social/@joranelias/112039358805771553

[edzer]

Similar to: https://github.com/rstudio/renv/issues/1832

[daniel-meister]

Why avoid Posit? If you use CRAN builds, they are tested intensively very often. Posit does not test, or not often, and support for 4.2 may be patchy. Anyway better to upgrade now to R 4.3.3 just released, and to R 4.4 when it is released, most likely next month. Your macOS is probably a good deal younger than R 4.2.2.

Just debugged this problem in a different context and leaving a note for other people whom a search engine directs here in the future. The contents of the following two files are not idential:

• https://cran.rstudio.com/bin/macosx/big-sur-arm64/contrib/4.2/sf_1.0-16.tgz
• https://packagemanager.posit.co/cran/latest/bin/macosx/big-sur-arm64/contrib/4.2/sf_1.0-16.tgz

Specifically, the one from Posit package manager is missing two subfolders proj and gdal which explains many of these problems.

[rsbivand]

On macOS, when building binary packages not being installed on the same file system, the configure argument --with-data-copy=yes needs to be used to copy the GDAL and PROJ metadata files into the built package. On Windows this is handled by Makefile.ucrt. Could someone who understands the posit package manager please alert them to this requirement (also applies to other packages with PROJ or GDAL as system requirements)?