Side effects upon loading rgdal/sf affecting GDAL utils functionality on Windows system with GDAL3

[lbusett]

Hi all,

sorry for the long message, but the problem is a bit complicated to explain.

I noticed this problem because at the moment the “spatial stack” on Windows is in a sorry state, where rgdal/sp/sf binaries are linked to GDAL2/PROJ4 while the OSGEO installer/updater (IMO the “best option” to install spatial libraries on Windows) already provides GDAL3/PROJ6 on new installs, or performs updates on old installs.

This creates problems if a R user/package wants to use both rgdal/sf functionality AND gdal binary utilities (e.g., gdalwarp, ogr2ogr) called either “directly” with system2, or through the gdalUtils package, due to side-effects happening when loading/importing rgdal/sf.

In particular, there appears to be problems related with rgdal and sf modifying the PROJ_LIB environment variable on load, happening here and here.

I’ll try to clarify with an example:

On a recently updated windows machine with GDAL3/PROJ6 and all R packages up to date, in order for GDAL projection stuff to work the PROJ_LIB environment variable needs to be set to the “share/proj” subfolder of the OSgeo install folder.

If this is the case, on a “fresh” R session I can use gdal binaries from R with no problems:

# What is the "PROJ_LIB" variable?
# At the beginning, on a fresh 'R' session, it is the "correct" one for GDAL3/PROJ6
Sys.getenv("PROJ_LIB")
#> [1] "C:\\OSGeo4W64\\share\\proj"

# Get info on a file

in_file <- system.file("vectors", "cities.shp", package = "rgdal")
info <- system2("C:/OSGeo4W64/bin/gdalsrsinfo.exe",
                args = c(in_file, "-o proj4"),
                stderr = TRUE, stdout = TRUE)
info
#> [1] ""                                    "+proj=longlat +datum=WGS84 +no_defs"
#> [3] ""

# Perform a reprojection using 'ogr2ogr'

outfile <- tempfile(fileext = ".gpkg")
reproj  <- system2("C:/OSGeo4W64/bin/ogr2ogr.exe",
                   args = c(outfile, in_file, "-t_srs EPSG:3035"),
                   stderr = TRUE, stdout = TRUE)
reproj
#> character(0)

sf::st_read(outfile)
#> Reading layer `cities' from data source `C:\Users\LB\AppData\Local\Temp\RtmpCWStMT\file149448537612.gpkg' using driver `GPKG'
#> Simple feature collection with 606 features and 4 fields
#> geometry type:  POINT
#> dimension:      XY
#> bbox:           xmin: -5118229 ymin: -5398972 xmax: 16642620 ymax: 13173490
#> epsg (SRID):    3035
#> proj4string:    +proj=laea +lat_0=52 +lon_0=10 +x_0=4321000 +y_0=3210000 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs

However, as soon as rgdal or sf are loaded we start having problems , because the environment variable is changed so to point to the rgdal/proj subfolder of the gdal (or sf) package installation folder:

library(rgdal)
#> Loading required package: sp
#> rgdal: version: 1.4-8, (SVN revision 845)
#>  Geospatial Data Abstraction Library extensions to R successfully loaded
#>  Loaded GDAL runtime: GDAL 2.2.3, released 2017/11/20
#>  Path to GDAL shared files: C:/Users/LB/Documents/R/win-library/3.6/rgdal/gdal
#>  GDAL binary built with GEOS: TRUE 
#>  Loaded PROJ.4 runtime: Rel. 4.9.3, 15 August 2016, [PJ_VERSION: 493]
#>  Path to PROJ.4 shared files: C:/Users/LB/Documents/R/win-library/3.6/rgdal/proj
#>  Linking to sp version: 1.3-2

Sys.getenv("PROJ_LIB")
#> [1] "C:/Users/LB/Documents/R/win-library/3.6/rgdal/proj"

, leading the system calls to unexpectedly error because now PROJ_LIB points to the “proj folder” provided by rgdal, which is currently still PROJ4, while the binaries need to use that related to PROJ6.

info <- system2("C:/OSGeo4W64/bin/gdalsrsinfo.exe",
                args = c(in_file, "-o proj4"),
                stderr = TRUE, stdout = TRUE)
info
#> [1] "ERROR 1: PROJ: proj_as_proj_string: Cannot find proj.db"
#> [2] ""                                                       
#> [3] "+proj=longlat +datum=WGS84 +no_defs"                    
#> [4] ""

outfile <- tempfile(fileext = ".gpkg")
reproj  <- system2("C:/OSGeo4W64/bin/ogr2ogr.exe",
                   args = c(outfile, in_file, "-t_srs EPSG:3035"),
                   stderr = TRUE, stdout = TRUE)
#> Warning in system2("C:/OSGeo4W64/bin/ogr2ogr.exe", args = c(outfile,
#> in_file, : running command '"C:/OSGeo4W64/bin/ogr2ogr.exe" C:
#> \Users\LB\AppData\Local\Temp\RtmpCWStMT\file14943c847c59.gpkg C:/Users/LB/
#> Documents/R/win-library/3.6/rgdal/vectors/cities.shp -t_srs EPSG:3035' had
#> status 1
reproj
#> [1] "ERROR 1: PROJ: proj_create_from_database: Cannot find proj.db"
#> [2] "ERROR 1: Failed to process SRS definition: EPSG:3035"         
#> attr(,"status")
#> [1] 1
sf::st_read(outfile)
#> Error: Cannot open "C:\Users\LB\AppData\Local\Temp\RtmpCWStMT\file14943c847c59.gpkg"; 
#> The source could be corrupt or not supported. See `st_drivers()` for a list of supported formats.

The same problem currently prevents the use of gdalUtils functionality on Windows machines with GDAL3 installed. Infact, because gdalUtils imports rgdal, even starting from a “fresh” session we get this:

# At the beginning, PROJ_LIB is "correct" for GDAL3/PROJ6
Sys.getenv("PROJ_LIB")
#> [1] "C:\\OSGeo4W64\\share\\proj"

in_file <- system.file("vectors", "cities.shp", package = "rgdal")
outfile <- tempfile(fileext = ".gpkg")
reproj <- gdalUtils::ogr2ogr(in_file, outfile, t_srs = "EPSG:3035", verbose = TRUE)
#> Warning in system(cmd, intern = TRUE): running command '"C:
#>Checking gdal_installation...
#>Scanning for GDAL installations...
#>Checking the gdalUtils_gdalPath option...
#>GDAL version 3.0.2
#>GDAL command being used: "C:\OSGeo4W64\bin\ogr2ogr.exe" -t_srs "EPSG:3035," 
#>"C:\Users\LB\AppData\Local\Temp\RtmpCwTlHH\file7fc4c3660f9.gpkg" 
#>"C:/Users/LB/Documents/R/win-library/3.6/rgdal/vectors/cities.shp" 

#>Warning message:
#>In system(cmd, intern = TRUE) :
#>  running command '"C:\OSGeo4W64\bin\ogr2ogr.exe" -t_srs "EPSG:3035," 
#>"C:\Users\LB\AppData\Local\Temp\RtmpCwTlHH\file7fc4c3660f9.gpkg" 
#>"C:/Users/LB/Documents/R/win-library/3.6/rgdal/vectors/cities.shp" ' had status 1

reproj
#>[1] "ERROR 1: PROJ: proj_create_from_database: Cannot find proj.db"
#>[2] "ERROR 1: Failed to process SRS definition: EPSG:3035"         
#>attr(,"status")
#>[1] 1

because, apparently, as soon as gdalUtils::ogr2ogr is called the PROJ_LIB variable is changed:

Sys.getenv("PROJ_LIB")
#> [1] "C:/Users/LB/Documents/R/win-library/3.6/rgdal/proj"

Now, although I agree with @rsbivand who strongly suggests to have only one “version” of GDAL libraries installed and used, in particular for unexperienced users, I still think that this behaviour is problematic, for two reasons:

1. On windows, installing from OSGEO is the “way to go” to get GDAL, for both beginners and advanced users. Therefore, until the R binaries start relying on GDAL 3, there may be problems in “mixing” rgdal/gdalUtils/gdal binaries functionalities, and packages exploiting that “mix” could have unexpected breakage. Advanced users can still “downgrade” their GDAL system installation (although it is not ideal), but I fear that less experieced ones would struggle. Also, I know that packages authors could switch to using gdalUtilities or sf::gdal_utils to avoid the problem (I am already doing that within MODIStsp), but 1) unfortunately not all gdal binary utilities are available in gdalUtilities (see for example https://github.com/r-spatial/sf/issues/1213\#issuecomment-563977401), and 2) that would not solve the issue of packages currently relying on gdalUtils or system calls and unable/unwilling to do the switch.

2. Even when the binaries will catch-up on GDAL3/PROJ6, I think that this kind of side effect should be avoided (and I think it is discouraged by CRAN policies, though I may be mistaken).

What is your take on this? I think a way to solve the problem could be to exploit Sys.setenv followed by on.exit calls in any function “requiring” PROJ_LIB to be properly set for rgdal/sf use, so that the environment variable can be automatically reset to its “former” value. Something like:

oldprojlib <- Sys.getenv("PROJ_LIB")
Sys.setenv(PROJ_LIB = "whatevere_needed_by_rgdal")
on.exit(Sys.setenv(PROJ_LIB = oldprojlib))

This seems however a bit cumbersome, because it would need to be replicated several times. Do you see any other way?

Hope this helps, and that I am not missing something obvious here.

Lorenzo

^{Created on 2019-12-12 by the reprex package (v0.3.0)}

[edzer]

Thanks @lbusett ! I hadn't thought of this option as a possible GDAL3/2 fall out.

One way to mitigate this is to use sf::gdal_utils, e.g. sf::gdal_utils("vectortranslate") instead of system("ogr2ogr"), and so on. Then you'll be sure that you use the same GDAL and PROJ as the rest of sf. But only the utils available in the GDAL utils C API are available this way.

Re: your PROJ_LIB suggestion: see here: https://github.com/r-spatial/sf/blob/master/R/init.R#L73-L76 - sf when linked to PROJ6 (using proj.h) already uses the C API call rather than the PROJ_LIB env variable to set the path. Doing this, the problem will go away for sf but obviously only after we have windows binaries of sf with GDAL3/PROJ6.

As of your suggestion for a temporary fix: I'd be happy to submit a "temporary fix" sf PR doing this, but will not write it and also discard it after sf binaries have moved to GDAL3/PROJ6. When these binaries will appear is not entirely clear now: they would require help from @jeroen , and I'm not sure that this is worth the effort now or whether R 4 + the new toolchain are our better short term bet.

[lbusett]

@edzer

thanks for the reply.

Concerning the use of sf_gdal_utils or gdalUtilities: Thanks for the hint. I know about that and agree that, when possible, it is the best approach (also because it allows to avoid "requiring" a previous installation of GDAL.

Concerning the possible "fix" on sf: If I understand correctly, when sf will be linked on PROJ6, Sys.setenv() will no longer be called, thus "automatically" fixing the issue, because no side effect would happen anymore. Therefore, I agree that a temporary fix could be overkill, in particular if the linking with GDAL3 will happen soon.

However, although apparently the side effect related with sf is going to "disappear" soon, I am not sure about rgdal, that I think would continue altering the environmental variable even when Windows binaries will be updated (although without the current "bad" effects). However, maybe that is a "minor" issue that you do not think is worth tackling. I don't know...

Finally, I think a temporary fix at least for avoiding gdalUtils errors could be useful, but maybe it could be more easily implemented over gdalUtils than over sf or rgdal. It would just require calling Sys.setenv() before each call to a gdal binay... Pinging @jgrn307 : what do you think?

[mgageo]

I think a temporary fix in gdalUtils would be useful. I have an error then loading the library : https://i.imgur.com/v3hOEcV.png

I try to use vectortranslate

> sf::gdal_utils(util='vectortranslate', source='rgdal.geojson', destination='rgdal.json', options= c('t_srs','EPSG:2154'))
Error in sf::gdal_utils(util = "vectortranslate", source = "rgdal.geojson",  : 
  gdal_utils vectortranslate: an error occured
In addition: Warning message:
In CPL_gdalvectortranslate(source, destination, options) :
  GDAL Error 1: Couldn't fetch requested layer 't_srs'!

[lbusett]

Ops... I intended to ping @jgrn307 (gdalUtils package maintainer). Sorry, @mgageo.

[edzer]

@mgageo you are hijacking the issue really, but you forgot the - in -t_srs.

[jgrn307]

So I feel like this is a problem on the rgdal side not resetting back the environment variable. Consider the case where someone does an rgdal command FIRST -- there isn't a clean way I can tell for gdalUtils to figure out what the "correct" environment variable is, since its been changed prior to gdalUtils being given the "chance" to fix it. Given how many unique installations there are of gdalUtils, "fixing" the environment variable seems an incredibly challenging problem if its been lost -- is there some gdal command that lets me "recover" the correct variables?

I sent Roger Bivand an email about this, but really if a function is changing a user's environment variable, it should really be setting it back at the end of the run.

[jgrn307]

I'm moving this from https://github.com/gearslaboratory/gdalUtils/issues/24 --> this is really an issue with sf and rgdal setting system environment variables -- gdalUtils doesn't do this, but it does need the environment variables "respected". My suggestion is that for sf and rgdal, the easiest solution is if you are going to change a user's system environment variable, you preserve the old variable as a NEW environment variable, like OLD_PROJ_LIB=the_original_lib

I am happy to tweak gdalUtils to use this (I can easily set this and then set it back to sf/rgdal's preferred variable) but I need the system variable preserved somehow in the session, and I think it would be easiest to just make an "archived" variable and let me know what the name is. Thoughts?

Incidentally, this isn't a Windows-only thing -- this happens with OS X as well which can have > 1 install of GDAL.

[rsbivand]

The OLD_PROJ_LIB idea seems sensible, but a bit fragile (someone else may be using it). Can we assume that the packages are loaded in the same R session, and packages setting the environmental variables can (somehow) write to a known file name in tempdir()instead? If that file is found, it could be read and used instead of using the by-user environmental variable namespace?

[jgrn307]

So I was originally thinking the idea of writing a file like that would be good, but I have repeatedly been told by the CRAN gurus they do not like packages auto-writing to a user's directory. I'd personally be ok with it, but the CRAN gurus may get grumpy. You would also need to set a variable in the user's workspace to point to that file, which can also be fragile.

Is it the name you were concerned about (OLD_PROJ_LIB)? We could pick anything, honestly -- that was just a suggestion. As long as the other packages do not mess with it (I wouldn't with gdalUtils) I think it would be ok?

Maybe RGDAL_OLD_PROJ_LIB?

I'd do this with all the environment variables you are setting just to be on the safe side -- I think only PROJ is causing gdalUtils a headache but I don't know about other programs.

[edzer]

If this is only about PROJ_LIB (and not about GDAL_DATA), then the way forward is to use PROJ6's proj_context_set_search_paths(), an API point to set the directory that drops the need to set or reset environment variables; in sf: https://github.com/r-spatial/sf/blob/master/R/init.R#L73 , https://github.com/r-spatial/sf/blob/master/src/proj.cpp#L19

[rsbivand]

But there are many issues here. 1) gdalUtils does not use compiled code; 2) CRAN Windows and OSX binary packages like sf, gdalcubes, lwgeom and rgdal probably shouldn't be happy about mixing PROJ/GDAL versions (binary CRAN packages won't see the extra DLLs, but will see out-of-sync metadata files); 3) and so on. If we can go with a kludge, using tempdir() might work, but basically gdalUtils could migrate to using sf::gdal_utils(), couldn't it?

[jgrn307]

I didn't know about sf::gdal_utils() --> how does that work? Are the binaries for all the utilities now installed "natively" by sf, or has the API changed so you don't need those standalone executables now?

So just as a bit of a history lesson, I actually developed gdalUtils because the format drivers were not all being installed with rgdal -- hdf4 and hdf5 in particular on Windows boxes -- the only way to get a functional gdal with those drivers on Windows was to hand-install GDAL (OsGEO4W originally). The other functionality grew from that simple need.

If sf has a complete set of utilities and all of the drivers that e.g. Osgeo4w has, that actually makes my life WAY easier (and would mean I can retire from supporting gdalUtils :)

[edzer]

sf::gdal_utils() uses the gdal_utils.h API. Some of the "gdal utilities" may not be in here, either because nobody made the effort or for instance because they are Python scripts that need a Python interpreter anyway.

Recent windows builds for R packages using GDAL have a pretty extensive set of drivers, see https://github.com/rwinlib/gdal2 @jgrn307 I think we didn't keep you in that loop because you don't use binary linking to GDAL.

[edzer]

So, to be clear, sf::gdal_utils does not call any external binary (and will never do that).

[jgrn307]

Ok, very cool -- it looks like it might be time to start retiring some of gdalUtils and maybe, if you are interested, moving some of the value-added stuff.

So, can I propose, for now, you guys just set an "OLD_PROJ" or whatever environment variable so I can (quickly) tweak gdalUtils to work for existing users, and I can commit to starting to migrate gdalUtils functionality to take advantage of sf:gdal_utils() for backwards compatibility? The migration isn't going to happen quickly, so I was hoping for a quick-fix for now.

I like not needing to call an external binary -- this was always a "hack" in my opinion, but one that we needed to get our own work done. It looks like a lot of progress had been made and I'm happy to move towards that solution.

[edzer]

You could develop gdalUtils in a direction where, instead of calling external binaries, it would call sf::gdal_utils, for those functions supported. Still external to gdalUtils, but at least not external to R. I don't know how many functions gdalUtils supports that are not interfaced through sf::gdal_utils, you'll have to decide what to do with them.

[jgrn307]

@edzer that is exactly what I was thinking -- BUT, this is a development angle that is going to take some time that I don't have in the next month, so I was hoping you and @rsbivand would do a quick "tweak" to create that environment variable and I'll do a quick fix with gdalUtils to take advantage of it.

[rsbivand]

Before I even think of doing anything to rgdal, is my assumption that rgdal and gdalUtils are running in the same R session correct? I would see:

1) rgdal puts the on-load values of PROJ_LIB and GDAL_DATA into its own cache. It could also cache the assigned values for the Windows and CRAN binaries.

2) rgdal could have get_*() ans set_*() functions for those cached values - maybe get_*() are enough; the functions would be exported.

3) gdalUtils could check those values if rgdal is already loaded (look in all("package:rgdal" %in% search()) to avoid provoking the loading of the namespace), then if not loaded OK, no action required on load.

4) if rgdal is loaded later, gdalUtils checks the cached values; if unchanged from what it itself sees, OK.

5) If changed, gdalUtils changes the environment variabes to its preferred values, and always changes them back (both on successful conclusion and on error), so that rgdal isn't left looking at the gdalUtils values.

Something like that? That doesn't work for workflows involving other packages using the same environment variables - does that matter?

[edzer]

Right, it is harder than you'd think: point 5 doesn't look like a quick fix. It gets even harder if sf and rgdal are both loaded, since both store PROJ_LIB and GDAL_DATA in local cache and set it back on unload. The last one loaded gets the values from the first one loaded, and these are not the values that gdalUtils would need.

[edzer]

Personally, I'm more in favor of working towards getting GDAL3/PROJ6 to the CRAN windows and OSX build farms.

[jeroen]

I'll start looking into this so that we can bump the windows stack after the 4.0 release.

[jgrn307]

Step five would be an easy fix, actually. Each gdalUtils command is ultimately just a system call, so setting and resetting are approximately two lines of code.

[jgrn307]

@edzer and @rsbivand are you ok with the quick-fix on your end so we can get gdalUtils functional for users? I'll build towards a tighter integration with sf over the coming months, but as I said this isn't something I have the bandwidth for in the near-future, so I want to make sure our users can effectively use both for now.

[rsbivand]

Draft built source package on: http://spatial.nhh.no/R/rgdal/rgdal_1.4-9.tar.gz. Now no Winbuilder, offline until tomorrow. After installing, look at ?get_cached_orig_PROJ_LIB in details, actual details in R/AAA.R to see what gets cached.

[rsbivand]

This will not help if other packages setting the environment variables are also in play, but you should have what you need for a quick fix for gdalUtils, except for maybe for Windows OSGeo4W users installing rgdal from source - needs checking.

[florisvdh]

Not sure whether this is the right place for following comment. For some reason, a colleague running Windows had a problem with gdalUtils 2.0.3.2, notably the gdalsrsinfo() function. It was a problem with +init=epsg:xxxx, similar to problems in issue 1231 of sf, and maybe related to gdalUtils issue 24.

Note that this seems to happen with GDAL 2.2.3 & PROJ.4 4.9.3.

> gdalsrsinfo("+init=epsg:31370", as.CRS = TRUE)
Error in CRS(gsub("'", "", cmd_output)) :
  PROJ4 argument-value pairs must begin with +: Warning 1: +init=epsg:XXXX syntax is deprecated. It might return a CRS with non-EPSG compliant axis order. +proj=lcc +lat_0=90 +lon_0=4.36748666666667 +lat_1=51.1666672333333 +lat_2=49.8333339 +x_0=150000.013 +y_0=5400088.438 +ellps=intl +towgs84=-106.8686,52.2978,-103.7239,0.3366,-0.457,1.8422,-1.2747 +units=m +no_defs
In addition: Warning messages:
1: In if (!is.na(projargs)) { :
  the condition has length > 1 and only the first element will be used
2: In if (!is.na(projargs)) { :
  the condition has length > 1 and only the first element will be used
3: In if (is.na(projargs)) uprojargs <- projargs else uprojargs <- paste(unique(unlist(strsplit(projargs,  :

 Show Traceback

 Rerun with Debug
 Error in CRS(gsub("'", "", cmd_output)) :
  PROJ4 argument-value pairs must begin with +: Warning 1: +init=epsg:XXXX syntax is deprecated. It might return a CRS with non-EPSG compliant axis order. +proj=lcc +lat_0=90 +lon_0=4.36748666666667 +lat_1=51.1666672333333 +lat_2=49.8333339 +x_0=150000.013 +y_0=5400088.438 +ellps=intl +towgs84=-106.8686,52.2978,-103.7239,0.3366,-0.457,1.8422,-1.2747 +units=m +no_defs

Environment info of my colleague is below, including external software versions.

Environment

```r > sessionInfo() R version 3.6.1 (2019-07-05) Platform: i386-w64-mingw32/i386 (32-bit) Running under: Windows >= 8 x64 (build 9200) Matrix products: default locale: [1] LC_COLLATE=Dutch_Belgium.1252 LC_CTYPE=Dutch_Belgium.1252 LC_MONETARY=Dutch_Belgium.1252 [4] LC_NUMERIC=C LC_TIME=Dutch_Belgium.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] gdalUtils_2.0.3.2 DBI_1.1.0 odbc_1.2.2 RODBC_1.3-16 kableExtra_1.1.0 git2rdata_0.2.0 [7] n2khab_0.1.0 knitr_1.27 forcats_0.4.0 stringr_1.4.0 dplyr_0.8.3 purrr_0.3.3 [13] readr_1.3.1 tidyr_1.0.2 tibble_2.1.3 ggplot2_3.2.1 tidyverse_1.3.0 sf_0.8-1 loaded via a namespace (and not attached): [1] httr_1.4.1 jsonlite_1.6 munsell_0.5.0 whisker_0.4 pillar_1.4.3 [6] pkgconfig_2.0.3 webshot_0.5.2 rgeos_0.5-2 R.utils_2.9.2 compiler_3.6.1 [11] lazyeval_0.2.2 hms_0.5.3 httpuv_1.5.2 R6_2.4.1 pkgbuild_1.0.6 [16] yaml_2.2.0 oai_0.3.0 data.table_1.12.8 xtable_1.8-4 tools_3.6.1 [21] promises_1.1.0 mime_0.8 usethis_1.5.1 shiny_1.4.0 modelr_0.1.5 [26] geoaxe_0.1.0 R.oo_1.23.0 reprex_0.3.0 pander_0.6.3 inborutils_0.1.0.9069 [31] git2r_0.26.1 nlme_3.1-143 prettyunits_1.1.1 rgbif_2.1.0 sp_1.3-2 [36] foreach_1.4.8 remotes_2.1.1 digest_0.6.23 KernSmooth_2.23-16 class_7.3-15 [41] codetools_0.2-16 vctrs_0.2.2 later_1.0.0 rvest_0.3.5 withr_2.1.2 [46] xfun_0.12 devtools_2.2.2 tidyselect_0.2.5 testthat_2.3.2 scales_1.1.0 [51] readxl_1.3.1 memoise_1.1.0 stringi_1.4.4 xml2_1.2.2 R.methodsS3_1.7.1 [56] assertthat_0.2.1 units_0.6-5 lattice_0.20-38 bit64_0.9-7 Rcpp_1.0.3 [61] fs_1.3.1 dbplyr_1.4.2 cli_2.0.1 callr_3.4.1 e1071_1.7-3 [66] sessioninfo_1.1.1 grid_3.6.1 classInt_0.4-2 blob_1.2.1 plyr_1.8.5 [71] viridisLite_0.3.0 rprojroot_1.3-2 raster_3.0-12 bookdown_0.17 pkgload_1.0.2 [76] processx_3.4.1 glue_1.3.1 magrittr_1.5 htmlwidgets_1.5.1 curl_4.3 [81] lubridate_1.7.4 parallel_3.6.1 lifecycle_0.1.0 rgdal_1.4-8 htmltools_0.4.0 [86] RSQLite_2.2.0 crayon_1.3.4 leaflet_2.0.3 desc_1.2.0 rstudioapi_0.10 [91] gtable_0.3.0 colorspace_1.4-1 fortunes_1.5-4 ellipsis_0.3.0 cellranger_1.1.0 [96] fansi_0.4.1 assertable_0.2.7 fastmap_1.0.1 ps_1.3.0 bit_1.1-15.2 [101] broom_0.5.4 rmarkdown_2.1 crosstalk_1.0.0 iterators_1.0.12 generics_0.0.2 [106] haven_2.2.0 evaluate_0.14 rlang_0.4.3 backports_1.1.5 > > sf::sf_extSoftVersion() GEOS GDAL proj.4 GDAL_with_GEOS USE_PROJ_H "3.6.1" "2.2.3" "4.9.3" "true" "false" > > rgdal::rgdal_extSoftVersion() GDAL GDAL_with_GEOS PROJ.4 sp "2.2.3" "TRUE" "4.9.3" "1.3-2" ```

We currently worked around the problem by avoiding a CRS object in the specific case, but clearly the above problem is not wanted.

[Canderson156]

Hi all. I'm currently experiencing this issue, and the conversation here is a bit over my head. Would anyone consider providing a simplified explanation of what one should do when experiencing this problem? I'm having this issue when trying to use a function from gdalUtils, and I found another user recently posted the same issue when using RQGIS3.

https://stackoverflow.com/questions/62141199/r-package-rqgis3-cannot-find-proj-db

[jeroen]

@Canderson156 a workaround is to Sys.unsetenv("PROJ_LIB").

@rsbivand Why is rgdal still setting the PROJ_LIB environment variable? I thought this was no longer needed now that we are shipping PROJ6 on Windows, which sets the data-dir via the C api?

> Sys.getenv("PROJ_LIB")
[1] ""

> library(rgdal)
Loading required package: sp
rgdal: version: 1.5-10, (SVN revision 1006)
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 3.0.4, released 2020/01/28
Path to GDAL shared files: C:/Program Files/R/R-4.0.0/library/rgdal/gdal
GDAL binary built with GEOS: TRUE 
Loaded PROJ runtime: Rel. 6.3.1, February 10th, 2020, [PJ_VERSION: 631]
Path to PROJ shared files: C:/Program Files/R/R-4.0.0/library/rgdal/proj
Linking to sp version:1.4-2
To mute warnings of possible GDAL/OSR exportToProj4() degradation,
use options("rgdal_show_exportToProj4_warnings"="none") before loading rgdal.

> Sys.getenv("PROJ_LIB")
[1] "C:/Program Files/R/R-4.0.0/library/rgdal/proj"

[jgrn307]

Hey folks, I finally have some time to work on this -- @rsbivand and @edzer can you let me know if the OLD_PROJ_LIB suggestion was implemented? If so, I'll try to run some fixes in the next few days to resolve this.

[rsbivand]

For rgdal, please see https://r-forge.r-project.org/scm/viewvc.php/pkg/R/AAA.R?view=markup&root=rgdal, and give me something, a reprex, to go on. If people can halp, I can commit trial versions to R-forge which will be built for Windows (but I'm not sure if R 4.0, if not R 4.0, I can build locally and post here). @jeroen I could try to set the PROJ datadir through the API, but this still needs conditioning on OS, PROJ version, and whether users are running the CRAN binary in an OSGeo setting, that is do not want anything set but want to use the existing PROJ_LIB.

[jeroen]

Yes you need to condition on proj version.

I think the best convention is to use the PROJ_LIB variable only for proj4 data files, and use the C api when you need to load to proj6 data files. That way packages using proj4 and proj6 can happily live together, without conflicting.

[rsbivand]

As I suspected, use of proj_context_set_search_paths() is is non-trivial. In rgdal, all use of PROJ uses non-default (apparently thread-safe) contexts as in rgeos. However, in rgeos, the created context is held in an external pointer, passed under-the-hood to each call out to GEOS (sf does the same). In rgdal, contexts are created and destroyed on-the-fly, so subsequent calls to PROJ functions may happen in a different context (and search paths are set by context; different contexts have their own view of the world, error handling etc.). sf only uses PJ_DEFAULT_CTX. Either I can move to sf and only use the PROJ default context, or I can create a PROJ context on package load, destroy on unload, and pass it as an external pointer every time code calls out to PROJ, after editing all PROJ context use conditioning on PROJ version. Maybe I can do this, but not soon or without some help requiring knowledge of PROJ code and its curiosities.

[rsbivand]

rgdal rev 1008 committed to R-forge; if it builds, install.packages("rgdal", repos="http://R-Forge.R-project.org") should provide a Windows binary for the R Windows version on R-Forge, which appears to be 3.6.

The changes implemented are to switch (for now) to the default context for all code using contexts except network code only active for PROJ >= 7, to add a C++ function setting the path to the package proj data directory (not yet exposed to the user) and use it instead of overwriting PROJ_LIB in PROJ 6. Maybe the additional startup message is at the moment unhelpful, but I'm coding on Fedora and only trying Windows separately.

Feedback welcome, once R-Forge completes trying to build and check rev. 1008. I can try to put a 4.0-built Windows binary up later if someone wants one.

[rsbivand]

rgdal Windows binary 1.5-11.zip online at http://spatial.nhh.no/R/rgdal/rgdal_1.5-11.zip. Feedback essential quickly; no feedback, no adaptation to suit unresponsive users. It is important that standardised use cases be provided, otherwise we can't even check what is or is not going on. Do we know which PROJ thread is set? I've modified rgdal (rev 1008) to only use the default context, I can get and set the path apparently leaving PROJ_LIB untouched (probably the use case???), but does this work in the multiple Windows settings? Using QGIS at the same time? Using RQGIS? What context and package load order of which packages? How will this play with PROJ 7?

[jeroen]

I installed the latest rgdal on Windows from source using:

remotes::install_github("rforge/rgdal/pkg")

And that seems to work. It no longer sets PROJ_LIB on Windows, and still passes CMD check.

[rsbivand]

Great, thanks! Now what we need is confirmation from the users of RQGIS, etc., that our guess that this works checks out in their use cases.

I'm still concerned that there are OSGeo4W users who really want their stack used in R Windows packages to align shared data directories with the versions they are using in other programs, but they need to make a reproducible case.

[lbusett]

@rsbivand

I tested the gdalUtils example at the beginning of this thread after installing the rforge version, and it works without a glitch (not resetting the PROJ_LIB). Note however that also the CRAN version works no problem on my system, probably because also my OSGEO GDAL is also 3.0.4. I'd guess this should be tested on a Win machine with a 2.X GDAL.

[rsbivand]

The CRAN rgdal is also built with the current https://github.com/rwinlib/gdal3 versions, and no use of gdal2 is expected, so users of rgdal 1.4-8 on Windows should simply update the package. The CRAN Windows binary is using the same PROJ version coincidentally, but once I release 1.5-11 or later, the C API will be used, rather than setting PROJ_LIB for PROJ 6 only. PROJ 7 is harder, and neither sf nor rgdal handle multiple paths yet (see the final part of http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html).

[lovalery]

Dear all, Sorry for the delay. I just updated rgdal from source (using install.packages("rgdal", repos="http://R-Forge.R-project.org") and just tested RQGIS3 but the problem is still the same : when opening the application, R returns proj_create_from_database: Cannot find proj.db.

So there's probably still something to fix. I am at your disposal if you would like further information or if you would like me to carry out additional tests to help you find the solution.

Please find below an excerpt of the code:

> # setting the environment, i.e. specify all the paths necessary to run QGIS from within R
> set_env(root="C:/Program Files/QGIS 3.12")

$root
[1] "C:/Program Files/QGIS 3.12"

$qgis_prefix_path
[1] "C:/Program Files/QGIS 3.12/apps/qgis"

$python_plugins
[1] "C:/Program Files/QGIS 3.12/apps/qgis/python/plugins"

$platform
[1] "Windows"

> # opening the application
> open_app()
proj_create_from_database: Cannot find proj.db
proj_create_from_database: Cannot find proj.db

Here is my sessionInfo()

> sessionInfo()

R version 3.6.3 (2020-02-29)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 8.1 x64 (build 9600)

Matrix products: default

locale:
[1] LC_COLLATE=French_France.1252  LC_CTYPE=French_France.1252    LC_MONETARY=French_France.1252 LC_NUMERIC=C                  
[5] LC_TIME=French_France.1252    

attached base packages:
[1] tools     stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] sf_0.9-4          RQGIS3_1.0.1.9000 reticulate_1.16   maptools_1.0-1    link2GI_0.4.3     glue_1.4.1        listviewer_3.0.0  raster_3.1-5     
 [9] rgeos_0.5-3       rgdal_1.5-11      sp_1.4-2         

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.4.6       lattice_0.20-41    prettyunits_1.1.1  class_7.3-15       ps_1.3.3           assertthat_0.2.1   rprojroot_1.3-2   
 [8] digest_0.6.25      R6_2.4.1           backports_1.1.8    e1071_1.7-3        pillar_1.4.4       rlang_0.4.6        rstudioapi_0.11   
[15] callr_3.4.3        Matrix_1.2-18      desc_1.2.0         devtools_2.3.0     readr_1.3.1        stringr_1.4.0      foreign_0.8-75    
[22] htmlwidgets_1.5.1  compiler_3.6.3     xfun_0.15          pkgconfig_2.0.3    pkgbuild_1.0.8     htmltools_0.5.0    tidyselect_1.1.0  
[29] tibble_3.0.1       roxygen2_7.1.0     codetools_0.2-16   fansi_0.4.1        crayon_1.3.4       dplyr_1.0.0        withr_2.2.0       
[36] grid_3.6.3         jsonlite_1.6.1     lifecycle_0.2.0    DBI_1.1.0          magrittr_1.5       units_0.6-7        KernSmooth_2.23-16
[43] cli_2.0.2          stringi_1.4.6      fs_1.4.1           remotes_2.1.1      testthat_2.3.2     xml2_1.3.2         ellipsis_0.3.1    
[50] generics_0.0.2     vctrs_0.3.1        purrr_0.3.4        hms_0.5.3          parallel_3.6.3     processx_3.4.2     pkgload_1.1.0 

[rsbivand]

@lovalery you should also cite sf_extSoftVersion() and rgdal_extSoftVersion(). Also from the rgdal attach messages, which revision?

[lovalery]

@rsbivand below is the requested information. At your disposal, to send you additional information if needed.

> sf_extSoftVersion()
          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H 
       "3.8.0"        "3.0.4"        "6.3.1"         "true"         "true" 

> rgdal_extSoftVersion()
          GDAL GDAL_with_GEOS           PROJ             sp 
       "3.0.4"         "TRUE"        "6.3.1"        "1.4-2" 

> library(rgdal)
rgdal: version: 1.5-11, (SVN revision (unknown))
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 3.0.4, released 2020/01/28
Path to GDAL shared files: C:/Users/toto/Documents/R/win-library/3.6/rgdal/gdal
GDAL binary built with GEOS: TRUE 
Loaded PROJ runtime: Rel. 6.3.1, February 10th, 2020, [PJ_VERSION: 631]
Path to PROJ shared files: C:/Users/toto/Documents/R/win-library/3.6/rgdal/proj
Linking to sp version:1.4-2
To mute warnings of possible GDAL/OSR exportToProj4() degradation,
use options("rgdal_show_exportToProj4_warnings"="none") before loading rgdal.

[rsbivand]

Thanks. In which order are the packages loaded? With the rgdal release you seem to be running, use is made of rgdal:::set_proj_search_paths() calling proj_context_set_search_paths() in PROJ (for PROJ 6; the function is not present in <6, in 7 it takes a list of paths). The setting is called in rgdal:::load_stuff() if PROJis6ormore() is TRUE and file.exists(system.file("proj/nad.lst", package = "rgdal")[1]) is TRUE. If PROJis6ormore() is FALSE, Sys.setenv() will be used as before.

[lovalery]

I'm not sure if I'm answering your question correctly. So feel free to ask me again if there is any information missing.

The order the libraries are loaded is as follows:

library(sp)
library(rgdal)
library(rgeos)
library(raster)
library(listviewer)
library(glue)
library(tools)
library(link2GI)
library(maptools)
library(reticulate, lib.loc="C:/Users/toto/documents/RQGIS3")
library(RQGIS3, lib.loc="C:/Users/toto/Documents/RQGIS3")
library(sf)

[rsbivand]

Check Sys.getenv("PROJ_LIB") before starting, and after adding each package, to see where changes occur.

[lovalery]

You will find below the whole sequence: since the start of R, I loaded each library and executed the Sys.getenv("PROJ_LIB") command between each added package. I finished by defining the environment parameters for RQGIS3 then trying to open the application... with always the same error message at the end.

> Sys.getenv("PROJ_LIB")
[1] ""

> library(sp)
> Sys.getenv("PROJ_LIB")
[1] ""

> library(rgdal)
rgdal: version: 1.5-11, (SVN revision (unknown))
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 3.0.4, released 2020/01/28
Path to GDAL shared files: C:/Users/toto/Documents/R/win-library/3.6/rgdal/gdal
GDAL binary built with GEOS: TRUE 
Loaded PROJ runtime: Rel. 6.3.1, February 10th, 2020, [PJ_VERSION: 631]
Path to PROJ shared files: C:/Users/toto/Documents/R/win-library/3.6/rgdal/proj
Linking to sp version:1.4-2
To mute warnings of possible GDAL/OSR exportToProj4() degradation,
use options("rgdal_show_exportToProj4_warnings"="none") before loading rgdal.
> Sys.getenv("PROJ_LIB")
[1] ""

> library(rgeos)
rgeos version: 0.5-3, (SVN revision 634)
 GEOS runtime version: 3.8.0-CAPI-1.13.1 
 Linking to sp version: 1.4-1 
 Polygon checking: TRUE 
> Sys.getenv("PROJ_LIB")
[1] ""

> library(raster)
> Sys.getenv("PROJ_LIB")
[1] ""

> library(listviewer)
> Sys.getenv("PROJ_LIB")
[1] ""

> library(glue)
Attachement du package : ‘glue’
The following object is masked from ‘package:raster’:
    trim
> Sys.getenv("PROJ_LIB")
[1] ""

> library(tools)
> Sys.getenv("PROJ_LIB")
[1] ""

> library(link2GI)
> Sys.getenv("PROJ_LIB")
[1] ""

> library(maptools)
Checking rgeos availability: TRUE
> Sys.getenv("PROJ_LIB")
[1] ""

> library(reticulate, lib.loc="C:/Users/toto/documents/RQGIS3")
> Sys.getenv("PROJ_LIB")
[1] ""

> library(RQGIS3, lib.loc="C:/Users/toto/Documents/RQGIS3")
> Sys.getenv("PROJ_LIB")
[1] ""

> library(sf)
Linking to GEOS 3.8.0, GDAL 3.0.4, PROJ 6.3.1
> Sys.getenv("PROJ_LIB")
[1] ""

> # set the environment, i.e. specify all the paths necessary to run QGIS from within R
> set_env(root="C:/Program Files/QGIS 3.12")
$root
[1] "C:/Program Files/QGIS 3.12"

$qgis_prefix_path
[1] "C:/Program Files/QGIS 3.12/apps/qgis"

$python_plugins
[1] "C:/Program Files/QGIS 3.12/apps/qgis/python/plugins"

$platform
[1] "Windows"

> Sys.getenv("PROJ_LIB")
[1] ""

> # opening of the application :
> open_app()
proj_create_from_database: Cannot find proj.db
proj_create_from_database: Cannot find proj.db

[rsbivand]

So none of the attached packages sets PROJ_LIB. The problem is that open_app() provokes two messages from libproj called by something unidentified. This does not involve the problem addressed here, which was that PROJ_LIB was defined before R was started, and was changed when using CRAN binaries of sf or rgdal to something else. Here, sf and rgdal are respecting the previous value of the environment variable. @jannes-m any ideas?

[jannes-m]

AFAIK the message does not affect the usage of QGIS algorithms, see https://github.com/r-spatial/RQGIS3/issues/20

[lovalery]

@jannes-m Thank you for your answer. Yes, I already know this thread. But the problem is not solved at all. Please find below a detailed example with the "polygonize" tool of QGIS (rgdal) which shows that when using this function R returns the following two error messages : proj_create_from_wkt: Cannot find proj.db and proj_identify: Cannot find proj.db. And clearly the function doesn't produce any results :

$OUTPUT
[1] "D:/test/seg_poly2.shp"

Error in FUN(X[[i]], ...) : 
  Unfortunately, QGIS did not produce: D:/test/seg_poly2.shp

Here is the detailed example :

> # 1- I set the environment for specifying all the paths necessary to run QGIS from within R
> set_env(root="C:/Program Files/QGIS 3.12")
$root
[1] "C:/Program Files/QGIS 3.12"

$qgis_prefix_path
[1] "C:/Program Files/QGIS 3.12/apps/qgis"

$python_plugins
[1] "C:/Program Files/QGIS 3.12/apps/qgis/python/plugins"

$platform
[1] "Windows"

> # 2- I try to open the application and R return the double error message
> open_app()
proj_create_from_database: Cannot find proj.db
proj_create_from_database: Cannot find proj.db

> # 3- I fill the mandatory parameters for the function "polygonize"
> params<-get_args_man(alg="gdal:polygonize")
> params$INPUT<-"pz5etr_bs_1500_0.9_0.1.tif"
> params$BAND<-1
> params$FIELD<-"DN"
> params$EIGHT_CONNECTEDNESS<-TRUE
> params$OUTPUT<-file.path("D:/test", "seg_poly2.shp")

> # 4- I run the function.... and gets the two error messages again.
> seg_poly <- run_qgis(alg = "gdal:polygonize",
+                 params = params,
+                 load_output = TRUE)
proj_create_from_wkt: Cannot find proj.db
proj_identify: Cannot find proj.db
$OUTPUT
[1] "D:/test/seg_poly2.shp"

Error in FUN(X[[i]], ...) : 
  Unfortunately, QGIS did not produce: D:/test/seg_poly2.shp

[jannes-m]

But the problem here are not the proj messages. This is a QGIS custom application bug, which we have come across several times and which is hard to debug, since QGIS returns the path to an output file which in fact has never been created. Sometimes it helped to use another file format, e. g. gpkg instead of shp but I don't think this will solve the problem here and was in any case just a crude solution. Sorry that I cannot provide a better answer.

[lovalery]

I confirm that changing .shp to .gpkg does not solve the problem (cf below). I thought it was a routine problem with the PROJ.LIB pathway. From reading you, I can see the problem is hard to fix. However, if you or someone else could try to fix this bug in the next weeks/months, that would be great :-) Of course I could try myself but it's way beyond my skills. Anyway, if you need someone to do some tests, I'm at your disposal to do them.

> # Test with .gpkg instead of .shp for the output file
> params<-get_args_man(alg="gdal:polygonize")
> params$INPUT<-"pz5etr_bs_1500_0.9_0.1.tif"
> params$BAND<-1
> params$FIELD<-"DN"
> params$EIGHT_CONNECTEDNESS<-TRUE
> params$OUTPUT<-file.path("D:/test", "seg_poly2.gpkg")
> seg_poly <- run_qgis(alg = "gdal:polygonize",
+                 params = params,
+                 load_output = TRUE)
$OUTPUT
[1] "D:/test/seg_poly2.gpkg"

Error in FUN(X[[i]], ...) : 
  Unfortunately, QGIS did not produce: D:/test/seg_poly2.gpkg