gdalUtils and gdal 3.0.2 : ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db

[mgageo]

Hello, I use gdaUtils with the gdal version from QGIS : https://www.qgis.org/fr/site/forusers/download.html https://qgis.org/downloads/QGIS-OSGeo4W-3.10.0-2-Setup-x86_64.exe

On a command like

ogr2ogr('rgdal.geojson','rgdal.json',t_srs="EPSG:2154",verbose=TRUE) Checking gdal_installation... Scanning for GDAL installations... Checking the gdalUtils_gdalPath option... GDAL version 3.0.2 GDAL command being used: "C:\Program Files\QGIS 3.10\bin\ogr2ogr.exe" -t_srs "EPSG:2154" "rgdal.json" "rgdal.geojson" [1] "ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db" "ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db"
[3] "ERROR 1: PROJ: proj_create_from_database: Cannot find proj.db" "ERROR 1: Failed to process SRS definition: EPSG:2154"

I have an other version of gdal from http://download.gisinternals.com/sdk/downloads/release-1911-x64-gdal-3-0-mapserver-7-4.zip in the directory d:\gdal30

I replace the system call "system(cmde, intern = TRUE)" by a small script, cf https://framagit.org/mgageo/osmdata/blob/master/scripts/ogr2ogr.R and it's good

[lbusett]

Hi @mgageo and @jgrn307 ,

I'd guess this is the same problem/related to what I just reported here:

https://github.com/r-spatial/discuss/issues/31

, and that I was going to report also here.

HTH,

Lorenzo

[jgrn307]

If you run this line (from above) from Windows CMD line, what happens?

"C:\Program Files\QGIS 3.10\bin\ogr2ogr.exe" -t_srs "EPSG:2154" "rgdal.json" "rgdal.geojson"

[mgageo]

D:\tmp>"C:\Program Files\QGIS 3.10\bin\ogr2ogr.exe" -t_srs "EPSG:2154" "rgdal.json" "rgdal.geojson" ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db ERROR 1: PROJ: proj_create_from_database: Cannot find proj.db ERROR 1: Failed to process SRS definition: EPSG:2154

D:\tmp>

[lbusett]

@mgageo @jgrn307

A way to fix the issue in https://github.com/r-spatial/discuss/issues/31 is to reassign properly (or set in the first place if it is not set) the "PROJ_LIB" environment variable on the fly, before issuing any GDAL command that requires it. I implemented a possible fix (at the moment, only valid for gdalwarp), here:

https://github.com/lbusett/gdalUtils/commit/5d6a1aeb6a2c46acda3e07f7b7752db22a5b2f59

@mgageo

Could you try if it works also for you? You can do that by installing my fork from github:

remotes::install_github("lbusett/gdalUtils", ref = "feature/projlibpath")
library(gdalUtils)

and then trying commands:

src_dataset <- system.file("external/tahoe_highrez.tif", package="gdalUtils")
# Command-line gdalwarp call:
gdalwarp(src_dataset,dstfile="tahoe_highrez_utm11.tif",
        t_srs='+proj=utm +zone=11 +datum=WGS84',output_Raster=TRUE,
        overwrite=TRUE,verbose=TRUE)

and see if it works.

If this seems ok/viable, the fix can be easily extended to other gdalUtils functions.

[jgrn307]

Ok, trying to get a TLDR here -- basically either rgdal or gdalUtils is causing the PROJ_LIB to change, which breaks some of the functionality?

1) Is this an accurate reading and, if so, which is changing the lib (rgdal, gdalUtils, something else)? 2) What is the TLDR fix you are proposing? Before the GDAL CMD is run, preserve the existing PROJ_LIB, run the gdal command,, and then set it back to the original PROJ_LIB?
3) Is this Windows specific?

If so, I think the simplest solution is to modify the GDAL command so this gets pre/post-appended to the command that is actually run (so I can universally change it in gdal_cmd_builder instead of having to tweak each command individually).

--j

[lbusett]

Hi @jgrn307 , I'll reply inline:

Ok, trying to get a TLDR here -- basically either rgdal or gdalUtils is causing the PROJ_LIB to change, which breaks some of the functionality?

1. Is this an accurate reading and, if so, which is changing the lib (rgdal, gdalUtils, something else)?

Yes, that is correct. The "culprit" here is rgdal, which namespace is imported by gdalUtils, so that as soon as the library is loaded (or even a function is called such as in gdalUtils::ogr2ogr(src_datasource_name,...), the environment variable is immediately modified (see https://github.com/cran/rgdal/blob/848067a0251323829dd776cf0789d1410cb77b3b/R/AAA.R#L14).

What is the TLDR fix you are proposing? Before the GDAL CMD is run, preserve the existing PROJ_LIB, run the gdal command,, and then set it back to the original PROJ_LIB?

The fix relies on introducing the new function set_projlibpath to do exactly that, using the on.exit mechanism to restore the previous value after each system call (to avoid breaking rgdal/sf functionality instead)

Is this Windows specific?

I encountered this only on Windows. It is due to the fact that at the moment the Windows rgdal/sf package binaries are linked to GDAL2/PROJ4 while the OSGEO installer/updater provides GDAL3/PROJ6 on new installs/updates. This leads gdalUtils to often rely on GDAL3 binaries, while rgdal forces the use of a PROJ_LIB folder related to a GDAL2 installation. I do not work on MAC, so I do not know if it is also affected. Consider however that the proposed solution, though currently being "applied" only on Windows OS, could be used with no problems on all OS, simply removing the Sys.info()[["sysname"]] == "Windows" in the new function.

If so, I think the simplest solution is to modify the GDAL command so this gets pre/post-appended to the command that is actually run (so I can universally change it in gdal_cmd_builder instead of having to tweak each command individually).

This is a good idea. However, consider that this would require moving also the system call within the gdal_cmd_builder function, instead than within the single gdalUtils functions (This because set_projlibpath needs to be called from within the function making the system call, in order to properly re-set the environment variable on exit - see https://yihui.org/en/2017/12/on-exit-parent/). I tested it, and it seems to work. I could prepare a PR if you wish so.

HTH,

Lorenzo

[mgageo]

[mgageo]

and after OK

[lbusett]

@mgageo

Thaks for testing. Could you try installing again from CRAN, test the following:

src_datasource_name <- system.file("external/tahoe_highrez_training.shp", package="gdalUtils")
dst_datasource_name <- paste(tempfile(),".shp",sep="")
# reproject the input to mercator
ogr2ogr(src_datasource_name,dst_datasource_name,t_srs="EPSG:3395",verbose=TRUE)

, and then re-test after reinstalling again from my fork on github?

[mgageo]

The version from CRAN

[mgageo]

After install from github

[lbusett]

Strange... it works for me...

could you please share your session info?

devtools::session_info()

[mgageo]

devtools::session_info() - Session info -------------------------------------------------------------------------------------------------------- setting value version R version 3.6.2 (2019-12-12) os Windows 10 x64 system x86_64, mingw32 ui RStudio language (EN) collate French_France.1252 ctype French_France.1252 tz Europe/Paris date 2020-01-23 - Packages ------------------------------------------------------------------------------------------------------------ package version date lib source assertthat 0.2.1 2019-03-21 [1] CRAN (R 3.6.1) backports 1.1.5 2019-10-02 [1] CRAN (R 3.6.1) callr 3.4.0 2019-12-09 [1] CRAN (R 3.6.2) cli 2.0.1 2020-01-08 [1] CRAN (R 3.6.2) codetools 0.2-16 2018-12-24 [2] CRAN (R 3.6.2) crayon 1.3.4 2017-09-16 [1] CRAN (R 3.6.1) desc 1.2.0 2018-05-01 [1] CRAN (R 3.6.1) devtools 2.2.1 2019-09-24 [1] CRAN (R 3.6.1) digest 0.6.23 2019-11-23 [1] CRAN (R 3.6.2) ellipsis 0.3.0 2019-09-20 [1] CRAN (R 3.6.1) fansi 0.4.1 2020-01-08 [1] CRAN (R 3.6.2) foreach 1.4.7 2019-07-27 [1] CRAN (R 3.6.1) fs 1.3.1 2019-05-06 [1] CRAN (R 3.6.1) gdalUtils 2.0.2.4 2020-01-23 [1] Github (lbusett/gdalUtils@709cc02) glue 1.3.1 2019-03-12 [1] CRAN (R 3.6.1) iterators 1.0.12 2019-07-26 [1] CRAN (R 3.6.1) lattice 0.20-38 2018-11-04 [2] CRAN (R 3.6.2) magrittr 1.5 2014-11-22 [1] CRAN (R 3.6.1) memoise 1.1.0 2017-04-21 [1] CRAN (R 3.6.1) pkgbuild 1.0.6 2019-10-09 [1] CRAN (R 3.6.2) pkgload 1.0.2 2018-10-29 [1] CRAN (R 3.6.1) prettyunits 1.1.0 2020-01-09 [1] CRAN (R 3.6.2) processx 3.4.1 2019-07-18 [1] CRAN (R 3.6.1) ps 1.3.0 2018-12-21 [1] CRAN (R 3.6.1) R.methodsS3 1.7.1 2016-02-16 [1] CRAN (R 3.6.0) R.oo 1.23.0 2019-11-03 [1] CRAN (R 3.6.1) R.utils 2.9.2 2019-12-08 [1] CRAN (R 3.6.1) R6 2.4.1 2019-11-12 [1] CRAN (R 3.6.2) raster 3.0-7 2019-09-24 [1] CRAN (R 3.6.1) Rcpp 1.0.3 2019-11-08 [1] CRAN (R 3.6.1) remotes 2.1.0 2019-06-24 [1] CRAN (R 3.6.1) rgdal 1.4-8 2019-11-27 [1] CRAN (R 3.6.1) rlang 0.4.2 2019-11-23 [1] CRAN (R 3.6.2) rprojroot 1.3-2 2018-01-03 [1] CRAN (R 3.6.1) rstudioapi 0.10 2019-03-19 [1] CRAN (R 3.6.1) sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 3.6.1) sp 1.3-2 2019-11-07 [1] CRAN (R 3.6.1) testthat 2.3.1 2019-12-01 [1] CRAN (R 3.6.2) usethis 1.5.1 2019-07-04 [1] CRAN (R 3.6.1) withr 2.1.2 2018-03-15 [1] CRAN (R 3.6.1) [1] C:/Users/marcg/Documents/R/win-library/3.6 [2] C:/Program Files/R/R-3.6.2/library

  | >

[lbusett]

@mgageo

seems that a reset of the r session is required. Could you retry after restarting the rsession/restarting rstudio, if you did not do that already?

[mgageo]

After a restart I have the popup but

[lbusett]

That should mean that it worked. What do you get from

file.exists(dst_datasource_name)

?

[mgageo]

file.exists(dst_datasource_name) [1] TRUE and

[lbusett]

Seems therefore that the "fix" would work.

@jgrn307 What do you think of the solution introduced in https://github.com/lbusett/gdalUtils/commit/709cc02 ?

I's based on moving the system calls within cmd_builder. Implementing it in all gdalUtils functions would be straightforward. I could make a PR if you wish.

Lorenzo

[jgrn307]

Let me work on it this weekend and get back to you!

On Fri, Jan 24, 2020, 8:21 AM Lorenzo Busetto notifications@github.com<mailto:notifications@github.com> wrote:

Seems therefore that the "fix" would work.

@jgrn307https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjgrn307&data=01%7C01%7Cjgreenberg%40unr.edu%7C10145e7aa5be401cc31308d7a0e97112%7C523b4bfc0ebd4c03b2b96f6a17fd31d8%7C1&sdata=UkW7XWK6EQ8N6XjoXnzHyurNOB6V6YPaZWly57WfS00%3D&reserved=0 What do you think of the solution introduced in lbusett@709cc02https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flbusett%2FgdalUtils%2Fcommit%2F709cc02&data=01%7C01%7Cjgreenberg%40unr.edu%7C10145e7aa5be401cc31308d7a0e97112%7C523b4bfc0ebd4c03b2b96f6a17fd31d8%7C1&sdata=pT7OjxMIuE8YuxLQLsENHyokRicVbtf%2FhpXSNSM5QM8%3D&reserved=0 ?

I's based on moving the system calls within cmd_builder. Implementing it in all gdalUtils functions would be straightforward. I could make a PR if you wish.

Lorenzo

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgearslaboratory%2FgdalUtils%2Fissues%2F24%3Femail_source%3Dnotifications%26email_token%3DAAFSITK3G7H3WE62JDT2LQ3Q7MIP3A5CNFSM4JYR6732YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ3J4OA%23issuecomment-578199096&data=01%7C01%7Cjgreenberg%40unr.edu%7C10145e7aa5be401cc31308d7a0e97112%7C523b4bfc0ebd4c03b2b96f6a17fd31d8%7C1&sdata=kymDqDBBZUF2U0nlCs3jz0hAjCuKqBKx2DZ6hqtm16I%3D&reserved=0, or unsubscribehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAFSITOXBXZEDOCCPPRM3FTQ7MIP3ANCNFSM4JYR673Q&data=01%7C01%7Cjgreenberg%40unr.edu%7C10145e7aa5be401cc31308d7a0e97112%7C523b4bfc0ebd4c03b2b96f6a17fd31d8%7C1&sdata=1oq6XLeM9WkCLMnISnsY2GBPJN4UiOf4hKSZf9M46Y8%3D&reserved=0.

[jgrn307]

So this REALLY seems like an rgdal problem the more I look into it. Let me see if Bivand is willing to tweak his code a bit. It seems like if someone runs rgdal for ANY reason before gdalUtils is run, the environment variable is "lost" in a way that is going to make it very hard to recover. Am I missing something?

[rsbivand]

No, it is a problem for those using Windows or OSX binary installs of CRAN packages, and the same problem will exist for all other packages not building from source locally (I'm not sure which, but rgdal and sf certainly, probably vapour). There is no easy way to provide half-binary Windows builds (OSX the same), depending on DLLs that the user has to install first. Yes, it is a problem, but no, there is no rgdal specific resolution. If we tweak rgdal on Windows to respect an existing PROJ_LIB environment variable, we will very quickly find that say an OSGeo4W PROJ or GDAL is a different version from the version used to build rgdal.dll, so will expect say proj/epsg not proj/proj.db. Protecting all rgdal functions on CRAN binary builds by setting and resetting PROJ_LIB (and the GDAL equivalent) on before and after each .Call() is overkill, I think.

If gdalUtils required https://github.com/rwinlib/gdal2 and that built the binaries too, all the Windows CRAN binary users would be on the same page. Or gdalUtils could wrap the use in sf of the utilities as library functions.

[jgrn307]

Roger, here's a potential very easy tweak you could make that I could easily use with gdalUtils: before you set the environment variable(s), can you simply create/set a new environment variable named something like "OLD_PROJ_LIB" and throw the "old" environment variable in there? This would then let me grab it easily (as long as I know the name of the variable) and I can do a set/reset back. Would you be willing to make this change? Just tack that right above the line you do the environment variable change?

[jgrn307]

@edzer I think this would need to be done with sf also.

[jgrn307]

Let's move this to: https://github.com/r-spatial/discuss/issues/31

[JepsonNomad]

Full disclosure: I'm not using gdalUtils. Not trying to hijack this discussion, but I've been doing some searching after running into an apparently related issue on Mac and I thought it might be relevant for your open issue. I'm running OSX 10.15.6 Catalina. Using sf::st_wrap_dateline() was causing me problems when rgdal is loaded:

> library(rgdal)
Loading required package: sp
rgdal: version: 1.5-16, (SVN revision 1050)
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 2.4.2, released 2019/06/28
Path to GDAL shared files: /Library/Frameworks/R.framework/Versions/3.6/Resources/library/rgdal/gdal
GDAL binary built with GEOS: FALSE 
Loaded PROJ runtime: Rel. 5.2.0, September 15th, 2018, [PJ_VERSION: 520]
Path to PROJ shared files: /Library/Frameworks/R.framework/Versions/3.6/Resources/library/rgdal/proj
Linking to sp version:1.4-2
Overwritten PROJ_LIB was /Library/Frameworks/R.framework/Versions/3.6/Resources/library/rgdal/proj
Warning message:
package ‘rgdal’ was built under R version 3.6.2 
> library(rnaturalearth)
> library(sf)
Linking to GEOS 3.8.1, GDAL 3.1.2, PROJ 7.1.1
> library(ggplot2)
> Sys.getenv("PROJ_LIB")
[1] "/Library/Frameworks/R.framework/Versions/3.6/Resources/library/rgdal/proj"
> sf_extSoftVersion()
          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H 
       "3.8.1"        "3.1.2"        "7.1.1"         "true"         "true" 
> 
> robin = st_crs("+proj=robin +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs")
> 
> ROI = ne_countries(returnclass = 'sf') %>%
+   st_combine() 
> 
> #Please check the math on this part.
> #KM/earth circumference * degrees in circle
> buffer_in_km <- 500
> buffer_as_arc_degrees<- buffer_in_km/40075*360
> 
> coastalWaters = ROI %>%
+   st_buffer(buffer_as_arc_degrees)  %>% 
+   st_wrap_dateline()
dist is assumed to be in decimal degrees (arc_degrees).
Warning messages:
1: In st_buffer.sfc(., buffer_as_arc_degrees) :
  st_buffer does not correctly buffer longitude/latitude data
2: In st_is_longlat(x) :
  bounding box has potentially an invalid value range for longlat data
> 
> ggplot() +
+   geom_sf(data = coastalWaters, fill = "lightblue", col = "transparent") +
+   geom_sf(data = ROI) +
+   coord_sf(crs = robin)
Error: node stack overflow
In addition: There were 50 or more warnings (use warnings() to see the first 50)
Error during wrapup: node stack overflow

Running warnings() resulted in this:

Warning messages:
1: In CPL_crs_from_input(x) :
  GDAL Error 1: PROJ: proj_create_from_database: Cannot find proj.db

... with the warning repeated 50 times before exhausting. However, if I repeat exactly the same script as above, but with library(rgdal) commented out, everything runs fine. Note however that Sys.getenv("PROJ_LIB") returns [1] "". My sessionInfo() result is:

> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Catalina 10.15.6

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] ggplot2_3.3.0       sf_0.9-5            rnaturalearth_0.1.0 rgdal_1.5-16        sp_1.3-2           

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.3         rstudioapi_0.10    magrittr_1.5       units_0.6-5        munsell_0.5.0     
 [6] tidyselect_0.2.5   colorspace_1.4-1   lattice_0.20-38    R6_2.4.1           rlang_0.4.2       
[11] dplyr_0.8.3        tools_3.6.1        grid_3.6.1         gtable_0.3.0       KernSmooth_2.23-15
[16] e1071_1.7-3        DBI_1.0.0          withr_2.1.2        rgeos_0.5-5        class_7.3-15      
[21] assertthat_0.2.1   lifecycle_0.1.0    tibble_2.1.3       crayon_1.3.4       purrr_0.3.3       
[26] glue_1.3.1         compiler_3.6.1     pillar_1.4.3       scales_1.1.0       classInt_0.4-2    
[31] pkgconfig_2.0.3   

Also possibly relevant, from the Terminal:

JepsonNomad:bin <redacted>$ gdalinfo --version
GDAL 3.1.2, released 2020/07/07
JepsonNomad:bin <redacted>$ proj
Rel. 7.1.1, September 1st, 2020
usage: proj [-bdeEfiIlmorsStTvVwW [args]] [+opt[=arg] ...] [file ...]

[rsbivand]

@JepsonNomad Post a minimal reproducible example on R-sig-geo after subscribing if need be. This is almost certainly the wrong place to ask.