search

rsbivand / rgrass

Interpreted interface between the GRASS geographical information system and R
https://rsbivand.github.io/rgrass/
24 stars 8 forks source link

initGrass Error : XML content does not seem to be XML #57

Open remp48 opened 2 years ago

remp48 commented 2 years ago

Hello,

I installed GRASS GIS 8.0.2 and tried to open it via R 4.0.3 package rgrass7, but with the following command I got the error message : initGRASS("C:/Program Files/GRASS GIS 8.0", override = TRUE, gisDbase = "GRASS_TEMP", mapset = "PERMANENT", location = "test") Error : XML content does not seem to be XML: 'ontains DATABASE.LAYOUT.VERSION.MINOR = 0 whereas a number >= 2 is expected. It comes from another PROJ installation.' Also: Warning message: In system(cmd0, intern = TRUE) : running the command 'g.proj.exe --interface-description' returns a status of 5

Do you know what is the issue ?

Thank you very much

rsbivand commented 2 years ago

I cannot reproduce this with GRASS 8.0.1 and 8.0.2 Windows standalones on Windows 10 with R 4.2.0 and rgrass 0.2-10 and XML 3.99-0.9. Could you please run update.packages() and retry? I have a vague recollection of something like this being reported, and think that updating packages helped. R 4.0 is also now old, but try updating just the packages first. Also why use gisDbase= etc., try without first until you resolve the problem.

LordRudolf commented 2 years ago

I have absolutely the same error, and I could not find any answers in Stack Overflow.

I have installed Grass 8.2 on my Windows.

Also, not using gisDBase did not help: image

My R session Info:

R version 4.2.0 (2022-04-22 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19043)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.utf8  LC_CTYPE=English_United Kingdom.utf8   
[3] LC_MONETARY=English_United Kingdom.utf8 LC_NUMERIC=C                           
[5] LC_TIME=English_United Kingdom.utf8    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rgrass7_0.2-10  XML_3.99-0.10   mapview_2.11.0  RPostgres_1.4.4

loaded via a namespace (and not attached):
 [1] tidyselect_1.1.2   terra_1.5-21       purrr_0.3.4        sf_1.0-7          
 [5] lattice_0.20-45    colorspace_2.0-3   vctrs_0.4.1        generics_0.1.2    
 [9] htmltools_0.5.2    stats4_4.2.0       base64enc_0.1-3    utf8_1.2.2        
[13] blob_1.2.3         rlang_1.0.2        e1071_1.7-11       pillar_1.7.0      
[17] glue_1.6.2         DBI_1.1.2          sp_1.5-0           bit64_4.0.5       
[21] lifecycle_1.0.1    munsell_0.5.0      raster_3.5-15      htmlwidgets_1.5.4 
[25] codetools_0.2-18   fastmap_1.1.0      crosstalk_1.2.0    class_7.3-20      
[29] fansi_1.0.3        leafem_0.2.0       Rcpp_1.0.8.3       KernSmooth_2.23-20
[33] scales_1.2.0       classInt_0.4-3     satellite_1.0.4    leaflet_2.1.1     
[37] webshot_0.5.3      bit_4.0.4          hms_1.1.1          png_0.1-7         
[41] digest_0.6.29      dplyr_1.0.9        grid_4.2.0         rgdal_1.5-32      
[45] cli_3.3.0          tools_4.2.0        magrittr_2.0.3     proxy_0.4-26      
[49] tibble_3.1.7       crayon_1.5.1       pkgconfig_2.0.3    ellipsis_0.3.2    
[53] assertthat_0.2.1   R6_2.5.1           units_0.8-0        compiler_4.2.0
rsbivand commented 2 years ago

The error message:

Error : XML content does not seem to be XML: 'ontains DATABASE.LAYOUT.VERSION.MINOR = 0 whereas a number >= 2 is expected. It comes from another PROJ installation.'

and the equivalent in the screen shot (please don't post screen shots, copy&paste within a markdown block) shows what the problem is clearly. Neither state whether GRASS is OSGEO4W or Windows native standalone.

Both contributors have multiple PROJ installations, where the one being looked for by GRASS is not found. In the second case, the R session has a large number of loaded packages, so it isn't obvious which is the problem (setting PROJ_LIB without protecting it). Please try in turn (each in a in fresh session) to test which of the possible packages:

library(terra)
library(rgrass7)
initGRASS("C:/Program Files/GRASS GIS 8.0", tempdir(), override=TRUE)
library(raster)
library(rgrass7)
initGRASS("C:/Program Files/GRASS GIS 8.0", tempdir(), override=TRUE)
library(sf)
library(rgrass7)
initGRASS("C:/Program Files/GRASS GIS 8.0", tempdir(), override=TRUE)
library(rgdal)
library(rgrass7)
initGRASS("C:/Program Files/GRASS GIS 8.0", tempdir(), override=TRUE)

is not protecting PROJ_LIB.

rsbivand commented 2 years ago

Please confirm that terra 1.5-21 is the problem. The current release is 1.5-34, for which no error occurs for me.

rsbivand commented 2 years ago

The error message is from PROJ detecting a different version of proj.db than expected, code: https://github.com/OSGeo/PROJ/blob/019dad3524e434edd7a35cd87251ad4eaab08614/src/iso19111/factory.cpp#L480-L542

This means in this case that g.proj in GRASS is wrongly using a proj.db shipped with an R package, rather than that shipped with WinGRASS or OSGEO4W GRASS. I think all current R packages shipping proj.db do protect their reference to their version, so g.proj should only detect the one it itself shipped with.

rsbivand commented 2 years ago

The proj.db version in terra 1.5-21 is:

> library("RSQLite")
> db <- DBI::dbConnect(RSQLite::SQLite(), dbname="proj.db")
> md <- DBI::dbReadTable(db, "metadata")
> md
                             key                                                    value
1  DATABASE.LAYOUT.VERSION.MAJOR                                                        1
2  DATABASE.LAYOUT.VERSION.MINOR                                                        0
3                   EPSG.VERSION                                                  v10.008
4                      EPSG.DATE                                               2020-12-16
5                   ESRI.VERSION                                            ArcMap 10.8.1
6                      ESRI.DATE                                               2020-05-24
7                    IGNF.SOURCE https://geodesie.ign.fr/contenu/fichiers/IGNF.v3.1.0.xml
8                   IGNF.VERSION                                                    3.1.0
9                      IGNF.DATE                                               2019-05-24
10                    NKG.SOURCE   https://github.com/NordicGeodesy/NordicTransformations
11                   NKG.VERSION                                                    1.0.0
12                      NKG.DATE                                               2020-12-21

In terra 1.5-34 the same version (same Rtools PROJ version), but possibly now protected from version conflict. In GRASS 8.2.0 Windows standalone:

> md
                             key                                                                           value
1  DATABASE.LAYOUT.VERSION.MAJOR                                                                               1
2  DATABASE.LAYOUT.VERSION.MINOR                                                                               2
3                      EPSG.DATE                                                                      2022-02-13
4                   EPSG.VERSION                                                                         v10.054
5                      ESRI.DATE                                                                      2022-02-18
6                   ESRI.VERSION                                                                     ArcMap 12.9
7                      IGNF.DATE                                                                      2019-05-24
8                    IGNF.SOURCE https://raw.githubusercontent.com/rouault/proj-resources/master/IGNF.v3.1.0.xml
9                   IGNF.VERSION                                                                           3.1.0
10                      NKG.DATE                                                                      2020-12-21
11                    NKG.SOURCE                          https://github.com/NordicGeodesy/NordicTransformations
12                   NKG.VERSION                                                                           1.0.0
13                  PROJ.VERSION                                                                           9.0.0
14             PROJ_DATA.VERSION                                                                             1.9
rsbivand commented 2 years ago

On freshly installed R 4.1.3, back-installing terra 1.5-21, no error is reproducible.

LordRudolf commented 2 years ago

Thank you, @rsbivand, for your prompt response. Unfortunately, it is tough to track it down.

I did all the following:

Error : XML content does not seem to be XML: 'roj.db contains DATABASE.LAYOUT.VERSION.MINOR = 0 whereas a number >= 2 is expected. It comes from another PROJ installation.'

The only post with the same problem I find (besides this one) is this: https://gis.stackexchange.com/questions/421932/osgeo4w-ogr2ogr-script-with-filegdb-conversion-working-but-getting-a-weird-erro.

At this point, it seems that it is not directly related to the grass package. But I would appreciate advice on which GitHub group shall I post this issue to, as many technical terms mentioned are out of my scope of knowledge.

Maybe this could help: I tried to run the SQLite script you posted:

library("RSQLite") db <- DBI::dbConnect(RSQLite::SQLite(), dbname="proj.db") md <- DBI::dbReadTable(db, "metadata")

It did not work on my system as running the linemd <- DBI::dbReadTable(db, "metadata") I've got the following error: Error: no such table: metadata image

Or could it be related to QGIS I have installed? I saw other PROJ-related issues in the same context with QGIS. My QGIS version: image

fgashakamba commented 1 year ago

I had the exact same error message on a Windows11 machine. Then I installed standalone GRASS v8.2 and PROJ v9.2 software and set the PROJ_DATA environment variable as indicated in the installation manual. Now the error is gone when I run:

library(rgrass)
initGRASS(gisBase = "C:/Program Files/GRASS GIS 8.2", 
          home = tempdir(), 
          override = TRUE)

Instead, it has now been replaced by a warning with the message:

In system(syscmd, intern = intern, ignore.stderr = ignore.stderr,  :
  running command 'g.proj.exe -w' had status 1

I checked and rechecked to see whether there is any other loaded package that could be loading another proj.db as suggested by @rsbivand. I can't see any.

rsbivand commented 1 year ago

On Windows, the terra and GRASS binaries both install proj.db, and when terra is loaded, its version takes over PROJ_LIB. Please provide the output of sessionInfo() and terra::gdal(lib=TRUE). From the version of PROJ given by the latter, it is possible to guess the version of proj.db. If the versions differ much, the layout of this SQLite database will differ, and not be read correctly. You should not need any PROJ installation of your own.

rsbivand commented 1 year ago

With:

> sessionInfo()
R version 4.3.0 (2023-04-21 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Matrix products: default

locale:
[1] C

time zone: Europe/Oslo
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] RSQLite_2.3.1 terra_1.7-29  rgrass_0.3-8  XML_3.99-0.14

loaded via a namespace (and not attached):
 [1] codetools_0.2-19 fastmap_1.1.1    bit_4.0.5        cachem_1.0.8     blob_1.2.4      
 [6] pkgconfig_2.0.3  memoise_2.0.1    bit64_4.0.5      cli_3.6.1        vctrs_0.6.2     
[11] DBI_1.1.3        compiler_4.3.0   tools_4.3.0      Rcpp_1.0.10      rlang_1.1.1

I see:

> library(rgrass)
Loading required package: XML
GRASS GIS interface loaded with GRASS version: (GRASS not running)
> 

> initGRASS("C:/Program Files/GRASS GIS 8.2", tempdir(), override=TRUE)
gisdbase    C:/Users/RB/AppData/Local/Temp/RtmpisLW19 
location    file34683e7d479d 
mapset      file346840423f36 
rows        1 
columns     1 
north       1 
south       0 
west        0 
east        1 
nsres       1 
ewres       1 
projection:
 WARNING: <PROJ_INFO> file not found for location <file34683e7d479d>
WARNING: <PROJ_UNITS> file not found for location <file34683e7d479d>
ERROR: Projection files missing 
Warning message:
In system(syscmd, intern = intern, ignore.stderr = ignore.stderr,  :
  running command 'g.proj.exe -w' had status 1
> library(terra)
terra 1.7.29
> f <- system.file("ex/elev.tif", package="terra")
> r <- rast(f)
> initGRASS("C:/Program Files/GRASS GIS 8.2", tempdir(), SG=r, override=TRUE)
WARNING: Concurrent mapset locking is not supported on Windows
WARNING: Concurrent mapset locking is not supported on Windows
gisdbase    C:/Users/RB/AppData/Local/Temp/RtmpisLW19 
location    file34689c54bf6 
mapset      file346855b0584e 
rows        90 
columns     95 
north       50.19167 
south       49.44167 
west        5.741667 
east        6.533333 
nsres       0.008333333 
ewres       0.008333326 
projection  +proj=longlat +datum=WGS84 +no_defs +type=crs 
Warning message:
In system(syscmd, intern = intern, ignore.stderr = ignore.stderr,  :
  running command 'g.proj.exe -w' had status 884
> Sys.getenv("PROJ_LIB")
[1] ""
> cat(crs(r), "\n")
GEOGCRS["WGS 84",
    ENSEMBLE["World Geodetic System 1984 ensemble",
        MEMBER["World Geodetic System 1984 (Transit)"],
        MEMBER["World Geodetic System 1984 (G730)"],
        MEMBER["World Geodetic System 1984 (G873)"],
        MEMBER["World Geodetic System 1984 (G1150)"],
        MEMBER["World Geodetic System 1984 (G1674)"],
        MEMBER["World Geodetic System 1984 (G1762)"],
        MEMBER["World Geodetic System 1984 (G2139)"],
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]],
        ENSEMBLEACCURACY[2.0]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    USAGE[
        SCOPE["Horizontal component of 3D system."],
        AREA["World."],
        BBOX[-90,-180,90,180]],
    ID["EPSG",4326]] 
> library("RSQLite")
> db <- DBI::dbConnect(RSQLite::SQLite(), dbname=system.file("proj/proj.db", package="terra"))
> DBI::dbReadTable(db, "metadata")
                             key
1  DATABASE.LAYOUT.VERSION.MAJOR
2  DATABASE.LAYOUT.VERSION.MINOR
3                      EPSG.DATE
4                   EPSG.VERSION
5                      ESRI.DATE
6                   ESRI.VERSION
7                      IGNF.DATE
8                    IGNF.SOURCE
9                   IGNF.VERSION
10                      NKG.DATE
11                    NKG.SOURCE
12                   NKG.VERSION
13                  PROJ.VERSION
14             PROJ_DATA.VERSION
                                                                             value
1                                                                                1
2                                                                                2
3                                                                       2023-02-06
4                                                                          v10.082
5                                                                       2023-19-01
6                                                                   ArcGIS Pro 3.1
7                                                                       2019-05-24
8  https://raw.githubusercontent.com/rouault/proj-resources/master/IGNF.v3.1.0.xml
9                                                                            3.1.0
10                                                                      2020-12-21
11                          https://github.com/NordicGeodesy/NordicTransformations
12                                                                           1.0.0
13                                                                           9.2.0
14                                                                            1.13
> DBI::dbDisconnect(db)
> db <- DBI::dbConnect(RSQLite::SQLite(), dbname="C:/Program Files/GRASS GIS 8.2/share/proj/proj.db")
> DBI::dbReadTable(db, "metadata")
                             key
1  DATABASE.LAYOUT.VERSION.MAJOR
2  DATABASE.LAYOUT.VERSION.MINOR
3                      EPSG.DATE
4                   EPSG.VERSION
5                      ESRI.DATE
6                   ESRI.VERSION
7                      IGNF.DATE
8                    IGNF.SOURCE
9                   IGNF.VERSION
10                      NKG.DATE
11                    NKG.SOURCE
12                   NKG.VERSION
13                  PROJ.VERSION
14             PROJ_DATA.VERSION
                                                                             value
1                                                                                1
2                                                                                2
3                                                                       2022-08-31
4                                                                          v10.076
5                                                                       2022-07-09
6                                                                   ArcGIS Pro 3.0
7                                                                       2019-05-24
8  https://raw.githubusercontent.com/rouault/proj-resources/master/IGNF.v3.1.0.xml
9                                                                            3.1.0
10                                                                      2020-12-21
11                          https://github.com/NordicGeodesy/NordicTransformations
12                                                                           1.0.0
13                                                                           9.1.1
14                                                                            1.12
> DBI::dbDisconnect(db)

GRASS 8.2.1 is bundling a slightly older version of proj.db. I suggest moving this to the grass-stats list https://lists.osgeo.org/mailman/listinfo/grass-stats. This is partly about how GRASS distributions bundle PROJ too.

fgashakamba commented 1 year ago

Here are are the outputs of sessionInfo() and terra::gdal(lib=TRUE) respectively:

> library(rgrass)
Loading required package: XML
GRASS GIS interface loaded with GRASS version: (GRASS not running)
> sessionInfo()
R version 4.3.0 (2023-04-21 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 11 x64 (build 22621)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8   
[3] LC_MONETARY=English_United States.utf8 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.utf8    

time zone: Africa/Johannesburg
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rgrass_0.3-8  XML_3.99-0.14

loaded via a namespace (and not attached):
 [1] RPostgres_1.4.5  codetools_0.2-19 bit_4.0.5        remotes_2.4.2    blob_1.2.4      
 [6] pkgconfig_2.0.3  bit64_4.0.5      lifecycle_1.0.3  cli_3.6.1        terra_1.7-29    
[11] vctrs_0.6.2      DBI_1.1.3        compiler_4.3.0   rstudioapi_0.14  tools_4.3.0     
[16] hms_1.1.3        Rcpp_1.0.10      rlang_1.1.1     
> terra::gdal(lib=TRUE)
    gdal     proj     geos 
 "3.6.2"  "9.2.0" "3.11.2" 
> initGRASS(gisBase = "C:/Program Files/GRASS GIS 8.2", 
+           home = tempdir(), 
+           override = TRUE)
gisdbase    C:/Users/FAUSTI~1.GAS/AppData/Local/Temp/RtmpeILrg9 
location    file4b8c4b7c6014 
mapset      file4b8c4723dde 
rows        1 
columns     1 
north       1 
south       0 
west        0 
east        1 
nsres       1 
ewres       1 
projection:
 WARNING: <PROJ_INFO> file not found for location <file4b8c4b7c6014>
WARNING: <PROJ_UNITS> file not found for location <file4b8c4b7c6014>
ERROR: Projection files missing 
Warning message:
In system(syscmd, intern = intern, ignore.stderr = ignore.stderr,  :
  running command 'g.proj.exe -w' had status 1

If the issue is with the proj.db database bundled with GRASS, what could be the solution to that issue? are there earlier versions of either GRASS or rgrass that would play nicely with each other?

hellik commented 1 year ago

Normally, winGRASS, based upon OSGeo4W, uses up-to-date proj versions. Original report talks about GRASS GIS 8.0.2, that is an outdated winGRASS version.

hellik commented 1 year ago

How often is terra updating proj versions?

rsbivand commented 1 year ago

R uses MXE, fairly actively, and is currently working through the cmake shift for PROJ, GDAL and GEOS. The Rtools maintainer updates usually quite early: https://cran.r-project.org/bin/windows/Rtools/rtools43/rtools.html. Status is at https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/toolchain_libs/mxe/src/, https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/toolchain_libs/mxe/src/proj.mk for PROJ. The key thing for Rtools42 and Rtools43 is that they use UCRT, and are future-proof.

hellik commented 1 year ago

The proj update cycle in OSGeo4W is more or less real time, though standalone installer is not updated. AFAIK no standalone installers of other software are updated in such a way, they are shipped with library versions of that time of creation. So the question may be too If R and packages should be a bit more conservative in updating dependency libraries, as upstream software synchronizing with downstrean R(packages) would be nice, but hardly to achieve, at least in the windows side of the world.

In the upcoming GRASS code sprint in the next days, a target will be an attempt to cmake compiling, also on windows. Any help and input welcome.

rsbivand commented 1 year ago

Good luck! The major change for R was going for UCRT, which took a lot of work, but which should help in longer-term maintenance (that was an application-wide coming problem). Resolving that took us to MXE and its recipes, and to cross-compiling with recent GCC build trains.

Is information about version updates for external software for Windows and/or macOS binaries published anywhere? R packages typically have a function reporting these versions.

hellik commented 1 year ago

Is information about version updates for external software for Windows and/or macOS binaries published anywhere? R packages typically have a function reporting these versions.

regarding OSGeo4W (and the standalone installers are actually build on that environment):

https://download.osgeo.org/osgeo4w/v2/x86_64/setup.ini

e.g.

@ proj92-runtime
sdesc: "The PROJ library and commands for coordinate system transformations (Runtime)."
ldesc: "The PROJ library and commands for coordinate system transformations (Runtime)."
category: Libs
requires: msvcrt2019 sqlite3 libtiff curl proj-runtime-data proj-data
version: 9.2.1-1
install: x86_64/release/proj/proj92-runtime/proj92-runtime-9.2.1-1.tar.bz2 3600994 b51133e437da68965f04288310dd25bb
source: x86_64/release/proj/proj-9.2.1-1-src.tar.bz2 1425 507dc1068259ec542ef9b1513e1868e2
[prev]
version: 9.2.0-1
install: x86_64/release/proj/proj92-runtime/proj92-runtime-9.2.0-1.tar.bz2 3600332 60a8625f82efb4b9e2346294bfc290de
source: x86_64/release/proj/proj-9.2.0-1-src.tar.bz2 1431 2a29bb5e75f0a775d8b6142b15e17878

and within a GRASS session

g.version -e
GRASS 8.3.dev (2023)
PROJ: 9.1.1 <=
GDAL/OGR: 3.6.2
GEOS: 3.11.1
SQLite: 3.39.4
rsbivand commented 1 year ago

Could g.version -e also report the version of proj.db, or should that be inferred from the PROJ version?

hellik commented 1 year ago

Could g.version -e also report the version of proj.db, or should that be inferred from the PROJ version?

I couldn't find any information of proj-tools giving the info which proj.db is used. @rouault any hint?

so I guess: yes it should be inferred from the PROJ version. e.g. see [PROJ] PROJ 9.2.1RC1 & PROJ-data 1.14RC1

rouault commented 1 year ago

https://proj.org/en/9.2/development/reference/functions.html#c.proj_context_get_database_metadata and https://proj.org/en/9.2/development/reference/functions.html#c.proj_context_get_database_path should give you all what you need to know

rsbivand commented 1 year ago

@rouault thanks, that will help, as both sides can then negotiate about whose proj.db to use.

fgashakamba commented 1 year ago

@hellik When can we expect this code sprint you talked about to hopefully resolve this issue? I kept looking at the WinGRASS download website to see if there is a new version but no luck.

rsbivand commented 1 year ago

@fgashakamba please consider contributing, since this is important to you. I am occupied with other topics for the rest of 2023, so contributions are welcome. If there was an easy resolution, it would have been done; since this hasn't happened, the problem is intractable.