search

rsbivand / rgrass

Interpreted interface between the GRASS geographical information system and R
https://rsbivand.github.io/rgrass/
26 stars 8 forks source link

running GRASS within R, Windows 10 #87

Open dylanbeaudette opened 6 months ago

dylanbeaudette commented 6 months ago

Hi Roger. I'm attempting to run GRASS within R by manually specifying the GRASS installation like this. Note that I'm using GRASS via QGIS 3.32.3. Starting GRASS, then R works as expected.

Any suggestions?

library(terra)
library(rgrass)

(e <- rast('grids/elev_pcs.tif'))

.gb <- 'c:/Program Files/QGIS 3.32.3/apps/grass/grass83'
loc <- initGRASS(gisBase = .gb, home = 'e:/temp/GRASS', SG  = r, override = TRUE)

I get the following error message:

Error in if (gv == "sh: line 1: g.version: command not found") { : 
  argument is of length zero
In addition: Warning messages:
1: In system(paste("g.dirseps.exe -g", shQuote(Sys.getenv("GISRC"))),  :
  running command 'g.dirseps.exe -g "junk"' had status 309
2: In system(paste("g.dirseps.exe -g", shQuote(gisDbase)), intern = TRUE) :
  running command 'g.dirseps.exe -g "C:\Users\DYLAN~1.BEA\AppData\Local\Temp\RtmpGE8LUO"' had status 309
3: In dir.create(loc_path) :
  cannot create dir 'NA\file314cff728d9', reason 'No such file or directory'
4: In dir.create(paste(loc_path, "PERMANENT", sep = "/")) :
  cannot create dir 'NA\file314cff728d9\PERMANENT', reason 'No such file or directory'
5: In dir.create(paste(loc_path, mapset, sep = "/")) :
  cannot create dir 'NA\file314cff728d9\file314cf286007', reason 'No such file or directory'
6: In system(paste("g.version", get("addEXE", envir = .GRASS_CACHE),  :
  running command 'g.version.exe' had status 309

This seems to be related to unexpected (?) output from

For example, running rgrass:::.grassVersion() results in :

Error : '' does not exist in current working directory ('e:/working_copies/DSS_coweeta').
In addition: Warning message:
In system(cmd0, intern = TRUE) :
  running command 'g.version.exe --interface-description' had status 309
[1] "sh: line 1: g.version: command not found"

Output from sessionInfo():

R version 4.3.1 (2023-06-16 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8    LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C                           LC_TIME=English_United States.utf8    

time zone: America/Los_Angeles
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rgrass_0.4-2 terra_1.7-71

loaded via a namespace (and not attached):
[1] compiler_4.3.1    tools_4.3.1       rstudioapi_0.15.0 Rcpp_1.0.12       xml2_1.3.5        codetools_0.2-19
rsbivand commented 6 months ago

Hi @dylanbeaudette could you check directly in the Win-GRASS stand-alone? And how was QGIS installed?

dylanbeaudette commented 6 months ago

Hi @dylanbeaudette could you check directly in the Win-GRASS stand-alone?

I'll give that a try and report back.

rsbivand commented 6 months ago

If you installed using OSGeo4W, could you try to give the GRASS directory directly, not under QGIS?

rsbivand commented 6 months ago

With the QGIS standalone installer, I see a number of "The code execution cannot proceed because zstd.dll was not found." system errors, which I have to acknowledge to proceed, leading to failure. QGIS must extend the %PATH% variable when it starts so that GRASS modules can access their upstream dependencies.

library(terra)
library(rgrass)
.gb <- "C:/Program Files/QGIS 3.36.0/apps/grass/grass83"
(e <- rast(system.file('ex/elev.tif', package="terra")))
td <- tempdir()
loc <- initGRASS(gisBase = .gb, home = td, SG  = e, override = TRUE)
> loc <- initGRASS(gisBase = .gb, home = td, SG  = e, override = TRUE)
Error in if (gv == "sh: line 1: g.version: command not found") { : 
  argument is of length zero
In addition: Warning messages:
1: In system(paste("g.dirseps.exe -g", shQuote(Sys.getenv("GISRC"))),  :
  running command 'g.dirseps.exe -g "junk"' had status 309
2: In system(paste("g.dirseps.exe -g", shQuote(gisDbase)), intern = TRUE) :
  running command 'g.dirseps.exe -g "C:\Users\RB\AppData\Local\Temp\RtmpAxSfdg"' had status 309
3: In dir.create(loc_path) :
  cannot create dir 'NA\file13241d1b5f2a', reason 'No such file or directory'
4: In dir.create(paste(loc_path, "PERMANENT", sep = "/")) :
  cannot create dir 'NA\file13241d1b5f2a\PERMANENT', reason 'No such file or directory'
5: In dir.create(paste(loc_path, mapset, sep = "/")) :
  cannot create dir 'NA\file13241d1b5f2a\file132433967abb', reason 'No such file or directory'
6: In system(paste("g.version", get("addEXE", envir = .GRASS_CACHE),  :
  running command 'g.version.exe' had status 309

Unless we know exactly how Windows standalone QGIS modifies the path, or that QGIS makes it possible to start R from within (I looked at the QGIS python console without success), the QGIS windows standalone installer of GRASS cannot be used with rgrass - this is the reason that Rgui must be started in the OSGeo4W console:

C:\OSGeo4W>echo %PATH%
C:\OSGeo4W\apps\qt5\bin;C:\OSGeo4W\apps\Python39\Scripts;C:\OSGeo4W\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32\WBem
C:\OSGeo4W>"C:\Program Files\R\R-4.3.3\bin\x64\Rgui.exe"

and after running initGRASS:

Sys.getenv("PATH")
[1] "C:/OSGeo4W/apps/grass/grass82\\extrabin;C:/OSGeo4W/apps/grass/grass82\\bin;C:/OSGeo4W/apps/grass/grass82\\lib;C:\\rtools43/x86_64-w64-mingw32.static.posix/bin;C:\\rtools43/usr/bin;C:\\OSGeo4W\\apps\\qt5\\bin;C:\\OSGeo4W\\apps\\Python39\\Scripts;C:\\OSGeo4W\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\system32\\WBem"
>

rather than directly (which initGRASS traps).

rsbivand commented 6 months ago

From the QGIS standalone python console:

os.getenv('PATH')
'C:\\PROGRA~1\\QGIS33~1.0\\apps\\qgis\\bin;C:\\PROGRA~1\\QGIS33~1.0\\apps\\grass\\grass83\\lib;C:\\PROGRA~1\\QGIS33~1.0\\apps\\grass\\grass83\\bin;C:\\PROGRA~1\\QGIS33~1.0\\apps\\qt5\\bin;C:\\PROGRA~1\\QGIS33~1.0\\apps\\Python39\\Scripts;C:\\PROGRA~1\\QGIS33~1.0\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\system32\\WBem'

From the OSGeo4W console bundled with the QGIS standalone:

C:\Program Files\QGIS 3.36.0>echo %PATH%
C:\PROGRA~1\QGIS33~1.0\apps\qt5\bin;C:\PROGRA~1\QGIS33~1.0\apps\Python39\Scripts;C:\PROGRA~1\QGIS33~1.0\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32\WBem

Starting Rgui from that console:

> Sys.getenv("PATH")
[1] "C:\\rtools43/x86_64-w64-mingw32.static.posix/bin;C:\\rtools43/usr/bin;C:\\PROGRA~1\\QGIS33~1.0\\apps\\qt5\\bin;C:\\PROGRA~1\\QGIS33~1.0\\apps\\Python39\\Scripts;C:\\PROGRA~1\\QGIS33~1.0\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\system32\\WBem"
terra 1.7.71
> library(rgrass)
GRASS GIS interface loaded with GRASS version: (GRASS not running)
> .gb <- "C:/Program Files/QGIS 3.36.0/apps/grass/grass83"
> (e <- rast(system.file('ex/elev.tif', package="terra")))
class       : SpatRaster 
dimensions  : 90, 95, 1  (nrow, ncol, nlyr)
resolution  : 0.008333333, 0.008333333  (x, y)
extent      : 5.741667, 6.533333, 49.44167, 50.19167  (xmin, xmax, ymin, ymax)
coord. ref. : lon/lat WGS 84 (EPSG:4326) 
source      : elev.tif 
name        : elevation 
min value   :       141 
max value   :       547 
> td <- tempdir()
> loc <- initGRASS(gisBase = .gb, home = td, SG  = e, override = TRUE)
WARNING: Concurrent mapset locking is not supported on Windows
WARNING: Concurrent mapset locking is not supported on Windows
Warning messages:
1: In initGRASS(gisBase = .gb, home = td, SG = e, override = TRUE) :
  working directory not writable, using tempfile for GISRC
2: In system(syscmd, intern = intern, ignore.stderr = ignore.stderr,  :
  running command 'g.proj.exe -w' had status 884
>

where the first warning is because the QGIS standalone OSGeo4W console is at:

> getwd()
[1] "C:/Program Files/QGIS 3.36.0"

and the second may be about where g.proj looks for proj.db.

Please look and see whether you have a menu item like the following - starting R in that console should work: QGIS

rsbivand commented 6 months ago

I think that 0.4-3 detects the problem and error-exits, suggesting the use of the OSGeo4W shell. Could you @dylanbeaudette please remotes::install_github("rsbivand/rgrass") and try, both starting R (probably need absolute path) in the OSGeo4W shell and outside it? I can build you a Windows binary package if need be. Here is the source tarball: rgrass_0.4-3.tar.gz

dylanbeaudette commented 6 months ago

Thanks Roger. I've installed OSGeo GRASS and rgrass 0.4-3 as suggested.

Attempting to start GRASS from within R results in an error:

> loc <- initGRASS(gisBase = .gb, home = tempdir(), SG  = r, override = TRUE)
Error in initGRASS(gisBase = .gb, home = tempdir(), SG = r, override = TRUE) : 
  NOTE: If using OSGeo4W GRASS, start R in the OSGeo4W shell,
see help(initGRASS) for further details

Attempting to run RStudio from within GRASS, as launched from the OSGeo shell results in an RStudio crash (white screen that eventually quits).

Running 'R' from GRASS, as launched from the OSGeo shell works as expected.

rsbivand commented 6 months ago

@dylanbeaudette Yes, as I said, all users of OSGeo4W GRASS must always start R (and if used RStudio) from the OSGeo4W shell, and your case shows that this extends to all users of QGIS Windows standalone installations. You have not been starting R in your case from the OSGeo4W shell shipped with the QGIS Windows standalone installer - I did stress that this is the only feasible route, because we can only reasonably handle the GRASS case - establishing the minimal set of environment variable values that GRASS needs to run in initGRASS.

OSGeo4W, QGIS Windows standalone, and RStudio all silently and in hiding modify the environment variables and if set their values when they start, and the only way to capture these values and ensure that they are passed through is to only start R/RStudio when the external environment variables and their values have been set. Please see https://rsbivand.github.io/rgrass/articles/use.html. GRASS has explicit and stable policies for its environment variables - this does not apply to the others.

dylanbeaudette commented 6 months ago

Thanks for the explanation Roger. It looks like I have no way to run GRASS from within R on my current setup.

Details:

rsbivand commented 6 months ago

@dylanbeaudette Please explain exactly what your current setup is, otherwise other users reading this thread will also be confused.

I have explained I think in siple terms that, if your Windows GRASS installation was made by the OSGeo4W inataller or by the Windows QGIS standalone installer, you have to start R or RStudio in the OSGeo4W shell provided by that installer, because the OSGeo4W shell sets up important environment variables modifying where programs and apps can be found.

You have not stated that you tried starting R or RStudio in the appropriate OSGeo4W shell (the OSGeo4W shell iccon can be found in the Windows app navigation bar to the left for QGIS and OSGeo4W). Did you try that, or not?

rsbivand commented 6 months ago

In the QGIS standalone installer case, checked on Windows 10, the OSGeo4W shell is stored in C:\ProgramData\Microsoft\Windows\Start Menu\Programs\QGIS 3.36.0.

In the OSGeo4W installer case, in C:\Users\RB\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\OSGeo4W, so stored in the user's HOME appdata.

Both are available directly from the left app navbar. In the QGIS case, the OSGeo4W shell opens in C:\Program Files\QGIS 3.36.0, so the user should change directory at some point before needing to write to the current directory.

Because the PATH variable now does not include the location of RStudio or Rgui, they must be started by absolute address, for me for my installed versions "C:\Program Files\R\R-4.3.3\bin\x64\Rgui.exe" or "C:\Program Files\RStudio\rstudio.exe", in quotes because of the space in the path.

Starting RStudio from the OSGeo4W shell, note that RStudio itself changes the working directory to HOME/Documents, so helpfully overriding the read-only directory we began from. The code:

library(terra)
library(rgrass)
.gb <- "C:/Program Files/QGIS 3.36.0/apps/grass/grass83"
(e <- rast(system.file('ex/elev.tif', package="terra")))
td <- tempdir()
loc <- initGRASS(gisBase = .gb, home = td, SG  = e, override = TRUE)

runs adequately; as can be seen, the temporary location is created successfully:

image

rsbivand commented 6 months ago

The OSGeo4W installer shell starts in C:\OSGeo4W. Starting RStudio, again note that the working directory is writable. Using the code now specifying the OSGeo4W GRASS:

library(terra)
library(rgrass)
.gb <- "C:/OSGeo4W/apps/grass/grass82"
(e <- rast(system.file('ex/elev.tif', package="terra")))
td <- tempdir()
loc <- initGRASS(gisBase = .gb, home = td, SG  = e, override = TRUE)

and the temporary location is again created: image

The reported warnings on loading terra and during subsequent work are because the OSGeo4W express installer does not install GDAL non-free plugins properly - terra appears to work fine. Correcting by updating GDAL in the OSGeo4W custom installer and refreshing the non-free plugins removes the problem.

rsbivand commented 6 months ago

Starting Rgui in either OSGeo4W shell does not change the current directory, so the user must themselves change, for example using setwd(Sys.getenv("HOME")) with the OSGeo4W installer and its OSGeo4W shell:

image

and with the QGIS standalone installer and its OSGeo4W shell:

image

The extra warning messages about unfound GDAL plugins are still present for the OSGeo4W express installer, but as found above are no longer present when the custom installer found under OSGeo4W Setup is used to update GDAL and refresh the non-free plugins.

rsbivand commented 6 months ago

@veroandreo could you please check the most recent commit https://github.com/rsbivand/rgrass/commit/28b7cb0cb292b333e0cfc91ab92fe7b95cb745a6 and the rendering of https://rsbivand.github.io/rgrass/articles/use.html ?

dylanbeaudette commented 6 months ago

@dylanbeaudette Please explain exactly what your current setup is, otherwise other users reading this thread will also be confused.

I have explained I think in siple terms that, if your Windows GRASS installation was made by the OSGeo4W inataller or by the Windows QGIS standalone installer, you have to start R or RStudio in the OSGeo4W shell provided by that installer, because the OSGeo4W shell sets up important environment variables modifying where programs and apps can be found.

You have not stated that you tried starting R or RStudio in the appropriate OSGeo4W shell (the OSGeo4W shell iccon can be found in the Windows app navigation bar to the left for QGIS and OSGeo4W). Did you try that, or not?

Thanks for the reminder, I've added relevant details above.

Starting R from within within the OSGeo4W shell works as expected, RStudio hangs with a white screen.

Attempting OSGeo4W shell → R → initGRASS() results in the following:

image

rsbivand commented 6 months ago

@dylanbeaudette you can see that I had no such remaining difficulty with:

R version 4.3.3 (2024-02-29 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

RStudio 2023.12.1 Build 402 rgrass 0.4-3 QGIS 3.36.0 standalone bundling GRASS 8.3.1 (2023)

The only remaining uncertainty for me is "running command 'g.proj.exe -w' had status 884" which I think is when terra sets PROJ_LIB or PROJ_DATA (or unsets them), but here the proj.db used by g.proj is given by:

> Sys.getenv("PROJ_LIB") 
[1] "C:\\PROGRA~1\\QGIS33~1.0\\share\\proj" 
> Sys.getenv("PROJ_DATA") 
[1] ""

so should not complain. And g.proj does output other formats, so I'm unsure what this means. I've added screendumps and text to the use vignette.

rsbivand commented 6 months ago

Attempting OSGeo4W shell → R → initGRASS() results in the following:

This seems to happen at https://github.com/rsbivand/rgrass/blob/28b7cb0cb292b333e0cfc91ab92fe7b95cb745a6/R/initGRASS.R#L392-L393

Could you put debug(initGRASS) before the call, and step through until that point? Then get the value of Sys.getenv("PATH") before the failure? Then I can compare with my PATH at that point?

rsbivand commented 6 months ago

Installing QGIS 3.32.3 from https://ftp.osuosl.org/pub/osgeo/download/qgis/windows/QGIS-OSGeo4W-3.32.3-1.msi, I get:

> (e <- rast(system.file('ex/elev.tif', package="terra")))
class       : SpatRaster 
dimensions  : 90, 95, 1  (nrow, ncol, nlyr)
resolution  : 0.008333333, 0.008333333  (x, y)
extent      : 5.741667, 6.533333, 49.44167, 50.19167  (xmin, xmax, ymin, ymax)
coord. ref. : lon/lat WGS 84 (EPSG:4326) 
source      : elev.tif 
name        : elevation 
min value   :       141 
max value   :       547 
> td <- tempdir()
> .gb <- "C:/Program Files/QGIS 3.32.3/apps/grass/grass83"
> loc <- initGRASS(gisBase = .gb, home = td, SG  = e, override = TRUE)
WARNING: Concurrent mapset locking is not supported on Windows
WARNING: Concurrent mapset locking is not supported on Windows
Warning messages:
1: In initGRASS(gisBase = .gb, home = td, SG = e, override = TRUE) :
  working directory not writable, using tempfile for GISRC
2: In system(syscmd, intern = intern, ignore.stderr = ignore.stderr,  :
  running command 'g.proj.exe -w' had status 884
>

So not a problem with the 3.32.3 version of the QGIS standalone installer. At the point at which g.region is called, my PATH is:

Sys.getenv("PATH")
[1] "C:/Program Files/QGIS 3.32.3/apps/grass/grass83\\extrabin;C:/Program Files/QGIS 3.32.3/apps/grass/grass83\\bin;C:/Program Files/QGIS 3.32.3/apps/grass/grass83\\lib;C:\\rtools43/x86_64-w64-mingw32.static.posix/bin;C:\\rtools43/usr/bin;C:\\PROGRA~1\\QGIS33~1.3\\apps\\qt5\\bin;C:\\PROGRA~1\\QGIS33~1.3\\apps\\Python39\\Scripts;C:\\PROGRA~1\\QGIS33~1.3\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\system32\\WBem"
Browse[2]>

Before starting Rgui, the PATH is:

C:\Program Files\QGIS 3.32.3>echo %PATH%
C:\PROGRA~1\QGIS33~1.3\apps\qt5\bin;C:\PROGRA~1\QGIS33~1.3\apps\Python39\Scripts;C:\PROGRA~1\QGIS33~1.3\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32\WBem

and after starting Rgui:

> Sys.getenv("PATH")
[1] "C:\\rtools43/x86_64-w64-mingw32.static.posix/bin;C:\\rtools43/usr/bin;C:\\PROGRA~1\\QGIS33~1.3\\apps\\qt5\\bin;C:\\PROGRA~1\\QGIS33~1.3\\apps\\Python39\\Scripts;C:\\PROGRA~1\\QGIS33~1.3\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\system32\\WBem"
>

There are no confusing components in C:\\rtools43/x86_64-w64-mingw32.static.posix/bin;C:\\rtools43/usr/bin.

libgdal307.dll is in C:\Program Files\QGIS 3.32.3\bin for the 3.32.3 installer; there should be no 308, because:

C:\Program Files\QGIS 3.32.3>gdalinfo --version
GDAL 3.7.2, released 2023/09/05

Something in your system wanted 308 but this should not be possible - @dylanbeaudette can you re-install the QGIS version (after removing the installed QGIS app completely from the Windows control panel)? Is the libgdal file visible in the QGIS bin? Is GRASS the version provided with that QGIS bundle - it must be, because it has to be built against the same library files? It looks as though your GRASS has been re-installed linking to GDAL 3.8, and then inserted into the QGIS directory tree.

veroandreo commented 6 months ago

@veroandreo could you please check the most recent commit 28b7cb0 and the rendering of https://rsbivand.github.io/rgrass/articles/use.html ?

@rsbivand found a couple of typos only :)

rsbivand commented 6 months ago

@veroandreo Thanks! Do you yourself use the QGIS standalone installer to install GRASS, or do you know of others who do so? I think the problem is in the QGIS installer rather than in rgrass, and that the resolution is to run from the OSGeo4W shell (as for OSGeo4W), but more reports would be helpful.

veroandreo commented 6 months ago

@veroandreo Thanks! Do you yourself use the QGIS standalone installer to install GRASS, or do you know of others who do so? I think the problem is in the QGIS installer rather than in rgrass, and that the resolution is to run from the OSGeo4W shell (as for OSGeo4W), but more reports would be helpful.

Hi @rsbivand, no, not really. That's certainly not my case, but I somehow see how people would want to use the GRASS that comes via QGIS installer (i.e., why bothering installing it again, right?). I am now developing several tutorials for GRASS, and I'll be testing some of these things in Windows too. Will report back, wish me luck :see_no_evil:

dylanbeaudette commented 6 months ago

Thanks @rsbivand and @veroandreo for the further clarification and testing.

I figured out that most of my difficulties were related to selecting the "correct" OSGeo4W shell... I had been accidentally using the QGIS version.

In summary, I was able to get "rgrass outside of a GRASS session" working:

I'm still not sure why I can't start RStudio from the OSGeo4W shell... it just crashes without any indication as to why.

dylanbeaudette commented 6 months ago

One final test, and success: I'm able to successfully start RStudio from the OSGeo4W shell using a complete path to the executable file,

e.g. "c:\Program Files\RStudio\rstudio.exe"

As far as the original issue is concerned, I think that all of my (rgrass) problems have been solved. Thanks!