wbt_mosaic returns error 101

[FNRobinne]

Hi, I'm trying to mosaic tiles from MERIT-Hydro for Western Canada. I have 121 elevation tif files (tiles) in a folder that I have set as my working directory.

This is what I'm running: wbt_mosaic(output = "MERIT_mosaic_wbt.tif", inputs = NULL, method = "nn") Based on the documentation, I don't specify my inputs so it automatically reads all the .tif in my working directory.

After less than a second, I get the following message:

Error running WhiteboxTools (mosaic)
  whitebox.exe_path: "C:/Users/robinne/Documents/R/win-library/4.1/whitebox/WBT/whitebox_tools.exe"; File exists? TRUE
  Arguments: --run=mosaic  --output=MERIT_mosaic_wbt.tif --method=nn -v

System command had status 101
mosaic - Elapsed Time: NA [did not run]`

I tried removing the inputs parameter and use a list of .tif files as inputs, but it does not work. Is there an issue with the total size of files to be handled (>7Giga)?

[jblindsay]

Trying to mosaic 121 6000x6000 DEM tiles is probably ill-advised, and if you have limited memory, will very likely result in a memory error. Nonetheless, the error that you have is not a memory error, but rather one of a missing file. It seems as though your install is unable to find the WBT executable file. Have you checked that location on your computer, "C:/Users/robinne/Documents/R/win-library/4.1/whitebox/WBT/whitebox_tools.exe", to confirm that the file actually exists? I've not used the R-frontend for WBT personally, but it is possible that it doesn't automatically download and install the WBT backend and that you need to install it yourself. If this is the case, you can get the necessary executable from here.

[FNRobinne]

Thank you John for your answer. Yes, the .exe file is there. I actually installed WBT from within R using the command: whitebox::install_whitebox() I can run other WBT tools fine (e.g., the watershed tools on one tile).

[bkielstr]

Hi @merlinlelutin!

Like you said, this indicates you have the WBT executable file: whitebox.exe_path: "C:/Users/robinne/Documents/R/win-library/4.1/whitebox/WBT/whitebox_tools.exe"; File exists? TRUE

I would start with something like two or three adjacent tiles and get it working first and then proceed with larger rasters.

Did you specify wd =? The whiteboxR working directory is different than the R working directory. See ?wbt_options. You might need to hard code that. Also consider setting verbose_mode = TRUE if more information becomes available.

Good luck!

[FNRobinne]

Hey @bkielstr ! Good to see you here!

I tried what you mentioned already, i.e., just mosaicing two adjacent tiles, but I get the same error.

But I followed your advice and specify the wd using whitebox::wbt_wd("") so it gets the directory at the beginning of my script, where my tiles are located. And...it worked...at least it started and after a couple minutes I ended up with an error System command had status 1033.

[jblindsay]

Can you please try mosaicking a couple of tiles using the WhiteboxTools Runner, so that we can determine whether this is an issue with the WBT R frontend or if it is an issue that belongs to the WBT backend?

[FNRobinne]

I was able to make a mosaic with the WhiteboxTools Runner. After a bit of fiddling, I was also able to create a mosaic with two tiles in R; it was not clear how the input list must be entered for the tool (i.e., inputs = "tile_1;tile_2;tile_n"). So, I guess the tool works fine when wd is well specified, but I might be running into memory issues with status 1033.

[UWyoHydro]

First off: What a fantastic package, thank you for the development!! Hope it's alright to piggyback on this, since I believe it's a similar issue. I am running into some problems with wbt_mosaic() as well. Both the way that the output path and the input folder path need to be specified were/are not clear to me. Using this thread, I was able to make the tool work, but the mosaic'd tif gets saved in the input folder, together with the files used for the mosaic'ing. I have an Rproj with folders "Inputs" and "Outputs". In order to make wbt_mosaic() work, I have to set the wd before I run the tool with

whitebox::wbt_wd(here("Inputs"))

otherwise, I receive an error 101 message. If I now specify an output that contains more than just the file name (i.e.,

output = here("Outputs", "dem_mosaic.tif")

instead of only

output = "dem_mosaic.tif" ,

the tool finishes running, but no output gets saved. This is regardless of whether I use here() or type out the file path. So, is there a way to specify a separate output folder?

Also (but this is indeed a different issue), the wbt_mosaic() appears to remove the CRS from the resulting .tif.

[giswqs]

@UWyoHydro Have you tried using absolute file paths for both the inputs and output?

For the lost CRS issue, I thought that issue has already been fixed at https://github.com/jblindsay/whitebox-tools/issues/6#issuecomment-450501932. If not, you might want to open an issue at https://github.com/jblindsay/whitebox-tools/issues

[brownag]

@UWyoHydro @giswqs I took a peek at this. Having to use the inputs=NULL with --wd specified to avoid errors appears to be related to needing shell quoting around the inputs argument when it contains semicolons. Semicolons are the suggested syntax from wbt_tool_help() and above in this thread, but that doesn't work without quoting (on some platforms?)

When I try to specify a list of relative paths unquoted with semicolons I get something like the following error (System command had status 126) suggesting WhiteboxTools is getting less than two input files.

wbt_mosaic(output = "Output/foo.tif", inputs = "Input/test1.tif;Input/test2.tif;Input/test3.tif")

thread 'main' panicked at 'There is something incorrect about the input files. At least two inputs are required to operate this tool.', whitebox-tools-app/src/main.rs:72:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
sh: 1: Input/test2.tif: Permission denied
sh: 1: Input/test3.tif: Permission denied

Error running WhiteboxTools (mosaic)
  whitebox.exe_path: '/home/andrew/workspace/whitebox-tools/target/release/whitebox_tools'; File exists? TRUE
  Arguments: --run=mosaic  --output=Output/foo1.tif --inputs=Input/test1.tif;Input/test2.tif;Input/test3.tif --method=nn -v

System command had status 126
mosaic - Elapsed Time: NA [did not run]

Note the two sh 'Permission denied' errors. This indicates the shell is trying to execute my .tif files (which are not executable) as if I separated multiple commands with a semicolon.

To get around this you can either use comma as a input file separator with or without quoting or use semi-colons and shell quote the inputs argument. This is just adding single quotes to the beginning/end or wrapping in shQuote(). It seems that relative and absolute paths both work when "properly" quoted for their separator type.

Coming up with a more general scheme for quoting/preprocessing R inputs before generating commands is a known issue / work in progress e.g. #20 #46

---

Here is an example:

library(whitebox)
library(terra)
#> terra 1.5.13
library(elevatr)

wbt_verbose(TRUE)

if (!dir.exists("~/wbtwdtest")) 
  dir.create("~/wbtwdtest")
setwd("~/wbtwdtest")
wbt_wd("")
#> Reset WhiteboxTools working directory to current R working directory: /home/andrew/wbtwdtest 
#> After next tool run package option will be unset so that --wd flag is dropped.Unset WhiteboxTools working directory flag `whitebox.wd` / `--wd`
dir.create("Input")
dir.create("Output")

# sample tile locations
xy <- data.frame(x = -seq(119, 120, 0.5), y =   37)

# download tiles
dat <- lapply(1:nrow(xy), function(i) {
  # need two points/tile to use get_elev_raster()
  # add +0.0001 to x coord to duplicate
  setNames(terra::rast(elevatr::get_elev_raster(rbind(xy[i,], transform(xy[i,], x = x + 0.0001)), 
                                                prj = "EPSG:4326", z = 10)), 
           nm = "Elevation")
})
#> Mosaicing & Projecting
#> Note: Elevation units are in meters.
#> Mosaicing & Projecting
#> Note: Elevation units are in meters.
#> Mosaicing & Projecting
#> Note: Elevation units are in meters.

# write sample tiles to file in Input folder
tiles <- lapply(seq_along(dat), function(i) { 
  writeRaster(dat[[i]], sprintf("Input/test%s.tif", i), overwrite = TRUE)
})

# does not work (all relative paths with semicolon separator)
wbt_mosaic(output = "Output/foo.tif", inputs = "Input/test1.tif;Input/test2.tif;Input/test3.tif")
#> 
#> Error running WhiteboxTools (mosaic)
#>   whitebox.exe_path: '/home/andrew/workspace/whitebox-tools/target/release/whitebox_tools'; File exists? TRUE
#>   Arguments: --run=mosaic  --output=Output/foo.tif --inputs=Input/test1.tif;Input/test2.tif;Input/test3.tif --method=nn -v
#> System command had status 126
#> mosaic - Elapsed Time: NA [did not run]

# WORKS: quoted relative path with semicolon separator
wbt_mosaic(output = "Output/foo1.tif", inputs = "'Input/test1.tif;Input/test2.tif;Input/test3.tif'")
#> mosaic - Elapsed Time (including I/O): 0.313s
rast("Output/foo1.tif")
#> Warning: [rast] can't  get proj4 from srs
#> class       : SpatRaster 
#> dimensions  : 905, 2829, 1  (nrow, ncol, nlyr)
#> resolution  : 0.0006213062, 0.0006213062  (x, y)
#> extent      : -120.2344, -118.4767, 36.59803, 37.16032  (xmin, xmax, ymin, ymax)
#> coord. ref. :  
#> source      : foo1.tif 
#> name        : foo1

# WORKS: shell quoted absolute path with ; separator
wbt_mosaic(output = "Output/foo2.tif", inputs = "'/home/andrew/wbtwdtest/Input/test1.tif;/home/andrew/wbtwdtest/Input/test2.tif;/home/andrew/wbtwdtest/Input/test3.tif'")
#> mosaic - Elapsed Time (including I/O): 0.319s
rast("Output/foo2.tif")
#> Warning: [rast] can't  get proj4 from srs
#> class       : SpatRaster 
#> dimensions  : 905, 2829, 1  (nrow, ncol, nlyr)
#> resolution  : 0.0006213062, 0.0006213062  (x, y)
#> extent      : -120.2344, -118.4767, 36.59803, 37.16032  (xmin, xmax, ymin, ymax)
#> coord. ref. :  
#> source      : foo2.tif 
#> name        : foo2

# WORKS: unquoted absolute path with comma separator
wbt_mosaic(output = "Output/foo3.tif", inputs = "/home/andrew/wbtwdtest/Input/test1.tif,/home/andrew/wbtwdtest/Input/test2.tif,/home/andrew/wbtwdtest/Input/test3.tif")
#> mosaic - Elapsed Time (including I/O): 0.327s
rast("Output/foo3.tif")
#> Warning: [rast] can't  get proj4 from srs
#> class       : SpatRaster 
#> dimensions  : 905, 2829, 1  (nrow, ncol, nlyr)
#> resolution  : 0.0006213062, 0.0006213062  (x, y)
#> extent      : -120.2344, -118.4767, 36.59803, 37.16032  (xmin, xmax, ymin, ymax)
#> coord. ref. :  
#> source      : foo3.tif 
#> name        : foo3

# WORKS: unquoted relative path with comma separator
wbt_mosaic(output = "Output/foo4.tif", inputs = "Input/test1.tif,Input/test2.tif,Input/test3.tif")
#> mosaic - Elapsed Time (including I/O): 0.316s
rast("Output/foo4.tif")
#> Warning: [rast] can't  get proj4 from srs
#> class       : SpatRaster 
#> dimensions  : 905, 2829, 1  (nrow, ncol, nlyr)
#> resolution  : 0.0006213062, 0.0006213062  (x, y)
#> extent      : -120.2344, -118.4767, 36.59803, 37.16032  (xmin, xmax, ymin, ymax)
#> coord. ref. :  
#> source      : foo4.tif 
#> name        : foo4

[giswqs]

@brownag This is very helpful. Thank you for the thorough tests.

[brownag]

the tool finishes running, but no output gets saved.

@UWyoHydro Could there be a settings.json file in the R working directory that is pointing working_directory somewhere else and creating the output there?

With the current release of the whitebox R package it is possible for the R package to get out of sync with the settings.json files that are created when you set a working directory. Running wbt_wd("") should reset this, or you could delete the settings.json file and start fresh.

[UWyoHydro]

@giswqs @brownag Thank you for your answers! (apologies for the delay in responding on my end...)

So, let's see... I had tried using the full file paths, to no avail. As far as specifying inputs is concerned, I had tried both comma and semi colon, but might not have quoted it correctly. I will give it a shot later, properly quoted. My workaround had been to use the script to copy and then delete the output, which...worked, but is very clumsy from a coding perspective :-)

As far as the non-output is concerned, I named the resulting tif unambiguously and then did a search. Granted, the Windows search is garbage but it did not come up with anything. Very strange.

About the disappearing CRS, it certainly was not retained in the one case where I used wbt_mosaic(). Need to try it on something else before I open a new issue, though!

Thanks again, very helpful responses!

[brownag]

About the disappearing CRS, it certainly was not retained in the one case where I used wbt_mosaic(). Need to try it on something else before I open a new issue, though

I can confirm that in my above example the CRS is dropped, I should have mentioned that. The input tiles have WGS84 geographic coordinate system (terra shows coord. ref. : lon/lat WGS 84 (EPSG:4326) whereas terra shows coord. ref. : NA and output has the WKT string for a generic cartesian system as seen in gdalinfo for the output

> sf::gdal_utils(source = "C:/Users/Andrew.G.Brown/Documents/wbtwdtest/Output/foo.tif") |> paste0("\n") |> cat()
Driver: GTiff/GeoTIFF
Files: C:/Users/Andrew.G.Brown/Documents/wbtwdtest/Output/foo.tif
Size is 2829, 905
Coordinate System is:
ENGCRS["unnamed",
    EDATUM[""],
    CS[Cartesian,2],
        AXIS["(E)",east,
            ORDER[1],
            LENGTHUNIT["unknown",1]],
        AXIS["(N)",north,
            ORDER[2],
            LENGTHUNIT["unknown",1]]]
Data axis to CRS axis mapping: 1,2
Origin = (-120.234375000000000,37.160316546736766)
Pixel Size = (0.000621306228240,-0.000621306228240)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
  TIFFTAG_SOFTWARE=WhiteboxTools
  TIFFTAG_XRESOLUTION=72
  TIFFTAG_YRESOLUTION=72
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (-120.2343750,  37.1603165) 
Lower Left  (-120.2343750,  36.5980344) 
Upper Right (-118.4766997,  37.1603165) 
Lower Right (-118.4766997,  36.5980344) 
Center      (-119.3555373,  36.8791755) 
Band 1 Block=2829x1 Type=Float32, ColorInterp=Gray
  NoData Value=-32768
Driver: GTiff/GeoTIFF
Files: C:/Users/Andrew.G.Brown/Documents/wbtwdtest/Output/foo.tif
Size is 2829, 905
Coordinate System is:
ENGCRS["unnamed",
    EDATUM[""],
    CS[Cartesian,2],
        AXIS["(E)",east,
            ORDER[1],
            LENGTHUNIT["unknown",1]],
        AXIS["(N)",north,
            ORDER[2],
            LENGTHUNIT["unknown",1]]]
Data axis to CRS axis mapping: 1,2
Origin = (-120.234375000000000,37.160316546736766)
Pixel Size = (0.000621306228240,-0.000621306228240)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
  TIFFTAG_SOFTWARE=WhiteboxTools
  TIFFTAG_XRESOLUTION=72
  TIFFTAG_YRESOLUTION=72
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (-120.2343750,  37.1603165) 
Lower Left  (-120.2343750,  36.5980344) 
Upper Right (-118.4766997,  37.1603165) 
Lower Right (-118.4766997,  36.5980344) 
Center      (-119.3555373,  36.8791755) 
Band 1 Block=2829x1 Type=Float32, ColorInterp=Gray
  NoData Value=-32768

[brownag]

This issue with quoting and semicolon separated multi-file inputs should be resolved in whitebox 2.1.4 so I will close this issue.

As far as the missing CRSs, I think I have seen it be a problem with several tools. In terms of the R package all we can do in theory is ensure CRS is propagated from input to output--but not in the current tool implementation and without some significant effort. Hopefully that issue will be fixed in WhiteboxTools itself soon.