Tools unable to use xarray opened with xarray.open_rasterio even though the file can be used directly

[dharhas]

Opening a raster with xarray.open_rasterio and passing it into whitebox fails because of lack of correct metadata in the .dep/.tas file being used as intermediate format.

The same file works fine when passed into whitebox tool directly as a file, so all the required metadata must be available.

i.e.

elev = xr.open_rasterio(elev_file).isel(band=0)
filled_dem = BreachDepressions(dem=elev)

fails

filled_dem = BreachDepressions(dem=elev_file)

succeeds

[PeterDSteinberg]

@dharhas If you have a traceback available for the failure, that would help here. The Xarray wrapper for WhiteBox should be adding the right keys/values in a Dataset's attrs regardless of how it was opened and read, so I'm not sure why it didn't do that. I can look into this further this week and hopefully fix.

[dharhas]

I can do that later this week. Along with an example file.

[snowman2]

elevation_raster = xr.open_rasterio(elev_file)

Output:

<xarray.DataArray (band: 1, y: 192, x: 237)>
array([[[ 19.746029,  19.725756, ...,  20.450933,  20.465611],
        [ 19.678217,  19.824434, ...,  20.485765,  20.43626 ],
        ..., 
        [ 19.692333,  19.667782, ...,  21.244215,  21.378166],
        [ 19.656511,  19.683601, ...,  21.35741 ,  21.371601]]], dtype=float32)
Coordinates:
  * band     (band) int64 1
  * y        (y) float64 32.23 32.23 32.23 32.23 32.23 32.23 32.23 32.23 ...
  * x        (x) float64 -91.49 -91.49 -91.49 -91.49 -91.49 -91.49 -91.49 ...
Attributes:
    crs:        +init=epsg:4269
    is_tiled:   0
    res:        (0.00027813488916465513, 0.0002784031233404501)
    transform:  (-91.48907218202051, 0.00027813488916465513, 0.0, 32.22977767...

filled_dem = BreachDepressions(dem=elevation_raster)

Output:

---------------------------------------------------------------------------
MissingDepMetadata                        Traceback (most recent call last)
~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/xarray_io.py in data_array_to_dep(arr, fname, tag, **dep_kwargs)
    272     try:
--> 273         attrs = case_insensitive_attrs(arr.attrs, typ_str)
    274     except MissingDepMetadata:

~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/xarray_io.py in case_insensitive_attrs(attrs, typ)
    146         missing = needs_keys - has_keys
--> 147         raise MissingDepMetadata('Did not find the following keys in DataArray: {} (found {})'.format(missing, lower))
    148     for k in OPTIONAL_DEP_FIELDS:

MissingDepMetadata: Did not find the following keys in DataArray: {'min', 'max', 'data_scale', 'z_units', 'east', 'west', 'north', 'south', 'cols', 'rows', 'xy_units'} (found {'crs': '+init=epsg:4269', 'palette_nonlinearity': 'high_relief.pal', 'is_tiled': 0, 'dtype': 'float', 'res': (0.00027813488916465513, 0.0002784031233404501), 'transform': (-91.48907218202051, 0.00027813488916465513, 0.0, 32.2297776796523, 0.0, -0.0002784031233404501)})

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-10-c1b6f3a30746> in <module>()
      1 # using elevation_raster read in by xarray rasterio fails, directly using the dem file works
      2 elevation_raster = xr.open_rasterio(elev_file)
----> 3 filled_dem = BreachDepressions(dem=elevation_raster)
      4 d8 = D8Pointer(dem=filled_dem)
      5 flowacc = D8FlowAccumulation(dem=filled_dem, out_type='sca')

~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/whitebox_cli.py in __call__(self, **kw)
    420         __doc__ = _fmt_help(tool)[0]
    421         def __call__(self, **kw):
--> 422             return validate_run(self._tool, **kw)
    423 
    424     globals()[tool] = Wrapped()

~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/whitebox_cli.py in validate_run(tool, **kwargs)
    410             raise ValueError('Parameter {} is not in {}'.format(k, ok_params))
    411     tool = partial(call_whitebox_func, tool, callback_func=partial(callback, silent=False))
--> 412     return tool(**kwargs)
    413 
    414 

~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/whitebox_cli.py in call_whitebox_func(tool, **kwargs)
    356         callback_func = partial(callback, silent=True)
    357     args = argparse.Namespace(**kwargs)
--> 358     return call_whitebox_cli(tool, args=args, callback_func=callback_func)
    359 
    360 

~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/whitebox_cli.py in call_whitebox_cli(tool, args, callback_func, silent, return_xarr, verbose)
    341                                              verbose=verbose)
    342         args = parser.parse_args()
--> 343     args, delayed_load_later = to_rust(tool, args)
    344     ret_val = wbt.run_tool(tool, args, callback_func, verbose=verbose)
    345     if ret_val:

~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/whitebox_cli.py in to_rust(tool, args)
    306         fname = os.path.join(WHITEBOX_TEMP_DIR, '{}-{}.dep'.format(output, tok))
    307         vars(args)[output] = fix_path(fname)
--> 308     delayed_load_later, kwargs = xarray_whitebox_io(**vars(args))
    309     for k, v in kwargs.items():
    310         if isinstance(v, bool):

~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/xarray_io.py in xarray_whitebox_io(**kwargs)
    323                 dumped_an_xarray = k
    324             elif isinstance(v, xr.DataArray):
--> 325                 kwargs[k] = data_array_to_dep(v, tag=k)[0]
    326                 dumped_an_xarray = k
    327         elif _is_output_field(k):

~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/xarray_io.py in data_array_to_dep(arr, fname, tag, **dep_kwargs)
    273         attrs = case_insensitive_attrs(arr.attrs, typ_str)
    274     except MissingDepMetadata:
--> 275         arr = add_dep_meta(arr, **dep_kwargs)
    276         attrs = case_insensitive_attrs(arr.attrs, typ_str)
    277     attrs['byte_order'] = 'LITTLE_ENDIAN'

~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/xarray_io.py in add_dep_meta(arr, projection, display_max, display_min, data_scale, z_units, xy_units, x_coord_name, y_coord_name, no_data, palette, palette_nonlinearity)
    366 
    367     if not v.ndim == 2:
--> 368         raise ValueError('Expected a 2-D raster (3-D array NotImplemented)')
    369     if no_data is None:
    370         if np.any(np.isnan(v)):

ValueError: Expected a 2-D raster (3-D array NotImplemented)

[snowman2]

elevation_raster = xr.open_rasterio(elev_file).isel(band=0)

Output:

<xarray.DataArray (y: 192, x: 237)>
array([[ 19.746029,  19.725756,  19.770012, ...,  20.486816,  20.450933,
         20.465611],
       [ 19.678217,  19.824434,  19.755169, ...,  20.472446,  20.485765,
         20.43626 ],
       [ 19.84296 ,  19.702999,  19.630199, ...,  20.44566 ,  20.475452,
         20.389683],
       ..., 
       [ 19.638786,  19.681364,  19.778816, ...,  20.851028,  20.916306,
         21.349106],
       [ 19.692333,  19.667782,  19.643562, ...,  20.881794,  21.244215,
         21.378166],
       [ 19.656511,  19.683601,  19.666609, ...,  21.069481,  21.35741 ,
         21.371601]], dtype=float32)
Coordinates:
    band     int64 1
  * y        (y) float64 32.23 32.23 32.23 32.23 32.23 32.23 32.23 32.23 ...
  * x        (x) float64 -91.49 -91.49 -91.49 -91.49 -91.49 -91.49 -91.49 ...
Attributes:
    crs:        +init=epsg:4269
    is_tiled:   0
    res:        (0.00027813488916465513, 0.0002784031233404501)
    transform:  (-91.48907218202051, 0.00027813488916465513, 0.0, 32.22977767...

filled_dem = BreachDepressions(dem=elevation_raster)

Output:

Running: /Users/rdchlads/tethys/miniconda/envs/earthsim/share/whitebox_tools/release/whitebox_tools '--run="BreachDepressions"' '--output="/Users/rdchlads/.whitebox_tools_tempdir/output-VyovrjP.dep"' '--dem="/Users/rdchlads/.whitebox_tools_tempdir/dem.dep"' '--wd="/Users/rdchlads/scripts/EarthSim/notebooks"' -v
********************************
* Welcome to BreachDepressions *
********************************
Reading data...
ERROR: thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ParseFloatError { kind: Invalid }', src/libcore/result.rs:860
note: Run with `RUST_BACKTRACE=1` for a backtrace.
/Users/rdchlads/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/ipykernel_launcher.py:2: FutureWarning: The value of this property will change in version 1.0. Please see https://github.com/mapbox/rasterio/issues/86 for details.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-13-bbb0f93b9e25> in <module>()
      1 # using elevation_raster read in by xarray rasterio fails, directly using the dem file works
      2 elevation_raster = xr.open_rasterio(elev_file).isel(band=0)
----> 3 filled_dem = BreachDepressions(dem=elevation_raster)
      4 d8 = D8Pointer(dem=filled_dem)
      5 flowacc = D8FlowAccumulation(dem=filled_dem, out_type='sca')

~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/whitebox_cli.py in __call__(self, **kw)
    420         __doc__ = _fmt_help(tool)[0]
    421         def __call__(self, **kw):
--> 422             return validate_run(self._tool, **kw)
    423 
    424     globals()[tool] = Wrapped()

~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/whitebox_cli.py in validate_run(tool, **kwargs)
    410             raise ValueError('Parameter {} is not in {}'.format(k, ok_params))
    411     tool = partial(call_whitebox_func, tool, callback_func=partial(callback, silent=False))
--> 412     return tool(**kwargs)
    413 
    414 

~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/whitebox_cli.py in call_whitebox_func(tool, **kwargs)
    356         callback_func = partial(callback, silent=True)
    357     args = argparse.Namespace(**kwargs)
--> 358     return call_whitebox_cli(tool, args=args, callback_func=callback_func)
    359 
    360 

~/tethys/miniconda/envs/earthsim/lib/python3.5/site-packages/whitebox_tools/whitebox_cli.py in call_whitebox_cli(tool, args, callback_func, silent, return_xarr, verbose)
    344     ret_val = wbt.run_tool(tool, args, callback_func, verbose=verbose)
    345     if ret_val:
--> 346         raise ValueError('WhiteBox {0} (wb-{0}) failed with args: {1}'.format(tool, args))
    347     arr_or_dset = delayed_load_later(ret_val)
    348     if return_xarr:

ValueError: WhiteBox BreachDepressions (wb-BreachDepressions) failed with args: ['--output="/Users/rdchlads/.whitebox_tools_tempdir/output-VyovrjP.dep"', '--dem="/Users/rdchlads/.whitebox_tools_tempdir/dem.dep"', '--wd="/Users/rdchlads/scripts/EarthSim/notebooks"']

[snowman2]

Here is the file I used: example_dem.zip

[PeterDSteinberg]

@dharhas @snowman2 - Thank you for the details. This will be easy for me to fix this week. In the meantime here's the workaround:

from whitebox_tools.xarray_io import add_dep_meta
from whitebox_tools import *
import xarray as xr
b = xr.open_rasterio('dc2e1333b55e48f99124ddf2c8ea8392.tif').isel(band=0,drop=True)
add_dep_meta(b) 
b.attrs['Display Max'] = 1.
b.attrs['Display Min'] = 0.
b.attrs['Nodata'] = -999999
BreachDepressions(dem=b)

Note that I had to give drop=True so that band is not retained as a singleton coordinate (currently only 2 dims are supported by the wrapper). Secondary problems were the lack of Display Max, Display Min and Nodata in attrs which were causing float parsing errors in rust. I'll make sure those get defined to something not-null and that the 2-D vs 3-D array bugs are fixed.