EMIT L2A reflectance bug on CyVerse

[serbinsh]

Using the HYR-SENSE kernel, hit a snag. Need to look into this to ensure we can open reflectance data following a similar notebook

My guess is its not loading

import sys
sys.path.append('../modules/')
from emit_tools import emit_xarray
help(emit_xarray)

properly

[Ckster]

Looks like it's missing the h5 engine. I've run into this before and will try and update it to include the engine

[serbinsh]

@Ckster FYI - working on a notebook and ran into this issue again trying to use emit_tools. I think we need to be sure the HYR-SENSE kernal has the hdf5bindings installed so we can work through and develop our tutorials. Is it possible to make sure that gets added ASAP so we can work on these example?

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[7], line 1
----> 1 ds = et.emit_xarray(fp, ortho=True)
      2 ds

File [~/HYR-SENSE/notebooks/agriculture/../../tools/emit/python/modules/emit_tools.py:61](https://a0db691fb.cyverse.run/lab/tree/HYR-SENSE/notebooks/agriculture/HYR-SENSE/tools/emit/python/modules/emit_tools.py#line=60), in emit_xarray(filepath, ortho, qmask, unpacked_bmask)
     58 # Read in Data as Xarray Datasets
     59 engine, wvl_group = "h5netcdf", None
---> 61 ds = xr.open_dataset(filepath, engine=engine)
     62 loc = xr.open_dataset(filepath, engine=engine, group="location")
     64 # Check if mineral dataset and read in groups (only ds[/loc](https://a0db691fb.cyverse.run/loc) for minunc)

File [/opt/conda/envs/hyr-sense/lib/python3.10/site-packages/xarray/backends/api.py:558](https://a0db691fb.cyverse.run/opt/conda/envs/hyr-sense/lib/python3.10/site-packages/xarray/backends/api.py#line=557), in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
    555 if from_array_kwargs is None:
    556     from_array_kwargs = {}
--> 558 backend = plugins.get_backend(engine)
    560 decoders = _resolve_decoders_kwargs(
    561     decode_cf,
    562     open_backend_dataset_parameters=backend.open_dataset_parameters,
   (...)
    568     decode_coords=decode_coords,
    569 )
    571 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)

File [/opt/conda/envs/hyr-sense/lib/python3.10/site-packages/xarray/backends/plugins.py:205](https://a0db691fb.cyverse.run/opt/conda/envs/hyr-sense/lib/python3.10/site-packages/xarray/backends/plugins.py#line=204), in get_backend(engine)
    203     engines = list_engines()
    204     if engine not in engines:
--> 205         raise ValueError(
    206             f"unrecognized engine {engine} must be one of: {list(engines)}"
    207         )
    208     backend = engines[engine]
    209 elif isinstance(engine, type) and issubclass(engine, BackendEntrypoint):

ValueError: unrecognized engine h5netcdf must be one of: ['netcdf4', 'scipy', 'pydap', 'rasterio', 'store', 'zarr']

Thanks!

I was trying to experiment with this notebook to make sure we can manipulate EMIT data https://github.com/CU-ESIIL/HYR-SENSE/blob/serbinsh_ag/notebooks/agriculture/How_to_Convert_to_ENVI.ipynb

[serbinsh]

In our draft dependency check notebook I added listing the xarray backends and indeed hdf5 is missing

According to this the best way to install it into the image is

https://docs.xarray.dev/en/latest/getting-started-guide/faq.html

conda install -c conda-forge h5netcdf

But first in terminal had to do

jovyan@a0db691fb:~/HYR-SENSE$ conda info --env
# conda environments:
#
base                     /opt/conda
hyr-sense                /opt/conda/envs/hyr-sense

jovyan@a0db691fb:~/HYR-SENSE$ conda activate hyr-sense
(hyr-sense) jovyan@a0db691fb:~/HYR-SENSE$ 

then install

[serbinsh]

I just manually installed into the kernel

conda install -c conda-forge h5netcdf

Re ran the depends check and hdf5 is there. going to test an example emit script now

[serbinsh]

Yup - after running that install the script is working

[Ckster]

Thanks for finding the fix @serbinsh, h5netcdf was added to the environment.yml file and the new image was pushed to DockerHub. New deployments of the hyr-sense app will have it installed.

[serbinsh]

Thanks for finding the fix @serbinsh, h5netcdf was added to the environment.yml file and the new image was pushed to DockerHub. New deployments of the hyr-sense app will have it installed.

Excellent - thank you!