KeyError: "Input scene does not contain reference dataset"

[arthur-e]

Making progress! I'm not sure if the difficulties I'm experiencing with the Simple_SLSTR notebook stem from my lack of experience with satpy... But starting off from the changes made after issue #3, I have:

# Create a scene by searching for files
file_list = find_files_and_readers(base_dir = indir, reader = 'slstr_l1b')
scn = Scene(file_list, reader = 'slstr_l1b')

# Load the bands
scn.load(bnames)

# Additionally load the solar and satellite angles
# This saves computing them internally
scn.load(['solar_zenith_angle',
          'solar_azimuth_angle',
          'satellite_zenith_angle',
          'satellite_azimuth_angle'])

# Create the PyCoxMunk class for processing
pcm = PyCoxMunk(scn, bnames)

Unfortunately, I'm getting a KeyError:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File /usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/utils.py:346, in get_satpos(data_arr, preference, use_tle)
    345 try:
--> 346     lon, lat = _get_sat_lonlat(data_arr, lonlat_prefixes)
    347     alt = _get_sat_altitude(data_arr, alt_prefixes)

File /usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/utils.py:381, in _get_sat_lonlat(data_arr, key_prefixes)
    380 def _get_sat_lonlat(data_arr, key_prefixes):
--> 381     orb_params = data_arr.attrs["orbital_parameters"]
    382     lon_keys = [prefix + "longitude" for prefix in key_prefixes]

KeyError: 'orbital_parameters'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
File /usr/local/dev/PyCoxMunk2/pycoxmunk/CM_SceneGeom.py:39, in cm_calcangles(inscn, refband)
     38 suna, sunz = _get_sun_angles(inscn[refband])
---> 39 sata, satz = _get_sensor_angles(inscn[refband])
     41 inscn['satellite_azimuth_angle'] = inscn[refband].copy()

File /usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/modifiers/angles.py:445, in _get_sensor_angles(data_arr)
    444 preference = satpy.config.get('sensor_angles_position_preference', 'actual')
--> 445 sat_lon, sat_lat, sat_alt = get_satpos(data_arr, preference=preference)
    446 area_def = data_arr.attrs["area"]

File /usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/utils.py:354, in get_satpos(data_arr, preference, use_tle)
    353         return _get_satpos_from_platform_name(data_arr)
--> 354     raise KeyError("Unable to determine satellite position. Either the "
    355                    "reader doesn't provide that information or "
    356                    "geolocation datasets were not available.")
    357 return lon, lat, alt

KeyError: "Unable to determine satellite position. Either the reader doesn't provide that information or geolocation datasets were not available."

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
Cell In[4], line 2
      1 # Create the PyCoxMunk class for processing
----> 2 pcm = PyCoxMunk(scn, bnames)

File /usr/local/dev/PyCoxMunk2/pycoxmunk/CM_Main.py:132, in PyCoxMunk.__init__(self, scn, band_names, oc_dir, angle_names, do_brdf, mask_bad, delete_when_done)
    130     self.angle_names = angle_names
    131 if angle_names == 'calc':
--> 132     self.scn = cm_calcangles(scn, refband=band_names[0])
    134 if oc_dir is None:
    135     self.use_occci = False

File /usr/local/dev/PyCoxMunk2/pycoxmunk/CM_SceneGeom.py:51, in cm_calcangles(inscn, refband)
     49     inscn['solar_zenith_angle'].data = sunz
     50 except KeyError:
---> 51     raise KeyError("Input scene does not contain reference dataset, please check inputs.")
     52 except ValueError:
     53     raise ValueError("Cannot retrieve solar and satellite angles, please compute manually and specify.")

KeyError: 'Input scene does not contain reference dataset, please check inputs.'

I did check:

scn.available_dataset_ids()

[DataID(name='F1', wavelength=WavelengthRange(min=3.55, central=3.74, max=3.93, unit='µm'), resolution=1000, calibration=<calibration.brightness_temperature>, view=<view.nadir>, stripe=<stripe.f>, modifiers=()),
 DataID(name='F1', wavelength=WavelengthRange(min=3.55, central=3.74, max=3.93, unit='µm'), resolution=1000, calibration=<calibration.brightness_temperature>, view=<view.oblique>, stripe=<stripe.f>, modifiers=()),
 DataID(name='F2', wavelength=WavelengthRange(min=10.4, central=10.85, max=11.3, unit='µm'), resolution=1000, calibration=<calibration.brightness_temperature>, view=<view.nadir>, stripe=<stripe.i>, modifiers=()),
 ...
 DataID(name='solar_zenith_angle', resolution=500, view=<view.nadir>, modifiers=()),
 DataID(name='solar_zenith_angle', resolution=500, view=<view.oblique>, modifiers=()),
 DataID(name='solar_zenith_angle', resolution=1000, view=<view.nadir>, modifiers=()),
 DataID(name='solar_zenith_angle', resolution=1000, view=<view.oblique>, modifiers=())]

[simonrp84]

This turned out to be a stupid mistake on my part, if you pull the latest main then the SLSTR example should now work!

[arthur-e]

@simonrp84 I'm making progress! I can run the entire Example_SLSTR.ipynb notebook now without error. I see that output *.tif files are written to my output directory. This is what the resulting file cox_munk_refl_S3_20230123_191305.tif looks like:

All of the outputs seem to have data and georeference information except for:

S1_20230123_191305.tif
S2_20230123_191305.tif
S3_20230123_191305.tif

Each of these is files is too small (ca. 30.8 kb) and has no valid data or georeference information:

rasterio.open('/home/arthur.endsley/Downloads/PyCoxMunk/outputs/S1_20230123_191305.tif')
/usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/rasterio/__init__.py:304: NotGeoreferencedWarning: Dataset has no geotransform, gcps, or rpcs. The identity matrix will be returned.
  dataset = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
Out[3]: <open DatasetReader name='/home/arthur.endsley/Downloads/PyCoxMunk/outputs/S1_20230123_191305.tif' mode='r'>

If the output reflectance image (above) looks correct, maybe the issue with bands S1, S2, S3 isn't serious(?)

[simonrp84]

Hi @arthur-e - can you try updating your copy of the repo? The problem with SLSTR should be corrected in the latest commit to the example notebook...

[arthur-e]

I cloned a fresh upstream repo and installed again; latest commit was 03f5d618 on Mon Feb 6 14:10:41 2023.

I'll try again after installing with conda in a conda environment.

[arthur-e]

I'm getting the same result in a clean conda environment, with pycoxmunk==1.0.0 installed using conda-forge. Perhaps there is something wrong with the SLSTR granule I chose? I re-downloaded the granule but am getting the same result. Is there a reference SLSTR dataset I can use?

[simonrp84]

I think this is because you're installing the conda-forge version, which doesn't have the fix pushed to it*. You'll need the current github main.

Can you try these steps?

conda create --name test_pcm python=3.10
conda activate test_pcm

git clone git@github.com:simonrp84/PyCoxMunk.git
cd PyCoxMunk
pip install -e .
pip install notebook netcdf4 h5netcdf rasterio

Then run the notebook. This works in a fresh environment for me, so hopefully it'll work for you too!

* I don't want to make a new PyCoxMunk release until we know the fixes are working and any other minor problems are corrected. But once we're happy the problem is resolved I'll do a patch release :)

[arthur-e]

@simonrp84 With pip version 20.0.2 and Python 3.8, it seems it's not possible to install a repo in editable mode unless it has a setup.py file.

conda create --name test_pcm
conda activate test_pcm
pip install -e .

ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /usr/local/dev/PyCoxMunk
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)

Independently, I'm having trouble with conda, where the conda environment's Python binary is not on my path...

$ conda activate test_pcm
$ which python
/usr/bin/python

Not sure why it is giving me the system-wide Python installation. I get a similar result with which python3. I don't have this problem with virtualenv:

$ git clone git@github.com:simonrp84/PyCoxMunk.git
$ cd PyCoxMunk
$ virtualenv /usr/local/python-env/PyCoxMunk/
$ source /usr/local/python-env/PyCoxMunk/bin/activate
$ which python
/usr/local/python-env/PyCoxMunk/bin/python

Anyways, after installing a virtualenv with pip I'm getting the same results; everything in the Simple_SLSTR.ipynb notebook runs smoothly, without error, but the S1, S2, S3 band GeoTIFFs are each only about ~30 kb and seem to contain no valid data. The Cox-Munk reflectance files are each a smooth, horizontal gradient.

cd /usr/local/dev/PyCoxMunk
pip install -e .
pip install notebook netcdf4 h5netcdf rasterio
jupyter notebook

This is with the PyCoxMunk trunk and:

numpy==1.23.5
scipy==1.10.0
pandas==1.5.3
dask==2023.2.0
pyorbital==1.7.3
pyresample==1.26.1
satpy==0.40.0
click==8.1.3
numba==0.56.4
xarray==2023.1.0

I don't want to hold up the JOSS review. I doubt that it matters whether conda or virtualenv are used but...who knows? If this Notebook is working for @molinav then it must just be a problem with my setup. Thank goodness, for my own sake, I only do terrestrial remote sensing. :sweat_smile:

EDIT: In general, I try not to mix pip and conda, but I understand that there is not another way to install a local package in a conda environment(?) If you add a setup.py file, I could use python setup.py install within the repo. As you already have a setup.cfg file, the setup.py could be quite minimal, as in this example. In fact, I guess I can try adding it myself...

[simonrp84]

Thanks for the info. I'll be away for the next few days but will investigate more towards the end of next week.

Can you confirm that the GOES example works? You should be able to test that with the PyCoxMunk from conda / pip (without needing editable mode) and just downloading the example notebook itself from github...

[arthur-e]

@simonrp84 Yes, the GOES-ABI example works great for me. I did not get through Complex_SEVIRI.ipynb because I didn't have ERA5 re-analysis data handy.

I did figure out how to get Python into my conda environment. I had avoided the python= argument because I'm using a different version of Python; I figured conda would create my environment with the default Python installation (as is the case with virtualenv) but, in fact, it seems it must be expressly requested:

$ conda create -n PyCoxMunk python=python3.8
$ conda activate PyCoxMunk
$ which python
/usr/local/dev/miniconda3/envs/PyCoxMunk/bin/python
$ which pip
/usr/local/dev/miniconda3/envs/PyCoxMunk/bin/pip

Success with installing again, though I needed to activate Jupyter Notebook differently...

pip install -e .
pip install notebook netcdf4 h5netcdf rasterio
python -m notebook

Yet the result for Example_SLSTR.ipynb is the same, unfortunately.

I'm happy to move the review forward if you are able to add documentation on the test suite, when you return.

[simonrp84]

Hi, I've finally come back to look at PyCoxMunk after some time on other projects. Have just tested on a completely fresh virtual machine:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
[ install conda]
conda install mamba
mamba install pycoxmunk jupyter

I then downloaded the Example_SLSTR.ipynb notebook and a sample SLSTR file from https://dataspace.copernicus.eu/ Then I ran the notebook, and the results were as expected. So unfortunately I cannot reproduce the problem you have experienced here and am not sure what the best approach to resolve it is. But I am happy that the GOES notebook at least worked successfully!

[pdebuyl]

I can't reproduce either (for info)

[simonrp84]

@arthur-e Just to follow up on this - can I close this issue?

[arthur-e]

Sure, no problem.