search

pytroll / pyspectral

Pyspectral is a package to read and manipulate satellite sensor spectral responses and solar irradiance spectra
https://pyspectral.readthedocs.org/
GNU General Public License v3.0
69 stars 39 forks source link

Failure to get RSR for AVHRR band '3b' via SatPy 'day_microphysical' composite #41

Closed pnuu closed 6 years ago

pnuu commented 6 years ago

Code Sample, a minimal, complete, and verifiable piece of code

from satpy import Scene
fname = "hrpt_noaa18_20180731_0934_67992.l1b"
glbl = Scene(reader='avhrr_aapp_l1b', filenames=[fname,])
glbl.load(['day_microphysics'])

Problem description

Use of PySpectral via SatPy for AVHRR data fails. The above script works fine for MSG/SEVIRI data.

Actual Result, Traceback if applicable

Traceback relevant to PySpectral:

[WARNING: 2018-08-09 12:04:56 : pyspectral.rsr_reader] Inconsistent instrument/satellite input - instrument set to avhrr/3
[DEBUG: 2018-08-09 12:04:56 : pyspectral.rsr_reader] Filename: /home/lahtinep/.local/share/pyspectral/rsr_avhrr3_NOAA-18.h5
[WARNING: 2018-08-09 12:04:56 : pyspectral.rsr_reader] No rsr file /home/lahtinep/.local/share/pyspectral/rsr_avhrr3_NOAA-18.h5 on disk
[INFO: 2018-08-09 12:04:56 : pyspectral.rsr_reader] Will download from internet...
[DEBUG: 2018-08-09 12:04:57 : urllib3.connectionpool] Starting new HTTPS connection (1): zenodo.org
[DEBUG: 2018-08-09 12:04:57 : urllib3.connectionpool] https://zenodo.org:443 "GET /record/1205138/files/pyspectral_rsr_data.tgz HTTP/1.1" 200 2798376
2798376it [00:02, 1097715.00it/s]
[WARNING: 2018-08-09 12:05:00 : pyspectral.rsr_reader] rsr data may not be up to date: /home/lahtinep/.local/share/pyspectral/rsr_avhrr3_NOAA-18.h5
[INFO: 2018-08-09 12:05:00 : pyspectral.rsr_reader] Will download from internet...
[DEBUG: 2018-08-09 12:05:00 : urllib3.connectionpool] Starting new HTTPS connection (1): zenodo.org
[DEBUG: 2018-08-09 12:05:01 : urllib3.connectionpool] https://zenodo.org:443 "GET /record/1205138/files/pyspectral_rsr_data.tgz HTTP/1.1" 200 2798376
2798376it [00:02, 1095147.97it/s]
[DEBUG: 2018-08-09 12:05:04 : pyspectral.rsr_reader] Filename: /home/lahtinep/.local/share/pyspectral/rsr_avhrr3_NOAA-18.h5
[DEBUG: 2018-08-09 12:05:04 : pyspectral.rsr_reader] No detectors found - assume only one...
[WARNING: 2018-08-09 12:05:04 : pyspectral.rsr_reader] Inconsistent instrument/satellite input - instrument set to avhrr/3
[DEBUG: 2018-08-09 12:05:04 : pyspectral.rsr_reader] Filename: /home/lahtinep/.local/share/pyspectral/rsr_avhrr3_NOAA-18.h5
[DEBUG: 2018-08-09 12:05:04 : pyspectral.rsr_reader] Filename: /home/lahtinep/.local/share/pyspectral/rsr_avhrr3_NOAA-18.h5
[DEBUG: 2018-08-09 12:05:04 : pyspectral.rsr_reader] No detectors found - assume only one...
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-31-2c22589a44ad> in <module>()
----> 1 glbl.load(['day_microphysics'])

~/Software/pytroll/packages/satpy/satpy/scene.py in load(self, wishlist, calibration, resolution, polarization, level, generate, unload, **kwargs)
    848         self.read(**kwargs)
    849         if generate:
--> 850             keepables = self.generate_composites()
    851         else:
    852             # don't lose datasets we loaded to try to generate composites

~/Software/pytroll/packages/satpy/satpy/scene.py in generate_composites(self, nodes)
    763             nodes = set(self.dep_tree.trunk(nodes=required_nodes)) - \
    764                 set(self.datasets.keys())
--> 765         return self._read_composites(nodes)
    766 
    767     def _remove_failed_datasets(self, keepables):

~/Software/pytroll/packages/satpy/satpy/scene.py in _read_composites(self, compositor_nodes)
    737         keepables = set()
    738         for item in compositor_nodes:
--> 739             self._generate_composite(item, keepables)
    740         return keepables
    741 

~/Software/pytroll/packages/satpy/satpy/scene.py in _generate_composite(self, comp_node, keepables)
    710             composite = compositor(prereq_datasets,
    711                                    optional_datasets=optional_datasets,
--> 712                                    **self.attrs)
    713 
    714             cid = DatasetID.from_dict(composite.attrs)

~/Software/pytroll/packages/satpy/satpy/composites/__init__.py in __call__(self, projectables, optional_datasets, **info)
    468         for wavelength outside [3, 4] µm.
    469         """
--> 470         self._init_refl3x(projectables)
    471         _nir, _ = projectables
    472         refl = self._get_reflectance(projectables, optional_datasets) * 100

~/Software/pytroll/packages/satpy/satpy/composites/__init__.py in _init_refl3x(self, projectables)
    489 
    490         _nir, _tb11 = projectables
--> 491         self._refl3x = Calculator(_nir.attrs['platform_name'], _nir.attrs['sensor'], _nir.attrs['name'])
    492 
    493     def _get_reflectance(self, projectables, optional_datasets):

~/Software/miniconda3/lib/python3.6/site-packages/pyspectral/near_infrared_reflectance.py in __init__(self, platform_name, instrument, band, **kwargs)
     59 
     60     def __init__(self, platform_name, instrument, band, **kwargs):
---> 61         super(Calculator, self).__init__(platform_name, instrument, band, **kwargs)
     62 
     63         from numbers import Number

~/Software/miniconda3/lib/python3.6/site-packages/pyspectral/radiance_tb_conversion.py in __init__(self, platform_name, instrument, band, **options)
    126         self.rsr_integral = 1.0
    127 
--> 128         self._get_rsr()
    129 
    130     def _get_rsr(self):

~/Software/miniconda3/lib/python3.6/site-packages/pyspectral/radiance_tb_conversion.py in _get_rsr(self)
    152             self.bandname = get_bandname_from_wavelength(self.instrument, self.band, self.rsr)
    153 
--> 154         self.wavelength_or_wavenumber = (self.rsr[self.bandname][self.detector][self.wavespace] *
    155                                          self._wave_si_scale)
    156         self.response = self.rsr[self.bandname][self.detector]['response']

KeyError: '3b'

Here '3b' is the channel name SatPy uses for 3.8 um AVHRR channel. Looking at the HDF5 file in ~/.local/share/pyspectral/ the group ("folder") names are ch1, ... ch3b, ... ch5, so there is an inconsistency in the channel naming between SatPy and PySpectral.

Versions of Python, package at hand and relevant dependencies

Conda, Python 3.6, current PySpectral master branch, current SatPy master branch.

adybbroe commented 6 years ago

@pnuu I don't get the same error as you! I seem to lack a platform_name? I run Python 3.6.4, latest pyspectral, pyorbital and satpy master branches. So, what's happening?

~/usr/src/satpy/satpy/composites/__init__.py in __call__(self, projectables, optional_datasets, **info)
    468         for wavelength outside [3, 4] µm.
    469         """
--> 470         self._init_refl3x(projectables)
    471         _nir, _ = projectables
    472         refl = self._get_reflectance(projectables, optional_datasets) * 100

~/usr/src/satpy/satpy/composites/__init__.py in _init_refl3x(self, projectables)
    489 
    490         _nir, _tb11 = projectables
--> 491         self._refl3x = Calculator(_nir.attrs['platform_name'], _nir.attrs['sensor'], _nir.attrs['name'])
    492 
    493     def _get_reflectance(self, projectables, optional_datasets):

KeyError: 'platform_name'
pnuu commented 6 years ago

I have a fix for the platform_name, I'll create PR shortly. You can also check satpy branch bugfix-aapp-platform_name.

adybbroe commented 6 years ago

Ok, nice. I am looking at it!

pnuu commented 6 years ago

PR for SatPy created: https://github.com/pytroll/satpy/pull/386

adybbroe commented 6 years ago

Ok, can you try my new PR branch: fix_avhrr_ch3b_naming @pnuu ? I get another error in satpy then...

~/usr/src/satpy/satpy/composites/__init__.py in _get_reflectance(self, projectables, optional_datasets)
    500 
    501         for dataset in optional_datasets:
--> 502             if (dataset.attrs['units'] == 'K' and
    503                     "wavelengh" in dataset.attrs and
    504                     dataset.attrs["wavelength"][0] <= 13.4 <= dataset.attrs["wavelength"][2]):

KeyError: 'units'
pnuu commented 6 years ago

I'll take a look on monday :-)

adybbroe commented 6 years ago

Thanks @pnuu . I believe I solved it now in the PR pytroll/satpy#386 Waiting for that to be merged....

pnuu commented 6 years ago

Great, now (with your update to pytroll/satpy#386) everything I've tried works!

adybbroe commented 6 years ago

Great, thanks @pnuu