search

pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.06k stars 294 forks source link

Fails to load data for modis Hkm true_color product #744

Open TAlonglong opened 5 years ago

TAlonglong commented 5 years ago

Failes to load data for modis Hkm true_color product.

from satpy.scene import Scene

files = ['/data/seadas/l1b/MOD03_A19119_212806_2019119214042.hdf',
         '/data/seadas/l1b/MOD02Hkm_A19119_212806_2019119214042.hdf']

scn = Scene(filenames=files, reader='modis_l1b')

scn.load(['true_color'])

This gives:

PPP_CONFIG_DIR=/software/pytroll/etc/satpy/etc PYTHONPATH=/software/pytroll/satpy-metno-local/:/software/pytroll/lib/python3.5/site-packages/ python3 /software/pytroll/scripts/test-modis.py 
/software/pytroll/lib/python3.5/site-packages/pyspectral/config.py:75: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = recursive_dict_update(config, yaml.load(fp_))
Traceback (most recent call last):
  File "/software/pytroll/scripts/test-modis.py", line 8, in <module>
    scn.load(['true_color'])
  File "/software/pytroll/lib/python3.5/site-packages/satpy/scene.py", line 941, in load
    raise KeyError("Unknown datasets: {}".format(unknown_str))
KeyError: "Unknown datasets: DatasetID(name='3', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=('sunz_corrected',)), DatasetID(name='1', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=('sunz_corrected',)), DatasetID(name='4', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=('sunz_corrected',))"

Environment Info:

pasing resolution=500 to load gives the same result

djhoese commented 5 years ago

Can you turn on debug logging and paste the log here?

TAlonglong commented 5 years ago

with debug_on

/software/pytroll/lib/python3.5/site-packages/pyspectral/config.py:75: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = recursive_dict_update(config, yaml.load(fp_))
[DEBUG: 2019-04-30 12:25:12 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/software/pytroll/etc/satpy/etc'
[DEBUG: 2019-04-30 12:25:12 : satpy.readers] Reading ['/software/pytroll/lib/python3.5/site-packages/satpy/etc/readers/modis_l1b.yaml']
[DEBUG: 2019-04-30 12:25:12 : satpy.readers.yaml_reader] Assigning to modis_l1b: ['/data/seadas/l1b/MOD03_A19119_212806_2019119214042.hdf', '/data/seadas/l1b/MOD02Hkm_A19119_212806_2019119214042.hdf']
[DEBUG: 2019-04-30 12:25:12 : satpy.composites] Looking for composites config file modis.yaml
Traceback (most recent call last):
  File "/software/pytroll/scripts/test-modis.py", line 11, in <module>
    scn.load(['true_color'])
  File "/software/pytroll/lib/python3.5/site-packages/satpy/scene.py", line 941, in load
    raise KeyError("Unknown datasets: {}".format(unknown_str))
KeyError: "Unknown datasets: DatasetID(name='1', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=('sunz_corrected',)), DatasetID(name='3', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=('sunz_corrected',)), DatasetID(name='4', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=('sunz_corrected',))"
djhoese commented 5 years ago

What's strange to me is it isn't saying that the '1' dataset isn't available or a specific resolution isn't available, it's saying the sunz_corrected version of these bands is unknown. Are you able to load the individual ['1', '3', '4'] bands in .load?

TAlonglong commented 5 years ago

yes, load of individual channels works if I say resolution=500

[DEBUG: 2019-04-30 12:31:13 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/software/pytroll/etc/satpy/etc'
[DEBUG: 2019-04-30 12:31:13 : satpy.readers] Reading ['/software/pytroll/lib/python3.5/site-packages/satpy/etc/readers/modis_l1b.yaml']
[DEBUG: 2019-04-30 12:31:13 : satpy.readers.yaml_reader] Assigning to modis_l1b: ['/data/seadas/l1b/MOD03_A19119_212806_2019119214042.hdf', '/data/seadas/l1b/MOD02Hkm_A19119_212806_2019119214042.hdf']
[DEBUG: 2019-04-30 12:31:13 : satpy.composites] Looking for composites config file modis.yaml
[DEBUG: 2019-04-30 12:31:13 : satpy.readers.yaml_reader] No coordinates found for DatasetID(name='latitude', wavelength=None, resolution=500, polarization=None, calibration=None, level=None, modifiers=())
[DEBUG: 2019-04-30 12:31:13 : satpy.readers.modis_l1b] Interpolating from 1000 to 500
[DEBUG: 2019-04-30 12:31:13 : satpy.readers.yaml_reader] No coordinates found for DatasetID(name='longitude', wavelength=None, resolution=500, polarization=None, calibration=None, level=None, modifiers=())
djhoese commented 5 years ago

@TAlonglong Can you try this again with current master? I think this may have been fixed.

TAlonglong commented 5 years ago

Hm, I think its the same.

[DEBUG: 2019-05-08 09:18:33 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/software/pytroll/etc/satpy/etc'
[DEBUG: 2019-05-08 09:18:33 : satpy.readers] Reading ['/software/pytroll/satpy-latest-install/lib/python3.5/site-packages/satpy-0.14.2+70.g8fd35b3-py3.5.egg/satpy/etc/readers/modis_l1b.yaml']
[DEBUG: 2019-05-08 09:18:33 : satpy.readers.yaml_reader] Assigning to modis_l1b: ['/data/seadas/l1b/MYD03_A19127_143413_2019127144108.hdf', '/data/seadas/l1b/MYD02Hkm_A19127_143413_2019127144108.hdf']
[DEBUG: 2019-05-08 09:18:33 : satpy.composites] Looking for composites config file modis.yaml
Traceback (most recent call last):
  File "/software/pytroll/scripts/test-modis.py", line 10, in <module>
    scn.load(['true_color'], resolution=500)
  File "/software/pytroll/satpy-latest-install/lib/python3.5/site-packages/satpy-0.14.2+70.g8fd35b3-py3.5.egg/satpy/scene.py", line 968, in load
    raise KeyError("Unknown datasets: {}".format(unknown_str))
KeyError: "Unknown datasets: DatasetID(name='1', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=('sunz_corrected',)), DatasetID(name='3', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=('sunz_corrected',)), DatasetID(name='4', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=('sunz_corrected',))"