search

pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.06k stars 292 forks source link

MTG LI reader is not Python 3 compatible #658

Closed gerritholl closed 1 year ago

gerritholl commented 5 years ago

Describe the bug

The MTG LI reader is not Python 3 compatible.

To Reproduce


from satpy.scene import Scene
from satpy.utils import debug_on; debug_on()
import glob
from datetime import datetime
lilist = glob.glob("/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/*0116.*")
liscene = Scene(sensor="li", reader='li_l2', filenames=lilist)
liscene.load(['af'])

Expected behavior

Ideally, this script should have no output apart from debug output, as it should load the data and do nothing with it.

Actual results

When run with Python 3.7, the output is:

[DEBUG: 2019-03-14 13:55:23 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/geotools/etc/'
[DEBUG: 2019-03-14 13:55:23 : satpy.readers] Reading ['/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/satpy/etc/readers/li_l2.yaml']
[DEBUG: 2019-03-14 13:55:23 : satpy.readers.yaml_reader] Assigning to li_l2: ['/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324083543_L2PF_DEV_20130620191630_20130620191700_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324083543_L2PF_DEV_20130620191600_20130620191630_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324075315_L2PF_DEV_20130620191030_20130620191100_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324082249_L2PF_DEV_20130620191400_20130620191430_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324082906_L2PF_DEV_20130620191500_20130620191530_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324082249_L2PF_DEV_20130620191430_20130620191500_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324082906_L2PF_DEV_20130620191530_20130620191600_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324080835_L2PF_DEV_20130620191200_20130620191230_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324084208_L2PF_DEV_20130620191700_20130620191730_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324084208_L2PF_DEV_20130620191730_20130620191800_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324085634_L2PF_DEV_20130620191900_20130620191930_N__T_0116.nc', '/media/nas/x21308/Testdata/MT
G-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324080835_L2PF_DEV_20130620191230_20130620191300_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324084915_L2PF_DEV_20130620191800_20130620191830_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324075315_L2PF_DEV_20130620191000_20130620191030_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324081615_L2PF_DEV_20130620191330_20130620191400_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324080043_L2PF_DEV_20130620191130_20130620191200_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324080043_L2PF_DEV_20130620191100_20130620191130_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324085634_L2PF_DEV_20130620191930_20130620192000_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324081615_L2PF_DEV_20130620191300_20130620191330_N__T_0116.nc', '/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324084915_L2PF_DEV_20130620191830_20130620191900_N__T_0116.nc']
[DEBUG: 2019-03-14 13:55:23 : satpy.readers.li_l2] Dimension : [   1    1 5568 5568]
[DEBUG: 2019-03-14 13:55:23 : satpy.readers.li_l2] Row/Cols: 5568 / 5568
[DEBUG: 2019-03-14 13:55:23 : satpy.readers.li_l2] Reading: /media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/W_XX-EUMETSAT-DARMSTADT,101,MTI1+LI-2-AF--FD--CHK-BODY--DIS-NC4E_C_EUMT_20160324083543_L2PF_DEV_20130620191630_20130620191700_N__T_0116.nc
Traceback (most recent call last):
  File "mwe.py", line 6, in <module>
    liscene = Scene(sensor="li", reader='li_l2', filenames=lilist)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/satpy/scene.py", line 156, in __init__
    reader_kwargs=reader_kwargs)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/satpy/scene.py", line 201, in create_reader_instances
    ppp_config_dir=self.ppp_config_dir)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/satpy/readers/__init__.py", line 748, in load_readers
    reader_instance.create_filehandlers(loadables, fh_kwargs=reader_kwargs)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 506, in create_filehandlers
    fh_kwargs=fh_kwargs)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 492, in new_filehandlers_for_filetype
    return list(filtered_iter)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 464, in filter_fh_by_metadata
    for filehandler in filehandlers:
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 407, in new_filehandler_instances
    yield filetype_cls(filename, filename_info, filetype_info, *req_fh, **fh_kwargs)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/satpy/readers/li_l2.py", line 60, in __init__
    logger.debug('Start: {}'.format(self.start_time))
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/satpy/readers/li_l2.py", line 65, in start_time
    return datetime.strptime(self.nc.attrs['sensing_start'], '%Y%m%d%H%M%S')
TypeError: strptime() argument 1 must be str, not numpy.bytes_
Exception ignored in: <function File.close at 0x7f45ef01b7b8>
Traceback (most recent call last):
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/h5netcdf/core.py", line 665, in close
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/h5py/_hl/files.py", line 412, in close
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 264, in h5py.h5f.get_obj_ids
  File "h5py/h5i.pyx", line 43, in h5py.h5i.wrap_identifier
ImportError: sys.meta_path is None, Python is likely shutting down

Environment Info:

Incidentally, this exposes a Python-2-only dependency on pyninjotiff, unrelated to the present problem:

Readers
=======                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
abi_l1b: ok                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
abi_l1b_scmi: ok
acspo: ok
ahi_hrit: ok
ahi_hsd: ok
amsr2_l1b: ok
avhrr_l1b_aapp: ok
avhrr_l1b_eps: ok
avhrr_l1b_gaclac: cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac')
avhrr_l1b_hrpt: cannot find module 'satpy.readers.hrpt' (No module named 'pygac')
caliop_l2_cloud: cannot find module 'satpy.readers.caliop_l2_cloud' (No module named 'pyhdf')
clavrx: cannot find module 'satpy.readers.clavrx' (No module named 'pyhdf')
electrol_hrit: ok
fci_l1c_fdhsi: ok
generic_image: ok
geocat: ok
ghrsst_l3c_sst: ok
goes-imager_hrit: ok
goes-imager_nc: ok
grib: cannot find module 'satpy.readers.grib' (No module named 'pygrib')
iasi_l2: ok
jami_hrit: ok
li_l2: ok
maia: ok
modis_l1b: cannot find module 'satpy.readers.modis_l1b' (No module named 'pyhdf')
msi_safe: cannot find module 'satpy.readers.msi_safe' (No module named 'glymur')
mtsat2-imager_hrit: ok
nucaps: ok
nwcsaf-geo: ok
nwcsaf-pps_nc: ok
olci_l1b: ok
olci_l2: ok
omps_edr: ok
sar-c_safe: ok
scatsat1_l2b: ok
seviri_l1b_hrit: ok
seviri_l1b_native: ok
seviri_l1b_nc: ok
slstr_l1b: ok
viirs_compact: ok
viirs_edr_flood: cannot find module 'satpy.readers.viirs_edr_flood' (No module named 'pyhdf')
viirs_l1b: ok
viirs_sdr: ok

Writers
=======
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/satpy/config.py", line 180, in check_satpy
    for writer, res in sorted(check_yaml_configs(configs_for_writer(), 'writer', 3).items()):
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/satpy/config.py", line 149, in check_yaml_configs
    res = yaml.load(stream)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/yaml/__init__.py", line 72, in load
    return loader.get_single_data()
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/yaml/constructor.py", line 37, in get_single_data
    return self.construct_document(node)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/yaml/constructor.py", line 46, in construct_document
    for dummy in generator:
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/yaml/constructor.py", line 398, in construct_yaml_map
    value = self.construct_mapping(node)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/yaml/constructor.py", line 204, in construct_mapping
    return super().construct_mapping(node, deep=deep)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/yaml/constructor.py", line 129, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/yaml/constructor.py", line 88, in construct_object
    data = constructor(self, tag_suffix, node)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/yaml/constructor.py", line 537, in construct_python_name
    return self.find_python_name(suffix, node.start_mark)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/yaml/constructor.py", line 521, in find_python_name
    __import__(module_name)
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/satpy/writers/ninjotiff.py", line 29, in <module>
    import pyninjotiff.ninjotiff as nt
  File "/media/nas/x21324/miniconda3/envs/py37b/lib/python3.7/site-packages/pyninjotiff/ninjotiff.py", line 1077
    print ""
           ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("")?

Additional context

This problem is caused by the str/bytes distinction in Python 3 and how NumPy handles it. It is resolved by #657. The same problem may affect other readers as well.

gerritholl commented 1 year ago

Fixed by #2271