pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.07k stars 295 forks source link

MTG LI reader fails #650

Closed gerritholl closed 1 year ago

gerritholl commented 5 years ago

Describe the bug

The MTG-LI reader fails with recent versions of satpy.

To Reproduce

from satpy.scene import Scene
import glob
from datetime import datetime

lilist = glob.glob("/media/nas/x21308/Testdata/MTG-LI/DIS/LI-2-AF/*0116.*")

liscene = Scene(sensor="li",
                start_time=datetime(2013, 6, 20, 19, 14, 1),
                end_time=datetime(2013, 6, 20, 19, 14, 29),
                reader='li_l2',
                filenames=lilist)

liscene.load(['af'])

Actual results

Using satpy versions 0.9.0—0.9.4 or newer up to latest master, it gives AttributeError: 'NoneType' object has no attribute 'data':

/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/scene.py:133: UserWarning: Deprecated: Use 'filter_parameters' to filter loaded files by 'start_time', 'end_time', or 'area'.
  "'end_time', or 'area'.")
Traceback (most recent call last):
  File "mwe.py", line 13, in <module>
    liscene.load(['af'])
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/scene.py", line 848, in load
    self.read(**kwargs)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/scene.py", line 756, in read
    return self._read_datasets(nodes, **kwargs)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/scene.py", line 615, in _read_datasets
    new_datasets = reader_instance.load(ds_ids, **kwargs)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/readers/yaml_reader.py", line 792, in load
    ds = self._load_dataset_with_area(dsid, coords)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/readers/yaml_reader.py", line 732, in _load_dataset_with_area
    ds = self._load_dataset_data(file_handlers, dsid, **slice_kwargs)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/readers/yaml_reader.py", line 586, in _load_dataset_data
    proj = self._load_dataset(dsid, ds_info, file_handlers)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/readers/yaml_reader.py", line 558, in _load_dataset
    projectable = fh.get_dataset(dsid, ds_info)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/readers/li_l2.py", line 120, in get_dataset
    out.data[:] = np.ma.getdata(ds)
AttributeError: 'NoneType' object has no attribute 'data'

Using versions 0.7.3–0.7.8 or 0.8.0–0.8.4, it is successful (although prints a warning starting with 0.8.0):

/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/scene.py:128: UserWarning: Deprecated: Use 'filter_parameters' to filter loaded files by 'start_time', 'end_time', or 'area'.
  "'end_time', or 'area'.")
No handlers could be found for logger "satpy.readers.li_l2"

Using versions 0.7.2 or older:

Traceback (most recent call last):
  File "mwe.py", line 11, in <module>
    filenames=lilist)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/scene.py", line 101, in __init__
    reader_kwargs=reader_kwargs)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/scene.py", line 147, in create_reader_instances
    reader_kwargs=reader_kwargs)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/readers/__init__.py", line 318, in __call__
    **reader_kwargs)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/readers/__init__.py", line 253, in load_reader
    reader_info = read_reader_config(reader_configs)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/readers/__init__.py", line 238, in read_reader_config
    conf.update(yaml.load(fd.read()))
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/yaml/__init__.py", line 71, in load
    return loader.get_single_data()
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/yaml/constructor.py", line 39, in get_single_data
    return self.construct_document(node)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/yaml/constructor.py", line 48, in construct_document
    for dummy in generator:
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/yaml/constructor.py", line 398, in construct_yaml_map
    value = self.construct_mapping(node)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/yaml/constructor.py", line 208, in construct_mapping
    return BaseConstructor.construct_mapping(self, node, deep=deep)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/yaml/constructor.py", line 133, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/yaml/constructor.py", line 90, in construct_object
    data = constructor(self, tag_suffix, node)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/yaml/constructor.py", line 522, in construct_python_name
    return self.find_python_name(suffix, node.start_mark)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/yaml/constructor.py", line 505, in find_python_name
    __import__(module_name)
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/site-packages/satpy/readers/yaml_reader.py", line 83, in <module>
    class AbstractYAMLReader(six.with_metaclass(ABCMeta, object)):
  File "/media/nas/x21324/miniconda3/envs/py27b/lib/python2.7/abc.py", line 87, in __new__
    cls = super(ABCMeta, mcls).__new__(mcls, name, bases, namespace)
TypeError: Error when calling the metaclass bases
    metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

I have tried to use git bisect, but there are too many merges going and different combinations of problems on different sides for me to find the commit that caused this problem, I also hit intermediate commits failing for different reasons yet (and then I'm unsure about good or bad, and skip just hits me another commit with the same problem).

Environment Info:

gerritholl commented 5 years ago

I have a hard time bisecting this:

It looks like there's too much non-linear history going on, I don't succeed in figuring out. This is my rather inefficient and also inconclusive git bisect log:

# good: [06248d5e4fee1efad3ac8e6cff6d46657639a6d0] update changelog
git bisect good 06248d5e4fee1efad3ac8e6cff6d46657639a6d0
# bad: [b574b1f755a5e63a6b6760ce6e425e8e105b0aae] Bump version: 0.8.1 → 0.9.0-alpha.0
git bisect bad b574b1f755a5e63a6b6760ce6e425e8e105b0aae
# bad: [774f7939a1b64b4382581e98d1c996d598acc4f9] Bugfix: remove unneeded +1
git bisect bad 774f7939a1b64b4382581e98d1c996d598acc4f9
# bad: [5da5a755774d6a7401ce980d6f4f44550d0de9a1] Remove unused dependencies
git bisect bad 5da5a755774d6a7401ce980d6f4f44550d0de9a1
# bad: [f67c47152ca0437fc6015a45b69531d379a04708] Do not resample datasets without an area
git bisect bad f67c47152ca0437fc6015a45b69531d379a04708
# bad: [f8f7502f89a989c74348161b557a7e88fdd3e691] Merge develop
git bisect bad f8f7502f89a989c74348161b557a7e88fdd3e691
# bad: [fe37a9ae9b638ac2c1fd4ed5b85d7135a0aff95a] Merge branch 'feature-xarray' of github.com:pytroll/satpy into feature-xarray
git bisect bad fe37a9ae9b638ac2c1fd4ed5b85d7135a0aff95a
# bad: [c603c1c63b569ef0eee8140a7c557bbc2ff7c692] Fix dask array depedency
git bisect bad c603c1c63b569ef0eee8140a7c557bbc2ff7c692
# good: [5fa54e7221bae206d147b3c8848320a7a64f4d48] Merge pull request #84 from eysteinn/composite-snowage-fix
git bisect good 5fa54e7221bae206d147b3c8848320a7a64f4d48
# bad: [7b18f701f45ed4f2cdcaefe63408fd0e437e60a3] Merge branch 'develop' into feature-xarray
git bisect bad 7b18f701f45ed4f2cdcaefe63408fd0e437e60a3
# good: [5fa54e7221bae206d147b3c8848320a7a64f4d48] Merge pull request #84 from eysteinn/composite-snowage-fix
git bisect good 5fa54e7221bae206d147b3c8848320a7a64f4d48
# bad: [1055effb83e0d185d310eba40a8a02dae574fd2e] Merge branch 'develop' into feature-xarray
git bisect bad 1055effb83e0d185d310eba40a8a02dae574fd2e
# good: [1616d2cdbbb7e87ec403673e6f48e31403728f70] Update documentation to add hrit_goes
git bisect good 1616d2cdbbb7e87ec403673e6f48e31403728f70
# bad: [08adb2d22f5bf051e37c5b341b7e25a22929015b] Create x and y dimensions for swathdefs also
git bisect bad 08adb2d22f5bf051e37c5b341b7e25a22929015b
# bad: [7b18f701f45ed4f2cdcaefe63408fd0e437e60a3] Merge branch 'develop' into feature-xarray
git bisect bad 7b18f701f45ed4f2cdcaefe63408fd0e437e60a3
# skip: [1c43d6eac566ed01a5a03ecf47fbe596168da5ab] Add xarray support for hrit_msg reader
git bisect skip 1c43d6eac566ed01a5a03ecf47fbe596168da5ab
# skip: [ca3ea366160ff576e80fafc17daf5099e53e9ff8] Make aapp_l1b reader work with xarray
git bisect skip ca3ea366160ff576e80fafc17daf5099e53e9ff8
# skip: [0251bc60e4b16685c9241cde466151d4c4b6db02] Add xarray support for fci_fdhsi reader
git bisect skip 0251bc60e4b16685c9241cde466151d4c4b6db02
# skip: [dffc2b76f240b11c65141266c55a5d9b477b058c] Convert hdfeos_l1b to xarray
git bisect skip dffc2b76f240b11c65141266c55a5d9b477b058c
# skip: [e34cb2d58fbd6d185015a60ad1f5a4a5516b9ef6] Add support for ABI with xarrays
git bisect skip e34cb2d58fbd6d185015a60ad1f5a4a5516b9ef6
# skip: [764062b2ee3c953e01ed796aff1bd8ec0168e363] Add x and y coordinates for datasets that miss them
git bisect skip 764062b2ee3c953e01ed796aff1bd8ec0168e363
# skip: [11928f37f69886f1d71e728ad72cf163c2c2b94b] Merge branch 'develop' into feature-xarray
git bisect skip 11928f37f69886f1d71e728ad72cf163c2c2b94b
# skip: [1bc998288c733926ba3983e7f96c899d9cd7c88d] Fix composites for xarrays
git bisect skip 1bc998288c733926ba3983e7f96c899d9cd7c88d
# skip: [47d7f3497724674a46e94f5062d401ac986d93e1] WIP xarray adaptation
git bisect skip 47d7f3497724674a46e94f5062d401ac986d93e1
# good: [2be73260bb27b6baa97d648f15ba7c16a5987639] Merge pull request #177 from pytroll/bugfix-viirs
git bisect good 2be73260bb27b6baa97d648f15ba7c16a5987639
# skip: [30807fba9c98a3a09571f9feeea4e77ede0bae2c] Do not compute sunz corrections in place
git bisect skip 30807fba9c98a3a09571f9feeea4e77ede0bae2c
# skip: [e0c093a9de5668c237e3cd552a280147668f88d0] Patch CF writer for geos and adding latlons
git bisect skip e0c093a9de5668c237e3cd552a280147668f88d0
# skip: [69cde3e68fec347100dd3319a1bd5201a88e6e2f] Add first attempt to cf's save_dataset
git bisect skip 69cde3e68fec347100dd3319a1bd5201a88e6e2f
# skip: [9bcea29c2b0e41ae1ac20bd2c6d0db8a5d5fc9e8] Fix yaml_reader tests
git bisect skip 9bcea29c2b0e41ae1ac20bd2c6d0db8a5d5fc9e8
# good: [56f5b0067d7475bae52ee85aeb79bc68c9ef5c33] Add GenericCompositor
git bisect good 56f5b0067d7475bae52ee85aeb79bc68c9ef5c33
# skip: [02b4f66202547315ff25401a76ce442ea6e4b75b] More xarray adaptations
git bisect skip 02b4f66202547315ff25401a76ce442ea6e4b75b
# skip: [d12597dff0b8e97542b8dbfbbbb184ff8100af4c] Adapt realistic_colors composite to xarray
git bisect skip d12597dff0b8e97542b8dbfbbbb184ff8100af4c
# skip: [9bcea29c2b0e41ae1ac20bd2c6d0db8a5d5fc9e8] Fix yaml_reader tests
git bisect skip 9bcea29c2b0e41ae1ac20bd2c6d0db8a5d5fc9e8
# skip: [6594749ccd45cd1e94f81648cf44454034d7f035] Try out xarray on abi l1b
git bisect skip 6594749ccd45cd1e94f81648cf44454034d7f035
# skip: [43d207cb16dfd3d444be87fd507fca3bb14d590d] Replace BWCompositor and RGBCompositor with GenericCompositor in YAML
git bisect skip 43d207cb16dfd3d444be87fd507fca3bb14d590d
# skip: [8be84c2d318ee2105ca428bce5b9ac51d0976260] Add depracition warnings to BWCompositor and RGBCompositor
git bisect skip 8be84c2d318ee2105ca428bce5b9ac51d0976260
# good: [d888da6cd36c9220faae9d9e8e1d251280416b48] Add two missing deprecation warning lines
git bisect good d888da6cd36c9220faae9d9e8e1d251280416b48
gerritholl commented 5 years ago

See also #399

ameraner commented 1 year ago

Closed after #2271