search

pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.07k stars 298 forks source link

Errorenous `sensor_name` in `composites/visir.yaml` can lead to `RecursionError` #2398

Open gerritholl opened 1 year ago

gerritholl commented 1 year ago

Describe the bug

When adding a composites/visir.yaml to SATPY_CONFIG_PATH and writing sensor_name: visir/seviri in this file, loading compositor configs for seviri fails with RecursionError.

To Reproduce

import os
import tempfile
import pathlib
with tempfile.TemporaryDirectory() as td:
    p = pathlib.Path(td)
    (p / "composites").mkdir(parents=True)
    with (p / "composites" / "visir.yaml").open(mode="wt", encoding="ascii") as fp:
        fp.write("sensor_name: visir/seviri\n")
    os.environ["SATPY_CONFIG_PATH"] = td
    from satpy.composites.config_loader import load_compositor_configs_for_sensor
    load_compositor_configs_for_sensor("visir")

Expected behavior

I don't know what I expect. I don't understand this part of Satpy well enough.

Actual results

Traceback ```python-traceback Traceback (most recent call last): File "/data/gholl/checkouts/protocode/mwe/recursion-error.py", line 11, in load_compositor_configs_for_sensor("visir") File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 226, in _call_without_config_path_wrapper return func(sensor_name) ^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 196, in _load_config dep_comps, dep_mods, dep_id_keys = load_compositor_configs_for_sensor(sensor_dep) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 226, in _call_without_config_path_wrapper return func(sensor_name) ^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 196, in _load_config dep_comps, dep_mods, dep_id_keys = load_compositor_configs_for_sensor(sensor_dep) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 226, in _call_without_config_path_wrapper return func(sensor_name) ^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 196, in _load_config dep_comps, dep_mods, dep_id_keys = load_compositor_configs_for_sensor(sensor_dep) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 226, in _call_without_config_path_wrapper return func(sensor_name) ^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 196, in _load_config dep_comps, dep_mods, dep_id_keys = load_compositor_configs_for_sensor(sensor_dep) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 226, in _call_without_config_path_wrapper return func(sensor_name) ^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 196, in _load_config dep_comps, dep_mods, dep_id_keys = load_compositor_configs_for_sensor(sensor_dep) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ... File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 196, in _load_config dep_comps, dep_mods, dep_id_keys = load_compositor_configs_for_sensor(sensor_dep) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 226, in _call_without_config_path_wrapper return func(sensor_name) ^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 196, in _load_config dep_comps, dep_mods, dep_id_keys = load_compositor_configs_for_sensor(sensor_dep) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 226, in _call_without_config_path_wrapper return func(sensor_name) ^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 196, in _load_config dep_comps, dep_mods, dep_id_keys = load_compositor_configs_for_sensor(sensor_dep) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 226, in _call_without_config_path_wrapper return func(sensor_name) ^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 196, in _load_config dep_comps, dep_mods, dep_id_keys = load_compositor_configs_for_sensor(sensor_dep) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 226, in _call_without_config_path_wrapper return func(sensor_name) ^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 196, in _load_config dep_comps, dep_mods, dep_id_keys = load_compositor_configs_for_sensor(sensor_dep) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 226, in _call_without_config_path_wrapper return func(sensor_name) ^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 196, in _load_config dep_comps, dep_mods, dep_id_keys = load_compositor_configs_for_sensor(sensor_dep) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 226, in _call_without_config_path_wrapper return func(sensor_name) ^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 196, in _load_config dep_comps, dep_mods, dep_id_keys = load_compositor_configs_for_sensor(sensor_dep) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 226, in _call_without_config_path_wrapper return func(sensor_name) ^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 196, in _load_config dep_comps, dep_mods, dep_id_keys = load_compositor_configs_for_sensor(sensor_dep) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 232, in _add_config_path_wrapper return _call_without_config_path_wrapper(sensor_name, config_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 226, in _call_without_config_path_wrapper return func(sensor_name) ^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 278, in load_compositor_configs_for_sensor return _load_config(composite_configs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/checkouts/satpy/satpy/composites/config_loader.py", line 180, in _load_config conf = recursive_dict_update(conf, yaml.load(conf_file, Loader=UnsafeLoader)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/__init__.py", line 81, in load return loader.get_single_data() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/constructor.py", line 49, in get_single_data node = self.get_single_node() ^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 36, in get_single_node document = self.compose_document() ^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 55, in compose_document node = self.compose_node(None, None) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node node = self.compose_mapping_node(anchor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node item_value = self.compose_node(node, item_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node node = self.compose_mapping_node(anchor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node item_value = self.compose_node(node, item_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node node = self.compose_mapping_node(anchor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node item_value = self.compose_node(node, item_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 82, in compose_node node = self.compose_sequence_node(anchor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 111, in compose_sequence_node node.value.append(self.compose_node(node, index)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node node = self.compose_mapping_node(anchor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node item_value = self.compose_node(node, item_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 82, in compose_node node = self.compose_sequence_node(anchor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/composer.py", line 110, in compose_sequence_node while not self.check_event(SequenceEndEvent): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/parser.py", line 98, in check_event self.current_event = self.state() ^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/parser.py", line 474, in parse_flow_sequence_first_entry return self.parse_flow_sequence_entry(first=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/parser.py", line 477, in parse_flow_sequence_entry if not self.check_token(FlowSequenceEndToken): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/scanner.py", line 116, in check_token self.fetch_more_tokens() File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/scanner.py", line 255, in fetch_more_tokens return self.fetch_plain() ^^^^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/scanner.py", line 671, in fetch_plain self.save_possible_simple_key() File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/scanner.py", line 309, in save_possible_simple_key self.index, self.line, self.column, self.get_mark()) ^^^^^^^^^^^^^^^ File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/yaml/reader.py", line 119, in get_mark return Mark(self.name, self.index, self.line, self.column, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RecursionError: maximum recursion depth exceeded while calling a Python object ``` **Environment Info:** satpy main
gerritholl commented 1 year ago

If I try to load_compositor_configs_for_sensor("seviri") instead, it fails similarly. If I pass anytihng else, there is no failure.

NB: td is of type str.

djhoese commented 1 year ago

As mentioned on slack, your file is supposed to be called seviri.yaml because your sensor in the file is visir/seviri. I suppose we could check that this is true or check if we've already loaded this sensor file before, but bottom line is that the sensor in the file is wrong. The builtin Vis/IR YAML for composites uses sensor_name of visir:

https://github.com/pytroll/satpy/blob/16d3184844af697b29355d627b5f7b4c2f34bd94/satpy/etc/composites/visir.yaml#L1