search

pytroll / trollflow2

Next generation Trollflow. Trollflow is for batch-processing satellite data using Satpy
https://trollflow2.readthedocs.org/
GNU General Public License v3.0
10 stars 15 forks source link

Cannot generate geo_color #211

Closed gerritholl closed 2 weeks ago

gerritholl commented 3 months ago

Describe the bug

An attempt to generate geo_color fails with a warning message 

[WARNING: 2024-08-23 12:51:06 : trollflow2.plugins] Skipping geo_color: "No dataset matching 'DataQuery(name='geo_color')' found"

To Reproduce


import hdf5plugin
import sys
from glob import glob
import trollflow2.launcher
from queue import Queue

fci_files = glob("/media/nas/x23352/MTG/FCI/L1c-cases/202309_10-cwg/10/01/08/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20231001*_IDPFI_VAL_20231001*_20231001*_N__C_0054_003?.nc")

conf = "/tmp/test.yaml"

s = """product_list:
  output_dir:
    /tmp/a
  reader:
    - fci_l1c_nc
  areas:
    germ:
      areaname: germ
      products:
        'geo_color':
          productname: 'geo_color'
          formats:
            - writer: geotiff
              fname_pattern: "test.tif"
              fill_value: 0

workers:
  - fun: !!python/name:trollflow2.plugins.create_scene
  - fun: !!python/name:trollflow2.plugins.load_composites
  - fun: !!python/name:trollflow2.plugins.resample
  - fun: !!python/name:trollflow2.plugins.save_datasets
"""

with open(conf, mode="w") as fp:
    fp.write(s)

if __name__ == "__main__":
    with trollflow2.launcher.logging_on(None):
        trollflow2.launcher.process_files(
                fci_files,
                {},
                conf,
                Queue())

Expected behavior

I expect a geocolor image to be produced.

Actual results

No image is produced. Final output lines:

[DEBUG: 2024-08-23 12:53:09 : satpy.modifiers.geometry] Applying sun zen correction
[DEBUG: 2024-08-23 12:53:09 : satpy.modifiers.geometry] Computing sun zenith angles.
[DEBUG: 2024-08-23 12:53:09 : satpy.modifiers.geometry] Applying sun-zenith signal reduction with correction_limit 80.0 deg, strength 1.3, and max_sza 90 deg.
[DEBUG: 2024-08-23 12:53:09 : satpy.composites] Sharpening image with high resolution red band
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Delaying generation of DataID(name='geo_color_night') because of dependency's delayed generation: DataID(name='geo_color_background_with_low_clouds')
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Delayed prerequisite for 'DataID(name='geo_color_night')': 'DataID(name='geo_color_background_with_low_clouds')'
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Delaying generation of DataID(name='geo_color') because of dependency's delayed generation: DataID(name='geo_color_night')
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Delayed prerequisite for 'DataID(name='geo_color')': 'DataID(name='geo_color_night')'
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Delaying generation of DataID(name='geo_color_night') because of dependency's delayed generation: DataID(name='geo_color_background_with_low_clouds')
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Delayed prerequisite for 'DataID(name='geo_color_night')': 'DataID(name='geo_color_background_with_low_clouds')'
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Delaying generation of DataID(name='geo_color_low_clouds') because of incompatible areas
[WARNING: 2024-08-23 12:53:09 : satpy.scene] The following datasets were not created and may require resampling to be generated: DataID(name='geo_color')
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='ir_38', wavelength=WavelengthRange(min=3.4, central=3.8, max=4.2, unit='µm'), resolution=2000, calibration=<2>, modifiers=())
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_04', wavelength=WavelengthRange(min=0.384, central=0.444, max=0.504, unit='µm'), resolution=1000, calibration=<1>, modifiers=())
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_05', wavelength=WavelengthRange(min=0.47, central=0.51, max=0.55, unit='µm'), resolution=1000, calibration=<1>, modifiers=())
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_06', wavelength=WavelengthRange(min=0.59, central=0.64, max=0.69, unit='µm'), resolution=1000, calibration=<1>, modifiers=())
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_08', wavelength=WavelengthRange(min=0.815, central=0.865, max=0.915, unit='µm'), resolution=1000, calibration=<1>, modifiers=())
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_06', wavelength=WavelengthRange(min=0.59, central=0.64, max=0.69, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected',))
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_06', wavelength=WavelengthRange(min=0.59, central=0.64, max=0.69, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_06', wavelength=WavelengthRange(min=0.59, central=0.64, max=0.69, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected', 'rayleigh_corrected', 'sunz_reduced'))
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_05', wavelength=WavelengthRange(min=0.47, central=0.51, max=0.55, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected',))
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_05', wavelength=WavelengthRange(min=0.47, central=0.51, max=0.55, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_05', wavelength=WavelengthRange(min=0.47, central=0.51, max=0.55, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected', 'rayleigh_corrected', 'sunz_reduced'))
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_08', wavelength=WavelengthRange(min=0.815, central=0.865, max=0.915, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected',))
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_08', wavelength=WavelengthRange(min=0.815, central=0.865, max=0.915, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected', 'sunz_reduced'))
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='ndvi_hybrid_green', resolution=1000)
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_04', wavelength=WavelengthRange(min=0.384, central=0.444, max=0.504, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected',))
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_04', wavelength=WavelengthRange(min=0.384, central=0.444, max=0.504, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[DEBUG: 2024-08-23 12:53:09 : satpy.scene] Unloading dataset: DataID(name='vis_04', wavelength=WavelengthRange(min=0.384, central=0.444, max=0.504, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected', 'rayleigh_corrected', 'sunz_reduced'))
[DEBUG: 2024-08-23 12:53:09 : satpy.writers] Reading ['/data/gholl/checkouts/satpy/satpy/etc/writers/geotiff.yaml']
[DEBUG: 2024-08-23 12:53:10 : satpy.writers] Adding enhancement configuration from file: /data/gholl/checkouts/satpy/satpy/etc/enhancements/generic.yaml
[DEBUG: 2024-08-23 12:53:10 : satpy.writers] Adding enhancement configuration from file: /home/gholl/Arbeit/checkouts-perforce/config/enhancements/generic.yaml
[DEBUG: 2024-08-23 12:53:10 : satpy.writers] Adding enhancement configuration from file: /home/gholl/checkouts/pytroll-dwd-config/enhancements/generic.yaml
[WARNING: 2024-08-23 12:53:10 : trollflow2.plugins] Skipping geo_color: "No dataset matching 'DataQuery(name='geo_color')' found"
[DEBUG: 2024-08-23 12:53:10 : trollflow2.launcher] Cleaning up

Environment Info:

Additional context

This is most likely a consequence of https://github.com/pytroll/satpy/issues/2733.

pnuu commented 2 weeks ago

This just came up in Slack, so I'll mention this also here:

The delay_composites: true is the default, so will result in generate=False to be passed to Satpy. Defining delay_composites: false in the Trollflow2 YAML config might resolve this issue. The other option is to pre-resample the background AUX data to the final target area and use these either in the sub-composite definitions.

gerritholl commented 2 weeks ago

Our workaround is the second option (creating different versions of the land-water mask and the night marble image).

pnuu commented 2 weeks ago

I'm closing this, the issue should be fixed in Satpy or use the workarounds presented above.

Satpy issues for reference: