search

pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.06k stars 292 forks source link

DayNightCompositor does not work with eg overview_sun as the day part #584

Closed TAlonglong closed 5 years ago

TAlonglong commented 5 years ago

I define a composite like this as a DayNight composite

  overview_with_night_ir:
    compositor: !!python/name:satpy.composites.DayNightCompositor
    standard_name: overview_with_night_ir
    prerequisites:
      - overview_sun
      - night_fog

This does not work and gives:

[DEBUG: 2019-01-22 14:21:13 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/software/pytroll/etc/satpy/etc'
[DEBUG: 2019-01-22 14:21:13 : satpy.readers] Reading ['/software/pytroll/lib/python3.5/site-packages/satpy/etc/readers/avhrr_l1b_aapp.yaml']
[DEBUG: 2019-01-22 14:21:13 : satpy.readers.yaml_reader] Assigning to avhrr_l1b_aapp: ['/data/pytroll/aapp-outdir-xl/noaa18_20190122_1404_70475/hrpt_noaa18_20190122_1404_70475.l1b']
[DEBUG: 2019-01-22 14:21:13 : satpy.readers.aapp_l1b] Reading time 0:00:00.000174
[DEBUG: 2019-01-22 14:21:13 : satpy.composites] Looking for composites config file avhrr-3.yaml
[DEBUG: 2019-01-22 14:21:13 : satpy.composites] Looking for composites config file visir.yaml
[DEBUG: 2019-01-22 14:21:13 : satpy.readers.yaml_reader] No coordinates found for DatasetID(name='longitude', wavelength=None, resolution=1050, polarization=None, calibration=None, level=None, modifiers=())
/software/pytroll/lib/python3.5/site-packages/geotiepoints/geointerpolator.py:101: RuntimeWarning: invalid value encountered in arccos
  90 - rad2deg(arccos(z__/EARTH_RADIUS)),
[DEBUG: 2019-01-22 14:21:16 : satpy.readers.aapp_l1b] Navigation time 0:00:02.222800
[DEBUG: 2019-01-22 14:21:16 : satpy.readers.yaml_reader] No coordinates found for DatasetID(name='latitude', wavelength=None, resolution=1050, polarization=None, calibration=None, level=None, modifiers=())
[INFO: 2019-01-22 14:21:16 : satpy.readers.aapp_l1b] No valid operational coefficients, fall back to pre-launch
[DEBUG: 2019-01-22 14:21:16 : satpy.readers.aapp_l1b] Calibration time 0:00:00.321944
[INFO: 2019-01-22 14:21:16 : satpy.readers.aapp_l1b] No valid operational coefficients, fall back to pre-launch
[DEBUG: 2019-01-22 14:21:16 : satpy.readers.aapp_l1b] Calibration time 0:00:00.324235
/software/pytroll/lib/python3.5/site-packages/satpy/readers/aapp_l1b.py:580: RuntimeWarning: invalid value encountered in log
  np.log(1 + ir_const_1 * cwnum * cwnum * cwnum / rad)
[DEBUG: 2019-01-22 14:21:17 : satpy.readers.aapp_l1b] Calibration time 0:00:00.456133
[DEBUG: 2019-01-22 14:21:17 : satpy.readers.aapp_l1b] Calibration time 0:00:00.466944
[DEBUG: 2019-01-22 14:21:18 : satpy.readers.aapp_l1b] Calibration time 0:00:00.505131
[DEBUG: 2019-01-22 14:21:19 : satpy.readers.aapp_l1b] Interpolate sun-sat angles: time 0:00:01.198460
[DEBUG: 2019-01-22 14:21:19 : satpy.composites] Applying sun zen correction
[DEBUG: 2019-01-22 14:21:19 : satpy.composites] Computing sun zenith angles.
[DEBUG: 2019-01-22 14:21:19 : satpy.composites] Apply the standard sun-zenith correction [1/cos(sunz)]
[DEBUG: 2019-01-22 14:21:19 : satpy.composites] Sun-zenith correction applied. Computation time:   0.0 (sec)
[DEBUG: 2019-01-22 14:21:19 : satpy.composites] Applying sun zen correction
[DEBUG: 2019-01-22 14:21:19 : satpy.composites] Computing sun zenith angles.
[DEBUG: 2019-01-22 14:21:19 : satpy.composites] Apply the standard sun-zenith correction [1/cos(sunz)]
[DEBUG: 2019-01-22 14:21:19 : satpy.composites] Sun-zenith correction applied. Computation time:   0.0 (sec)
[DEBUG: 2019-01-22 14:21:19 : satpy.composites] Computing sun zenith angles.
[DEBUG: 2019-01-22 14:21:19 : satpy.writers] Enhancement configuration options: [{'name': 'inverse', 'args': [[False, False, True]], 'method': <function invert at 0x7f09a236e6a8>}, {'kwargs': {'stretch': 'linear'}, 'name': 'stretch', 'method': <function stretch at 0x7f09a236e730>}, {'kwargs': {'gamma': 1.6}, 'name': 'gamma', 'method': <function gamma at 0x7f09a236e510>}]
[DEBUG: 2019-01-22 14:21:19 : trollimage.xrimage] Applying invert with parameters [False, False, True]
[DEBUG: 2019-01-22 14:21:19 : trollimage.xrimage] Applying stretch linear with parameters {}
[DEBUG: 2019-01-22 14:21:19 : trollimage.xrimage] Perform a linear contrast stretch.
[DEBUG: 2019-01-22 14:21:19 : trollimage.xrimage] Calculate the histogram quantiles: 
[DEBUG: 2019-01-22 14:21:19 : trollimage.xrimage] Left and right quantiles: 0.005 0.005
[DEBUG: 2019-01-22 14:21:19 : trollimage.xrimage] Applying gamma 1.6
[DEBUG: 2019-01-22 14:21:20 : satpy.writers] Enhancement configuration options: [{'kwargs': {'max_stretch': [2, 6, 293], 'stretch': 'crude', 'min_stretch': [-4, 0, 243]}, 'name': 'stretch', 'method': <function stretch at 0x7f09a236e730>}, {'kwargs': {'gamma': [1, 2, 1]}, 'name': 'gamma', 'method': <function gamma at 0x7f09a236e510>}]
[DEBUG: 2019-01-22 14:21:20 : trollimage.xrimage] Applying stretch crude with parameters {'max_stretch': [2, 6, 293], 'min_stretch': [-4, 0, 243]}
[DEBUG: 2019-01-22 14:21:20 : trollimage.xrimage] Applying gamma <xarray.DataArray (bands: 3)>
array([1, 2, 1])
Coordinates:
  * bands    (bands) <U1 'R' 'G' 'B'
Traceback (most recent call last):
  File "quicklook-avhrr.py", line 165, in <module>
    ql.load(['overview_with_night_ir']);
  File "/software/pytroll/lib/python3.5/site-packages/satpy/scene.py", line 886, in load
    keepables = self.generate_composites()
  File "/software/pytroll/lib/python3.5/site-packages/satpy/scene.py", line 801, in generate_composites
    return self._read_composites(nodes)
  File "/software/pytroll/lib/python3.5/site-packages/satpy/scene.py", line 775, in _read_composites
    self._generate_composite(item, keepables)
  File "/software/pytroll/lib/python3.5/site-packages/satpy/scene.py", line 750, in _generate_composite
    **self.attrs)
  File "/software/pytroll/lib/python3.5/site-packages/satpy/composites/__init__.py", line 935, in __call__
    night_data = zero_missing_data(night_data, day_data)
  File "/software/pytroll/lib/python3.5/site-packages/satpy/composites/__init__.py", line 990, in zero_missing_data
    return data1.where(~nans, 0)
  File "/software/pytroll/lib/python3.5/site-packages/xarray/core/common.py", line 799, in where
    return ops.where_method(self, cond, other)
  File "/software/pytroll/lib/python3.5/site-packages/xarray/core/ops.py", line 182, in where_method
    keep_attrs=True)
  File "/software/pytroll/lib/python3.5/site-packages/xarray/core/computation.py", line 987, in apply_ufunc
    exclude_dims=exclude_dims)
  File "/software/pytroll/lib/python3.5/site-packages/xarray/core/computation.py", line 205, in apply_dataarray_ufunc
    raise_on_invalid=False)
  File "/software/pytroll/lib/python3.5/site-packages/xarray/core/alignment.py", line 217, in deep_align
    exclude=exclude)
  File "/software/pytroll/lib/python3.5/site-packages/xarray/core/alignment.py", line 132, in align
    .format(dim))
ValueError: indexes along dimension 'y' are not equal

If I use overview instead of overview_sun it works

python3 satpy 0.11.1 with pip on ubuntu xenial

TAlonglong commented 5 years ago

Oh maybe @pnuu needs to see this?

pnuu commented 5 years ago

Confirmed. The indexing seems to change when the SZA correction is applied, and the indexes are propagated to the nans variable in function that masks then NaNs out before merging the two composites:

Without SZA correction:

(Pdb) data1.attrs['name']
'night_fog'
(Pdb) data2.attrs['name']
'overview_sun'
(Pdb) data1.indexes
bands: Index(['R', 'G', 'B'], dtype='object', name='bands')
(Pdb) data2.indexes
bands: Index(['R', 'G', 'B'], dtype='object', name='bands')
(Pdb) nans.indexes
bands: Index(['R', 'G', 'B'], dtype='object', name='bands')

With SZA correction:

(Pdb) data1.indexes
bands: Index(['R', 'G', 'B'], dtype='object', name='bands')
(Pdb) data2.indexes
bands: Index(['R', 'G', 'B'], dtype='object', name='bands')
y: RangeIndex(start=0, stop=5173, step=1, name='y')
x: RangeIndex(start=0, stop=2048, step=1, name='x')
(Pdb) nans.indexes
bands: Index(['R', 'G', 'B'], dtype='object', name='bands')
y: RangeIndex(start=0, stop=5173, step=1, name='y')
x: RangeIndex(start=0, stop=2048, step=1, name='x')

In both cases the shapes are equal.