search

pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.08k stars 298 forks source link

Let the radiances be negative across all readers #1911

Open mraspaud opened 2 years ago

mraspaud commented 2 years ago

Feature Request

At the moment, different readers have different behaviours when it comes to clipping or not negative radiances. As this has been pointed out by @simonrp84, it can be desirable to keep the negative radiances for eg sensor noise estimation. Even though different space agency probably recommend different behaviours, it is important that satpy stays consistent in this I think.

This issue is raised to have a place to discuss different aspects of this standardization.

Things to take into consideration:

  1. Should there be clipping before calibration to Reflectances and Brightness Temperatures? as @gerritholl points out, negative radiances generate negative reflectances, which could be accepted, and negative radiances generated invalid (NaN) brightness temperatures. Both could be interesting for the users.
  2. The removal of the clipping in the readers implies that the compositing should take care of that instead. This is not difficult, but should not be missed.
  3. As a more general point, this shows that scientific users might want to have readers alter the data as little as possible before calibration or corrections are applied, and we need to make the API more flexible to allow them have more control over the different steps involved.
  4. Documentation for users and guidelines for developers should be written/updated to make sure everyone understands what the standard in satpy is, and red flags should be raised for readers not following the standard.

There are certainly more to think about here, please do comment.

djhoese commented 2 years ago

Is there anything mathematically that requires that composites handle this? I think typically this is handled "automatically" by enhancements. The one compositor I can think of that these negatives may play a part is my "RatioSharpeningCompositor" stuff where I take the ratio of two bands (ex. red / low_res_red) and then multiply the other bands by that ratio. I've updated this compositor in the past to make sure the ratio is never negative so removing the clipping from the readers shouldn't be a problem there.

gerritholl commented 2 years ago

Mathematically no, but if one band out of three is masked (NaN brightness temperatures) the impact on the colour is much larger than when it's clipped to the lower end of a range.

pnuu commented 2 years ago

Mathematically no, but if one band out of three is masked (NaN brightness temperatures) the impact on the colour is much larger than when it's clipped to the lower end of a range.

Only if linear stretch is used, but yeah, some users (and built-in composites) do that.

djhoese commented 2 years ago

The GenericCompositor will "share" the mask between any input bands so you make a very good point @gerritholl. If a negative radiance produces a NaN temperature and is then included in a RGB then it would effect all pixels of the RGB. Maybe it should be up to the IR calibration in a reader to clip the radiances to 0? Hhhmmm but that's still a "esthetic" change rather than a "scientific" requirement.

mraspaud commented 2 years ago

log-based enhancements won't like this (eg cira-stretch, reinhardt).

djhoese commented 2 years ago

@mraspaud They already have to deal with negatives for other readers. And the enhancements are the component that should be dealing with this.

mraspaud commented 2 years ago

ok. We just have to check that we don't break anything when removing the clipping.

sfinkens commented 2 years ago

Good point, thanks for raising this! Would clipping remain the default behaviour? Your third point reads like a proposal for generic calibration algorithms in Satpy that users can choose from. That would be great! :slightly_smiling_face:

simonrp84 commented 2 years ago

If radiances can be clipped by the enhancements / composites then I'd strongly push for the default to be no clipping: There's a big risk that research users won't know about this behaviour and will just use the default.

sfinkens commented 2 years ago

That makes sense, they'll probably notice the negative values and if they don't want them they can enable clipping.

djhoese commented 2 years ago

I personally feel like there should be no option for clipping in the readers unless it is scientifically useful/meaningful. Otherwise it is more of an "esthetic" change and should be handled in the RGB composites or enhancements.

simonrp84 commented 2 years ago

I had a quick search through the code and it seems that the following readers do some manner of clipping:

aaap_l1b (line 548):

agri_l1 (line 63):

ahi_hsd:

goes_imager_hrit (line 439):

goes_imager_nc (lines 851, 896, 923):

sar_c_safe (line 599):

seviri_base (line 490):

viirs_compact (line 280):

mraspaud commented 2 years ago

Thanks for the investigation work @simonrp84 !

sfinkens commented 2 years ago

Just catched up with the conversation on slack where @simonrp84 mentioned that negative radiances can't be used for anything physical. So I think it would make sense to keep the clipping option. This would be convenient for scientific users who are processing the data in their algorithms.

ameraner commented 10 months ago

See also https://github.com/pytroll/satpy/pull/2499 where optional radiance clipping through a config key was implemented by @ghiggi (currently only for abi_l1b)