The Trouble with MODIS

[lobsiger]

The trouble In my GEO and LEO scripts I use delayed composite generation setting generate=False for speed. The issue is seen with Aqua and Terra MODIS data from EUMETCast. Using generate=False I get ugly big pink stains in images with composite 'true_color_thin'. No problem with other composites (so far) and with generate=True. I suspect that different possible resolutions in channels produce this bug.

To Reproduce I made a minimum script including data that reproduces the problem generating four images. The wetransfer link is:

https://we.tl/t-cIufngDaCJ

(issue0731.zip, 168MB, the link is active one week from now)

Expected behavior I expect that generate=False makes the same fine 'true_color_thin' image as generate=True (that I have used in the past).

Actual results No obvious difference is seen in normal text output. No special debug output has been generated.

Screenshots Reduced size output images can be found on the google list:

https://groups.google.com/g/pytroll/c/BejWK4jiEc4

Environment Info:

• GNU/Linux
• satpy 0.37.1 pyhd8ed1ab_0 conda-forge
• pyresample 1.25.1 py39h1832856_0 conda-forge

Additional context I'm currently updating my PyTROLL/Satpy Kit of scripts for EUMETCast. Making many composites I want to go the delayed composite generation route as far as possible. Up to now I only found that GEO composites with DAY/NIG background files do not work this way.

[djhoese]

Until @mraspaud can give a better guess, my guess is that this has to do with the FillingCompositor which the "thin" composites use. If this filling operation is performed after resampling then you get all the weirdness of resampling, like fill pixels being used or not used in certain places and differences depending on what resolution is being used (I'm not super familiar with the thin files).

https://github.com/pytroll/satpy/blob/acd074530f1eeaa860c1d18de22ecbfd75be8166/satpy/composites/__init__.py#L469-L478

Regardless, I did a little test using the native resampler before the actual resampling:

scn = Scene(filenames = files, reader = 'modis_l1b')
scn.load(composites, resolution=1000, generate=False)
new_scn1 = scn.resample(resampler='native')
new_scn = new_scn1.resample(area, radius_of_influence=2000)

And it produced the same images. This is pretty obvious that this should work when you think about it since this is still "resampling" going on. If I add generate=False to the native resample call here the pink lines come back.

There might be something else going on here. :thinking:

[lobsiger]

@djhoese one thing I noted is that when imaging MODIS channels '10' or '12', used by 'true_color_thin', I get these coarse stains as 'pixels' in both cases (generate=True or generate=False). If the filling compositor HAS TO be performed after resampling in the case of generate=False then this is rather a feature than a bug? But then a couple of other composites may have this feature as well. IIRC I have seen similar things in Metop NWCSAF cloudtype (not sure, I'll check).

[mraspaud]

Indeed, sounds like a problem with the filling compositor in combination with the resampler. I think we have another keyword called 'mask_data' to the resampler, maybe setting it to true would help?

[djhoese]

It looks like it is mask_area and it doesn't seem to make a difference for me.

[lobsiger]

@mraspaud and @djhoese I made some more experiments including MODIS multiscene also described on the google list.

https://groups.google.com/g/pytroll/c/BejWK4jiEc4

I'm not sure that the filling compositor is the (only) culprit here. I also get weird results if I only image single channel '12'. After coding one day to make my scripts automatically use generate=TRUE for 'true_color_thin' (and maybe other problematic composites) I gave the 'bilinear' resampler a try and BINGO: This piece has no problem with MODIS 'true_color_thin' and generate=FALSE. I updated my script for using different resamplers and automatic naming of the resulting files.

https://we.tl/t-exoyS9kLiM

issue2188.zip, 250MB, link valid one week from now

I wonder whether the problem also has to do with the reader used? Using my full scripts that also produce multipass images I realized, that neither 'bilinear' nor 'gradient_search' could handle these passes for MODIS data. No problems with avhrr though.

[lobsiger]

Testing different resamplers I had these most unexpected results:

1) The 'bilinear' resampler is the only resampler that can do the composite 'true_color_thin' without the ugly pink stains in both settings generate=True and generate=False.

2) Thin MODIS data from Terra behave differently from thin MODIS data from Aqua. The 'bilinear' resampler can resample 'natural_color' from Terra while it cannot do 'natural_color' from Aqua.

See my table with all results on the google list, posted Sept 12 2022:

https://groups.google.com/g/pytroll/c/8Fpo9QszZyc