Closed tommyjasmin closed 4 hours ago
djhoese
commented
4 days ago @gerritholl @ameraner Is native resampling possible with FCI? I'm not up to date on what's expected for the FCI reader(s). Do the high res bands include a stacked area definition? Does the full disk image need to be filled for missing segments?
djhoese
commented
4 days ago @tommyjasmin While we wait for help from the people who should know, could you print out the .attrs["area"] of the two variables you're loading in the Scene before resampling:
for data_arr in global_scene.values():
print(data_arr.attrs["area"])
tommyjasmin
commented
4 days ago @djhoese -
for data_arr in global_scene.values(): print(data_arr.attrs["area"])
Sure, and this is for two channels that will fail:
channel_list = ['vis_06', 'ir_105']
>>> for data_arr in global_scene.values():
... print(data_arr.attrs["area"])
...
Area ID: mtg_fci_fdss_500m
Description: MTG FCI Full Disk Scanning Service area definition with 500 m resolution
Projection: {'ellps': 'MERIT', 'h': '35786400', 'lon_0': '0', 'no_defs': 'None', 'proj': 'geos', 'type': 'crs', 'units': 'm', 'x_0': '0', 'y_0': '0'}
Number of columns: 22272
Number of rows: 22272
Area extent: (-5567499.9996, 5568499.9996, 5568499.9996, -5567499.9996)
Area ID: mtg_fci_fdss_1km
Description: MTG FCI Full Disk Scanning Service area definition with 1 km resolution
Projection: {'ellps': 'MERIT', 'h': '35786400', 'lon_0': '0', 'no_defs': 'None', 'proj': 'geos', 'type': 'crs', 'units': 'm', 'x_0': '0', 'y_0': '0'}
Number of columns: 11136
Number of rows: 11136
Area extent: (-5566999.9986, 5568999.9986, 5568999.9986, -5566999.9986)Also, if we want to drag @jon4than in, he noted:
I also spotted a bunch of these types of things in satpy:
fci_l1c_af_ir_87:
file_reader: !!python/name:satpy.readers.fci_l1c_nc.FCIL1cNCFileHandler
file_patterns:
- "{pflag}_{location_indicator},{data_designator},MTI{spacecraft_id:1d}-{data_source}-1C-RRAD-3KM-{coverage}-IR87-{component1}-x-{component3}-{purpose}-{format}_{oflag}_{originator}_{processing_time:%Y%m%d%H%M%S}_{facility_or_tool}_{environment}_{start_time:%Y%m%d%H%M%S}_{end_time:%Y%m%d%H%M%S}_{processing_mode}_{special_compression}_{disposition_mode}_{repeat_cycle_in_day:>04d}_{count_in_repeat_cycle:>04d}.nc"
- "{pflag}_{location_indicator},{data_designator},MTI{spacecraft_id:1d}+{data_source}-1C-RRAD-3KM-{coverage}-IR87-{component1}-x-{component3}-{purpose}-{format}_{oflag}_{originator}_{processing_time:%Y%m%d%H%M%S}_{facility_or_tool}_{environment}_{start_time:%Y%m%d%H%M%S}_{end_time:%Y%m%d%H%M%S}_{processing_mode}_{special_compression}_{disposition_mode}_{repeat_cycle_in_day:>04d}_{count_in_repeat_cycle:>04d}.nc"
expected_segments: 1
From https://github.com/pytroll/satpy/blob/main/satpy/etc/readers/fci_l1c_nc.yaml#L203-L208
[1:55](https://ssec-dev.slack.com/archives/C076W4N6NP3/p1732132509798209)
it's that 1C-RRAD-3KM and IR87 rather than just 1C-RRAD-FDHSI that makes me wonder if it's related to that channel subsetting snippet
[1:55](https://ssec-dev.slack.com/archives/C076W4N6NP3/p1732132554215819)
Though I could find no mention of files with that naming convention in the PUG
pnuu
commented
3 days ago The 3KM files are for African (the af in the dataset name) distribution. They have been downsampled for lower network and processing requirements.
ameraner
commented
3 days ago Hi @tommyjasmin, I just tried to reproduce this but I couldn't, it works for me
from glob import glob
from satpy import Scene
import hdf5plugin
from satpy.utils import debug_on; debug_on()
filenames = sorted(glob('/tcenas/fbf/mtg/RollingBuffer/VAL/idpfi/FCI-L1/20241121/*HRFI*_JLS_C_0073*'))
# filenames = filenames[0:39]
scn = Scene(filenames=filenames, reader='fci_l1c_nc')
scn.load(['vis_06', 'ir_105'])
scn_r = scn.resample(scn.coarsest_area(), resampler='native')I also tried to exclude some chunk to activate the padding but it doesn't cause any issue either... I have the latest satpy and pyresample from main, and
platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
python: 3.11.8
cartopy: 0.23.0
dask: 2024.7.1
fsspec: 2024.3.1
gdal: 3.8.4
geoviews: 1.12.0
h5netcdf: 1.3.0
h5py: 3.10.0
netcdf4: 1.6.5
numpy: 1.26.4
pyhdf: 0.11.3
pyproj: 3.6.1
rasterio: 1.3.9
xarray: 2024.7.0The native resampling should work in principle, since the resolutions are always in double or half steps from each other (apart from the African format, but that's a corner case that shouldn't bother us here). @djhoese the reader, per default, returns an AreaDefinition from the appended StackedAreaDefinitions, and padding is activated by default for missing chunks, before the StackedAreaDefinition merging.
I just noticed that you are testing this with some old data from January. I don't recall now exactly but it could be that the grids have slightly changed in the meanwhile and maybe that upsets the code. Could you check with some newer data just to be sure? Another check you could do is to use 'mtg_fci_fdss_1km' instead of scn.coarsest_area().
tommyjasmin
commented
3 days ago Thanks so much @ameraner - I'll try both suggestions and report back...
tommyjasmin
commented
4 hours ago Heh @ameraner - Woo-Hoo! I tested with new, near-realtime data and everything works great! Thanks also @gerritholl and @djhoese
Closing the ticket.
Description of the bug Hi all -
We found that for the MTG FCI high resolution bands, which for reference are a visible and near-ir band at 0.5 km, and two IR bands at 1.0 km, a scene resample of any bands at the same resolution works, but any combination of bands at different spatial resolution fails, with an error shown below. It's a lot of output, but as requested, I used debug_on(), and only used a single timestep:
To Reproduce
NOTE: I had to cut out some of the debug output, since the GitHub issue tracker complains with
There was an error creating your Issue: body is too long (maximum is 65536 characters).Expected behavior Resampling should work regardless of the set of bands chosen. Only when you choose bands of the same spatial resolution does resampling work. (Note; full files lists in output below are omitted since the form is getting sluggish with the full debug output included...)
Actual results
Included above.
Screenshots N/A
Environment Info:
Additional context Any questions please ask.
We suspect the problem is either here or here