search

ecmwf / cfgrib

A Python interface to map GRIB files to the NetCDF Common Data Model following the CF Convention using ecCodes
Apache License 2.0
407 stars 77 forks source link

cfgrib does not recognise channelNumber in ECMWF synthetic satellite products #405

Open jsSvensson opened 2 weeks ago

jsSvensson commented 2 weeks ago

What happened?

When opening a grib2 file from ECMWF model version 49r1 which includes messages with different channels of cloudy reflectance and cloudy brightness temperature from simulated satellite images, cfgrib only identifies one channel from cloudy reflectance and one from cloudy brightness temperature from each time step. The other channels seems to not be read or to be discarded. My guess is that cfgrib is unable to recognize different channels because it does not seem to consider channelNumber as a GRIB-entity.

What are the steps to reproduce the bug?

  1. Download a synthetic satellite data from ECMWF (so far only test data), assumes access to ECMWF MARS service. Here is an exemple of mars request (from https://confluence.ecmwf.int/display/FCST/Simulated+Satellite+Data+update): retrieve, channel=1/2/3/4/7, class=od, date=2024-10-24, expver=79, ident=71, instrument=210, param=260512, step=0/to/144/by/3, stream=oper, time=00:00:00, type=ssd, target="fci_vis.grib"

  2. Open in python with xarray and cfgrib:

ds=xr.open_dataset('fci_vis.grib.grib', engine='cfgrib')

and expand the only variable in the dataset:

ds['cdrfl'].

Only data from one channel is shown (the other channels seems not to be read or discarded). No metadata field like GRIB_channelNumber is shown.

  1. Compare with grib_dump: All channels are shown as separate messages with grib_dump, here is an example of output for a message:

GRIB {

Satellite remote sensing products (formerly Space products) (grib2/tables/33/0.0.table)

discipline = 3; editionNumber = 2;

European Centre for Medium-Range Weather Forecasts (common/c-11.table)

centre = 98; subCentre = 0;

Start of forecast (grib2/tables/33/1.2.table)

significanceOfReferenceTime = 1; dataDate = 20241028; dataTime = 1200;

Operational products (grib2/tables/33/1.3.table)

productionStatusOfProcessedData = 0;

Missing (grib2/tables/33/1.4.table)

typeOfProcessedData = 255;

Satellite Channel Data (grib2/grib2LocalSectionNumber.98.table)

grib2LocalSectionNumber = 24;

Operational archive (mars/class.table)

marsClass = 1;

Simulated satellite data (mars/type.table)

marsType = 87;

Atmospheric model (mars/stream.table)

marsStream = 1025; experimentVersionNumber = 0079; channelNumber = 7; numberOfDataPoints = 90857;

There is no appended list (grib2/tables/33/3.11.table)

interpretationOfNumberOfPoints = 0;

Latitude/longitude (Also called equidistant cylindrical, or Plate Carree) (grib2/tables/33/3.1.table)

gridDefinitionTemplateNumber = 0;

Earth assumed spherical with radius of 6 371 229.0 m (grib2/tables/33/3.2.table)

shapeOfTheEarth = 6; Ni = 377; Nj = 241; iScansNegatively = 0; jScansPositively = 0; jPointsAreConsecutive = 0; alternativeRowScanning = 0; latitudeOfFirstGridPointInDegrees = 72; longitudeOfFirstGridPointInDegrees = 355; latitudeOfLastGridPointInDegrees = 42; longitudeOfLastGridPointInDegrees = 42; iDirectionIncrementInDegrees = 0.125; jDirectionIncrementInDegrees = 0.125; gridType = regular_ll; NV = 0;

Analysis or forecast at a horizontal level or in a horizontal layer at a point in time for simulated (synthetic) (grib2/tables/33/4.0.table)

productDefinitionTemplateNumber = 32;

Quantitative products (grib2/tables/33/4.1.3.table)

parameterCategory = 1;

Cloudy reflectance (%) (grib2/tables/33/4.2.3.1.table)

parameterNumber = 31;

-READ ONLY- parameterUnits = %;

-READ ONLY- parameterName = Cloudy reflectance;

Missing (grib2/tables/33/4.3.table)

typeOfGeneratingProcess = 255; generatingProcessIdentifier = 158;

Hour (grib2/tables/33/4.4.table)

indicatorOfUnitForForecastTime = 1; stepUnits = h; forecastTime = 144; stepRange = 144; NB = 1; satelliteSeries = 334; satelliteNumber = 71; instrumentType = 210; scaleFactorOfCentralWaveNumber = 0; scaledValueOfCentralWaveNumber = 621100; shortName = cdrfl; name = Cloudy reflectance; cfName = unknown;

-READ ONLY- cfVarName = cdrfl;

-READ ONLY- modelName = IFS;

-READ ONLY- modelVersion = unknown;

numberOfValues = 90857; packingType = grid_ccsds; ccsdsFlags = 14; ccsdsBlockSize = 32; ccsdsRsi = 128;

A bit map does not apply to this product (grib2/tables/33/6.0.table)

bitMapIndicator = 255; bitmapPresent = 0; values(90857) = { 0.100552, 0.0930603, 0.0969818, 0.101712, 0.119809, 0.104611, 0.0875672, 0.0662354, 0.0633362, 0.0571869, 0.0493439, 0.0516937, 0.0566834, 0.059018, 0.058606, 0.0576905, 0.0583161, 0.0591553, 0.0635803, 0.0658081, 0.0656098, 0.0579651, 0.0547302, 0.0553253, 0.0536316, 0.0469483, 0.0463684, 0.056546, 0.0682495, 0.0775421, 0.0904663, 0.102597, 0.10835, 0.111401, 0.112759, 0.105389, 0.0929077, 0.0782288, 0.128735, 0.199628, 0.268475, 0.357922, 0.445798, 0.498853, 0.446347, 0.406583, 0.353665, 0.279599, 0.202069, 0.16908, 0.139752, 0.108151, 0.100186, 0.0924805, 0.0869416, 0.0922669, 0.0979737, 0.101407, 0.105954, 0.113889, 0.122388, 0.114697, 0.103009, 0.0978211, 0.092511, 0.086789, 0.0807312, 0.0821198, 0.0784577, 0.0664337, 0.0557984, 0.0846985, 0.104749, 0.111325, 0.101895, 0.126202, 0.150388, 0.150952, 0.151288, 0.154645, 0.166013, 0.17738, 0.183667, 0.193036, 0.206158, 0.221143, 0.245435, 0.270581, 0.29025, 0.306577, 0.319516, 0.322476, 0.329877, 0.321622, 0.293515, 0.265195, 0.244885, 0.228986, 0.213239, 0.224957 ... 90757 more values }

-READ ONLY- maximum = 0.839856;

-READ ONLY- minimum = 0.0300263;

-READ ONLY- average = 0.323615;

-READ ONLY- standardDeviation = 0.181049;

-READ ONLY- skewness = 0.414087;

-READ ONLY- kurtosis = -0.67145;

-READ ONLY- isConstant = 0;

-READ ONLY- numberOfMissing = 0;

-READ ONLY- getNumberOfValues = 90857;

}

Version

0.9.14.1

Platform (OS and architecture)

Ubuntu 20.04.6 LTS

Relevant log output

>>> import xarray as xr
>>> ds=xr.open_dataset('fci_vis.grib.grib', engine='cfgrib')
>>> ds
<xarray.Dataset> Size: 18MB
Dimensions:     (step: 49, latitude: 241, longitude: 377)
Coordinates:
    time        datetime64[ns] 8B ...
  * step        (step) timedelta64[ns] 392B 0 days 00:00:00 ... 6 days 00:00:00
  * latitude    (latitude) float64 2kB 72.0 71.88 71.75 ... 42.25 42.12 42.0
  * longitude   (longitude) float64 3kB -5.0 -4.875 -4.75 ... 41.75 41.88 42.0
    valid_time  (step) datetime64[ns] 392B ...
Data variables:
    cdrfl       (step, latitude, longitude) float32 18MB ...
Attributes:
    GRIB_edition:            2
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2024-11-04T16:15 GRIB to CDM+CF via cfgrib-0.9.1...

>>> ds['cdrfl']
<xarray.DataArray 'cdrfl' (step: 49, latitude: 241, longitude: 377)> Size: 18MB
[4451993 values with dtype=float32]
Coordinates:
    time        datetime64[ns] 8B ...
  * step        (step) timedelta64[ns] 392B 0 days 00:00:00 ... 6 days 00:00:00
  * latitude    (latitude) float64 2kB 72.0 71.88 71.75 ... 42.25 42.12 42.0
  * longitude   (longitude) float64 3kB -5.0 -4.875 -4.75 ... 41.75 41.88 42.0
    valid_time  (step) datetime64[ns] 392B ...
Attributes: (12/29)
    GRIB_paramId:                             260512
    GRIB_dataType:                            ssd
    GRIB_numberOfPoints:                      90857
    GRIB_stepUnits:                           1
    GRIB_stepType:                            instant
    GRIB_gridType:                            regular_ll
    ...                                       ...
    GRIB_name:                                Cloudy reflectance
    GRIB_shortName:                           cdrfl
    GRIB_units:                               %
    long_name:                                Cloudy reflectance
    units:                                    %
    standard_name:                            unknown

ds['cdrfl'].attrs
{'GRIB_paramId': 260512, 'GRIB_dataType': 'ssd', 'GRIB_numberOfPoints': 90857, 'GRIB_stepUnits': 1, 'GRIB_stepType': 'instant', 'GRIB_gridType': 'regular_ll', 'GRIB_uvRelativeToGrid': 0, 'GRIB_NV': 0, 'GRIB_Nx': 377, 'GRIB_Ny': 241, 'GRIB_cfName': 'unknown', 'GRIB_cfVarName': 'cdrfl', 'GRIB_gridDefinitionDescription': 'Latitude/longitude', 'GRIB_iDirectionIncrementInDegrees': 0.125, 'GRIB_iScansNegatively': 0, 'GRIB_jDirectionIncrementInDegrees': 0.125, 'GRIB_jPointsAreConsecutive': 0, 'GRIB_jScansPositively': 0, 'GRIB_latitudeOfFirstGridPointInDegrees': 72.0, 'GRIB_latitudeOfLastGridPointInDegrees': 42.0, 'GRIB_longitudeOfFirstGridPointInDegrees': 355.0, 'GRIB_longitudeOfLastGridPointInDegrees': 42.0, 'GRIB_missingValue': 3.4028234663852886e+38, 'GRIB_name': 'Cloudy reflectance', 'GRIB_shortName': 'cdrfl', 'GRIB_units': '%', 'long_name': 'Cloudy reflectance', 'units': '%', 'standard_name': 'unknown'}

Accompanying data

No response

Organisation

No response