search

USACE / cumulus

Cumulus project issue tracking and project planning
MIT License
3 stars 2 forks source link

NAEFS QPF dataset download always comes out with a all -inifinit grid for a few grid records #385

Open FHanbali opened 11 months ago

FHanbali commented 11 months ago

Describe the bug

A clear and concise description of what the bug is.

Steps To Reproduce

  1. Tested on SPN's Russian River Watershed
  2. Requested a time window of 01Nov2023 12:00 PST to 02Nov2023 12:00 PST, intending to retrieve the QPF set issued at 02Nov2023 12:00 PST
  3. Downloaded the resulting file
  4. Checked the contents of the file and found the first record only to be all -infinity

Expected behavior

Expected valid grids for the entire dataset

Actual behavior

Again, first records for some reason comes out all -infinity consistently, except in the last test summarized above the third record came out -infinity too

Screenshots

image

image

image

Desktop (please complete the following information):

Additional context

Downloaded file for above example is attached for your reference:

2023.11_NAEFS_DSS7.zip

FHanbali commented 2 months ago

@adamscarberry I verified the recent behavior and it seems Cumulus is always returning a null grid for the very first record of the requested time window.

Here's an example download time window and the all null values QPF record at the very beginning of that time window:

image

NAEFSmean_netcdf2024081800-QPF-202408180000.zip

The issue is only with the QPF dataset (not QTF) and happens with any download time window.

adamscarberry commented 2 months ago

Here is the gdalinfo from the null value tif that Fauwaz posted:

$ gdalinfo NAEFSmean_netcdf2024081800-QPF-202408180000.tif
Driver: GTiff/GeoTIFF
Files: NAEFSmean_netcdf2024081800-QPF-202408180000.tif
Size is 641, 750
Coordinate System is:
PROJCRS["NAD83 / Conus Albers",
    BASEGEOGCRS["NAD83",
        DATUM["North American Datum 1983",
            ELLIPSOID["GRS 1980",6378137,298.257222101,
                LENGTHUNIT["metre",1]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4269]],
    CONVERSION["Conus Albers",
        METHOD["Albers Equal Area",
            ID["EPSG",9822]],
        PARAMETER["Latitude of false origin",23,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8821]],
        PARAMETER["Longitude of false origin",-96,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8822]],
        PARAMETER["Latitude of 1st standard parallel",29.5,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8823]],
        PARAMETER["Latitude of 2nd standard parallel",45.5,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8824]],
        PARAMETER["Easting at false origin",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8826]],
        PARAMETER["Northing at false origin",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8827]]],
    CS[Cartesian,2],
        AXIS["easting (X)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["northing (Y)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Data analysis and small scale data presentation for contiguous lower 48 states."],
        AREA["United States (USA) - CONUS onshore - Alabama; Arizona; Arkansas; California; Colorado; Connecticut; Delaware; Florida; Georgia; Idaho; Illinois; Indiana; Iowa; Kansas; Kentucky; Louisiana; Maine; Maryland; Massachusetts; Michigan; Minnesota; Mississippi; Missouri; Montana; Nebraska; Nevada; New Hampshire; New Jersey; New Mexico; New York; North Carolina; North Dakota; Ohio; Oklahoma; Oregon; Pennsylvania; Rhode Island; South Carolina; South Dakota; Tennessee; Texas; Utah; Vermont; Virginia; Washington; West Virginia; Wisconsin; Wyoming."],
        BBOX[24.41,-124.79,49.38,-66.91]],
    ID["EPSG",5070]]
Data axis to CRS axis mapping: 1,2
Origin = (568000.000000000000000,1760000.000000000000000)
Pixel Size = (2000.000000000000000,-2000.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  568000.000, 1760000.000) ( 89d24'10.08"W, 38d41'27.63"N)
Lower Left  (  568000.000,  260000.000) ( 90d25'25.32"W, 25d13'49.02"N)
Upper Right ( 1850000.000, 1760000.000) ( 74d50' 6.24"W, 37d 1'16.48"N)
Lower Right ( 1850000.000,  260000.000) ( 78d 2' 2.47"W, 23d46'38.83"N)
Center      ( 1209000.000, 1010000.000) ( 83d11'45.20"W, 31d25'52.02"N)
Band 1 Block=641x1 Type=Float64, ColorInterp=Gray
  NoData Value=-9999

This is either a problem with the geoproc processor or the product.

Processor code: https://github.com/USACE/cumulus-geoproc/blob/main/src/cumulus_geoproc/processors/naefs-mean-06h.py

adamscarberry commented 2 months ago

Sample raw files from S3 as saved from LDM.

NAEFSmean_netcdf20240818-19.zip

adamscarberry commented 2 months ago 
$ gdalinfo NAEFSmean_netcdf2024081800.nc
Driver: netCDF/Network Common Data Format
Files: NAEFSmean_netcdf2024081800.nc
Size is 512, 512
Metadata:
  NC_GLOBAL#Conventions=CF-1.6,UGRID-0.9
  NC_GLOBAL#date_created=2024-08-18 09:45:30 GMT
  NC_GLOBAL#fews_build_number=125158
  NC_GLOBAL#fews_implementation_version=2022.02
  NC_GLOBAL#fews_patch_number=132901
  NC_GLOBAL#history=2024-08-18 09:45:30 GMT: exported from Delft-FEWS
  NC_GLOBAL#institution=Deltares
  NC_GLOBAL#Metadata_Conventions=Unidata Dataset Discovery v1.0
  NC_GLOBAL#references=http://www.delft-fews.com
  NC_GLOBAL#source=Export NETCDF-CF_GRID from Delft-FEWS
  NC_GLOBAL#summary=Data exported from Delft-FEWS
  NC_GLOBAL#title=Data
Subdatasets:
  SUBDATASET_1_NAME=NETCDF:"NAEFSmean_netcdf2024081800.nc":z
  SUBDATASET_1_DESC=[61x121] z (64-bit floating-point)
  SUBDATASET_2_NAME=NETCDF:"NAEFSmean_netcdf2024081800.nc":QPF
  SUBDATASET_2_DESC=[65x61x121] QPF (32-bit floating-point)
  SUBDATASET_3_NAME=NETCDF:"NAEFSmean_netcdf2024081800.nc":QTF
  SUBDATASET_3_DESC=[65x61x121] QTF (32-bit floating-point)
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  512.0)
Upper Right (  512.0,    0.0)
Lower Right (  512.0,  512.0)
Center      (  256.0,  256.0)
$ gdal_translate -b 1 NETCDF:"NAEFSmean_netcdf2024081800.nc":QPF output.tif
Input file size is 121, 61
0...10...20...30...40...50...60...70...80...90...100 - done.
$ gdalinfo output.tif
Driver: GTiff/GeoTIFF
Files: output.tif
Size is 121, 61
Coordinate System is:
GEOGCRS["WGS 84",
    ENSEMBLE["World Geodetic System 1984 ensemble",
        MEMBER["World Geodetic System 1984 (Transit)"],
        MEMBER["World Geodetic System 1984 (G730)"],
        MEMBER["World Geodetic System 1984 (G873)"],
        MEMBER["World Geodetic System 1984 (G1150)"],
        MEMBER["World Geodetic System 1984 (G1674)"],
        MEMBER["World Geodetic System 1984 (G1762)"],
        MEMBER["World Geodetic System 1984 (G2139)"],
        MEMBER["World Geodetic System 1984 (G2296)"],
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]],
        ENSEMBLEACCURACY[2.0]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    USAGE[
        SCOPE["Horizontal component of 3D system."],
        AREA["World."],
        BBOX[-90,-180,90,180]],
    ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (-125.250000000000000,54.250000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Metadata:
  crs#crs_wkt=GEOGCS["WGS 84",
DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],
UNIT["degree",0.01745329251994328,AUTHORITY["EPSG","9122"]],
AUTHORITY["EPSG","4326"]]
  crs#epsg_code=EPSG:4326
  crs#grid_mapping_name=latitude_longitude
  crs#inverse_flattening=298.257223563
  crs#longitude_of_prime_meridian=0
  crs#long_name=coordinate reference system
  crs#proj4_params=+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs
  crs#semi_major_axis=6378137
  NC_GLOBAL#Conventions=CF-1.6,UGRID-0.9
  NC_GLOBAL#date_created=2024-08-18 09:45:30 GMT
  NC_GLOBAL#fews_build_number=125158
  NC_GLOBAL#fews_implementation_version=2022.02
  NC_GLOBAL#fews_patch_number=132901
  NC_GLOBAL#history=2024-08-18 09:45:30 GMT: exported from Delft-FEWS
  NC_GLOBAL#institution=Deltares
  NC_GLOBAL#Metadata_Conventions=Unidata Dataset Discovery v1.0
  NC_GLOBAL#references=http://www.delft-fews.com
  NC_GLOBAL#source=Export NETCDF-CF_GRID from Delft-FEWS
  NC_GLOBAL#summary=Data exported from Delft-FEWS
  NC_GLOBAL#title=Data
  NETCDF_DIM_EXTRA={time}
  NETCDF_DIM_time_DEF={65,6}
  NETCDF_DIM_time_VALUES={28732320,28732680,28733040,28733400,28733760,28734120,28734480,28734840,28735200,28735560,28735920,28736280,28736640,28737000,28737360,28737720,28738080,28738440,28738800,28739160,28739520,28739880,28740240,28740600,28740960,28741320,28741680,28742040,28742400,28742760,28743120,28743480,28743840,28744200,28744560,28744920,28745280,28745640,28746000,28746360,28746720,28747080,28747440,28747800,28748160,28748520,28748880,28749240,28749600,28749960,28750320,28750680,28751040,28751400,28751760,28752120,28752480,28752840,28753200,28753560,28753920,28754280,28754640,28755000,28755360}
  QPF#coordinates=analysis_time
  QPF#grid_mapping=crs
  QPF#long_name=QPF
  QPF#units=MM
  QPF#_FillValue=-999
  time#axis=T
  time#long_name=time
  time#standard_name=time
  time#units=minutes since 1970-01-01 00:00:00.0 +0000
  x#axis=X
  x#long_name=x coordinate according to WGS 1984
  x#standard_name=longitude
  x#units=degrees_east
  x#_FillValue=9.969209999999999e+36
  y#axis=Y
  y#long_name=y coordinate according to WGS 1984
  y#standard_name=latitude
  y#units=degrees_north
  y#_FillValue=9.969209999999999e+36
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (-125.2500000,  54.2500000) (125d15' 0.00"W, 54d15' 0.00"N)
Lower Left  (-125.2500000,  23.7500000) (125d15' 0.00"W, 23d45' 0.00"N)
Upper Right ( -64.7500000,  54.2500000) ( 64d45' 0.00"W, 54d15' 0.00"N)
Lower Right ( -64.7500000,  23.7500000) ( 64d45' 0.00"W, 23d45' 0.00"N)
Center      ( -95.0000000,  39.0000000) ( 95d 0' 0.00"W, 39d 0' 0.00"N)
Band 1 Block=121x16 Type=Float32, ColorInterp=Gray
  NoData Value=-999
  Unit Type: MM
  Metadata:
    coordinates=analysis_time
    grid_mapping=crs
    long_name=QPF
    NETCDF_DIM_time=28732320
    NETCDF_VARNAME=QPF
    units=MM
    _FillValue=-999