search

Unidata / netcdf4-python

netcdf4-python: python/numpy interface to the netCDF C library
http://unidata.github.io/netcdf4-python
MIT License
740 stars 261 forks source link

netCDF4 unable to open netCDF file that opens as HDF and with Panoply #1313

Open danielfromearth opened 4 months ago

danielfromearth commented 4 months ago

The version of the software with which you are encountering an issue

h5netcdf version: 1.2.0 libnetcdf version: 4.9.2 netCDF4 version: 1.6.5 xarray version: 2024.2.0

Environmental information (i.e. Operating System, compiler info, java version, python version, etc.)

OS: MacOS 14.3.1 Chip: Apple M3 Pro Python version: 3.11.8

A description of the issue with the steps needed to reproduce it

The following error is raised when attempting to open the file (linked below) via the netCDF4 python library:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[10], line 1
----> 1 nc.Dataset(filepath)

File src[/netCDF4/_netCDF4.pyx:2495](http://localhost:8888/netCDF4/_netCDF4.pyx#line=2494), in netCDF4._netCDF4.Dataset.__init__()

File src[/netCDF4/_netCDF4.pyx:1891](http://localhost:8888/netCDF4/_netCDF4.pyx#line=1890), in netCDF4._netCDF4._get_grps()

File src[/netCDF4/_netCDF4.pyx:3616](http://localhost:8888/netCDF4/_netCDF4.pyx#line=3615), in netCDF4._netCDF4.Group.__init__()

File src[/netCDF4/_netCDF4.pyx:2001](http://localhost:8888/netCDF4/_netCDF4.pyx#line=2000), in netCDF4._netCDF4._get_vars()

AttributeError: 'NoneType' object has no attribute 'dimensions'

Here is where the file can be downloaded. It does require a free NASA Earthdata login account to access, but the file size even when compressed (51 MB) exceeds the GitHub issue limit of 25 MB so I couldn't upload it here.

Interestingly, as an HDF5, the file appears OK: The whole file can be opened without error by the underlying library libhdf5, as well as h5dump, h5netcdf, and the myHDF5 tool. However, the irradiance and irradiance_error variables — which exist in multiple groups, e.g., /frames/band_290_490_nm/ — do raise errors when the HDFView application attempts to open them.

Also, interestingly, the NASA Panoply data viewer opens the file without issue as well.

DennisHeimbigner commented 4 months ago

Are you using libnetcdf? If so, which version.

danielfromearth commented 4 months ago

I'm in a conda environment, which has libnetcdf version 4.9.2 installed.

jswhit2 commented 4 months ago

is ncdump able to read this file?

dopplershift commented 4 months ago

I can reproduce this problem (on my Mac M2) in Python 3.12:

# Name                    Version                   Build  Channel
libnetcdf                 4.9.2           nompi_h291a7c2_113    conda-forge
netcdf4                   1.6.5           nompi_py312h9035142_100    conda-forge

While ncdump -h works without error, this was interesting:

❯ ncdump TEMPO_IRR_L1_V02_20240314T041808Z.nc > /dev/null
NetCDF: Start+count exceeds dimension bound
Location: file ?; fcn ? line 478
dopplershift commented 4 months ago

ncdump -sh output if anyone is curious:

netcdf TEMPO_IRR_L1_V02_20240314T041808Z {
dimensions:
    mirror_step = 1 ;
variables:
    int mirror_step(mirror_step) ;
        mirror_step:long_name = "scan mirror position index" ;
        mirror_step:_Storage = "contiguous" ;
        mirror_step:_Endianness = "little" ;
    double time(mirror_step) ;
        time:standard_name = "time" ;
        time:long_name = "exposure start time" ;
        time:calendar = "gregorian" ;
        time:units = "seconds since 1980-01-06T00:00:00Z" ;
        time:comment = "Average of /frames/time" ;
        time:_Storage = "contiguous" ;
        time:_Endianness = "little" ;
    float exposure_time(mirror_step) ;
        exposure_time:units = "seconds" ;
        exposure_time:long_name = "exposure duration" ;
        exposure_time:valid_min = 0.f ;
        exposure_time:valid_max = 10.f ;
        exposure_time:_FillValue = 9.96921e+36f ;
        exposure_time:comment = "Average of /frames/exposure_time" ;
        exposure_time:_Storage = "contiguous" ;
        exposure_time:_Endianness = "little" ;
    float solar_phi(mirror_step) ;
        solar_phi:units = "degrees" ;
        solar_phi:long_name = "solar boresight azimuthal angle" ;
        solar_phi:valid_min = -180.f ;
        solar_phi:valid_max = 180.f ;
        solar_phi:_FillValue = 9.96921e+36f ;
        solar_phi:comment = "Average of /frames/solar_phi" ;
        solar_phi:_Storage = "contiguous" ;
        solar_phi:_Endianness = "little" ;
    float solar_theta(mirror_step) ;
        solar_theta:units = "degrees" ;
        solar_theta:long_name = "solar boresight polar angle" ;
        solar_theta:valid_min = 0.f ;
        solar_theta:valid_max = 180.f ;
        solar_theta:_FillValue = 9.96921e+36f ;
        solar_theta:comment = "Average of /frames/solar_theta" ;
        solar_theta:_Storage = "contiguous" ;
        solar_theta:_Endianness = "little" ;
    float earth_sun_distance ;
        earth_sun_distance:units = "m" ;
        earth_sun_distance:long_name = "Earth-sun distance" ;
        earth_sun_distance:_Storage = "contiguous" ;
        earth_sun_distance:_Endianness = "little" ;

// global attributes:
        :time_reference = "1980-01-06T00:00:00Z" ;
        :format_version = 2 ;
        :product_type = "IRR" ;
        :processing_level = "1" ;
        :sdpc_version = "TEMPO_SDPC_v4.3.0" ;
        :processing_version = 2 ;
        :production_date_time = "2024-03-14T04:39:37Z" ;
        :local_granule_id = "TEMPO_IRR_L1_V02_20240314T041808Z.nc" ;
        :version_id = 2 ;
        :pge_version = "0.1.0" ;
        :shortname = "TEMPO_IRR_L1" ;
        :begin_date = "2024-03-14" ;
        :begin_time = "04:18:08" ;
        :end_date = "2024-03-14" ;
        :end_time = "04:18:11" ;
        :time_coverage_start = "2024-03-14T04:18:08Z" ;
        :time_coverage_end = "2024-03-14T04:18:11Z" ;
        :time_coverage_start_since_epoch = 1394425106.5535 ;
        :time_coverage_end_since_epoch = 1394425109.21705 ;
        :history = "2024-03-14T04:39:44Z: /tempo/nas0/sdpc_soft/install/gnu-rh8/sdpc/v4p3/bin/L0_ccd -vv --Version 2 --config l0_ccd.cfg -i @hk.lis --trend trend_params.nc -d /tempo/nas0/sdpc_archive/temposdpc/ops3/L1/DRK/D16138/20240314T041121Z/TEMPO_DRK_L1_V02_20240314T041121Z.nc -o TEMPO_IRR_L1_V02_20240314T041808Z.nc TEMPO_IRR_L0_V01_20240314T041808Z.nc\n2024-03-14T04:57:25Z: /tempo/nas0/sdpc_soft/install/gnu-rh8/sdpc/v4p3/bin/wavecal_merge --meta --delete -t TEMPO_IRR_L1_V02_20240314T041808Z.nc wavecal_TEMPO_IRR_L1_V02_20240314T041808Z\n" ;
        string :input_files = "TEMPO_IRR_L0_V01_20240314T041808Z.nc", "TEMPO_HK_L0_V01_20240314T040946Z.nc", "TEMPO_HK_L0_V01_20240314T041410Z.nc", "TEMPO_HK_L0_V01_20240314T041840Z.nc", "TEMPO_HK_L0_V01_20240314T042310Z.nc", "TEMPO_DRK_L1_V02_20240314T041121Z.nc" ;
        :Conventions = "CF-1.6, ACDD-1.3" ;
        :day_of_year = 74LL ;
        :project = "TEMPO" ;
        :platform = "Intelsat 40e" ;
        :source = "UV-VIS hyperspectral imaging" ;
        :institution = "Smithsonian Astrophysical Observatory" ;
        :creator_url = "http://tempo.si.edu" ;
        :title = "TEMPO Level 1 irradiance measurement" ;
        :collection_shortname = "TEMPO_IRR_L1" ;
        :collection_version = 1LL ;
        :keywords = "EARTH SCIENCE>ATMOSPHERE>ATMOSPHERIC RADIATION>SOLAR IRRADIANCE" ;
        :summary = "Level 1 irradiance files provide solar irradiance measured using the working solar diffuser. Each file includes the measured solar irradiance for all the North-South cross-track pixels. The files are provided in netCDF4 format, and contain information on radiometrically and wavelength calibrated solar irradiance for the UV and visible bands, corresponding noise, parameterized wavelength grid, solar viewing geometry, quality flags and other ancillary information. The product is produced using the L0-1b processor which includes multiple steps: (1) Image processing to produce radiometrically calibrated radiance, and (2) Additional wavelength calibration to improve wavelength registration. Please refer to the ATBD for details." ;
        :coremetadata = "\nGROUP                  = INVENTORYMETADATA\n  GROUPTYPE            = MASTERGROUP\n\n  GROUP                  = ECSDATAGRANULE\n\n    OBJECT                 = LOCALGRANULEID\n      NUM_VAL              = 1\n      VALUE                = \"TEMPO_IRR_L1_V02_20240314T041808Z.nc\"\n    END_OBJECT             = LOCALGRANULEID\n\n    OBJECT                 = LOCALVERSIONID\n      NUM_VAL              = 1\n      VALUE                = (\"RFC1321 MD5 = not yet calculated\")\n    END_OBJECT             = LOCALVERSIONID\n\n    OBJECT                 = PRODUCTIONDATETIME\n      NUM_VAL              = 1\n      VALUE                = \"2024-03-14T04:39:37Z\"\n    END_OBJECT             = PRODUCTIONDATETIME\n\n  END_GROUP              = ECSDATAGRANULE\n\n  GROUP                  = COLLECTIONDESCRIPTIONCLASS\n\n    OBJECT                 = SHORTNAME\n      NUM_VAL              = 1\n      VALUE                = \"TEMPO_IRR_L1\"\n    END_OBJECT             = SHORTNAME\n\n    OBJECT                 = VERSIONID\n      NUM_VAL              = 1\n      VALUE                = 2\n    END_OBJECT             = VERSIONID\n\n  END_GROUP              = COLLECTIONDESCRIPTIONCLASS\n\n  GROUP                  = INPUTGRANULE\n\n    OBJECT                 = INPUTPOINTER\n      NUM_VAL              = 6\n      VALUE                = (\"TEMPO_IRR_L0_V01_20240314T041808Z.nc\", \"TEMPO_HK_L0_V01_20240314T040946Z.nc\", \"TEMPO_HK_L0_V01_20240314T041410Z.nc\", \"TEMPO_HK_L0_V01_20240314T041840Z.nc\", \"TEMPO_HK_L0_V01_20240314T042310Z.nc\", \"TEMPO_DRK_L1_V02_20240314T041121Z.nc\")\n    END_OBJECT             = INPUTPOINTER\n\n  END_GROUP              = INPUTGRANULE\n\n  GROUP = SPATIALDOMAINCONTAINER\n    OBJECT = GRANULESPATIALREPRESENTATION\n      CLASS = \"1\"\n      NUM_VAL = 1\n      VALUE = \"NO_SPATIAL\"\n    END_OBJECT = GRANULESPATIALREPRESENTATION\n  END_GROUP = SPATIALDOMAINCONTAINER\n\n  GROUP                  = RANGEDATETIME\n\n    OBJECT                 = RANGEENDINGDATE\n      NUM_VAL              = 1\n      VALUE                = \"2024-03-14\"\n    END_OBJECT             = RANGEENDINGDATE\n\n    OBJECT                 = RANGEENDINGTIME\n      NUM_VAL              = 1\n      VALUE                = \"04:18:11\"\n    END_OBJECT             = RANGEENDINGTIME\n\n    OBJECT                 = RANGEBEGINNINGDATE\n      NUM_VAL              = 1\n      VALUE                = \"2024-03-14\"\n    END_OBJECT             = RANGEBEGINNINGDATE\n\n    OBJECT                 = RANGEBEGINNINGTIME\n      NUM_VAL              = 1\n      VALUE                = \"04:18:08\"\n    END_OBJECT             = RANGEBEGINNINGTIME\n\n  END_GROUP              = RANGEDATETIME\n\n  GROUP                  = PGEVERSIONCLASS\n\n    OBJECT                 = PGEVERSION\n      NUM_VAL              = 1\n      VALUE                = \"0.1.0\"\n    END_OBJECT             = PGEVERSION\n\n  END_GROUP              = PGEVERSIONCLASS\n\nEND_GROUP              = INVENTORYMETADATA\n\nEND\n" ;
        :_NCProperties = "version=1|netcdflibversion=4.6.1|hdf5libversion=1.8.14" ;
        :_SuperblockVersion = 0 ;
        :_IsNetcdf4 = 1 ;
        :_Format = "netCDF-4" ;

group: band_290_490_nm {
  dimensions:
    xtrack = 2048 ;
    spectral_channel = 1028 ;
    xtrack = 2048 ;
  variables:
    int xtrack(xtrack) ;
        xtrack:_Storage = "contiguous" ;
        xtrack:_Endianness = "little" ;
    float irradiance(mirror_step, xtrack, spectral_channel) ;
        irradiance:units = "photons/s/cm^2/nm" ;
        irradiance:ancillary_variables = "irradiance_error" ;
        irradiance:_FillValue = -9.96921e+36f ;
        irradiance:comment = "Average of /frames/band_290_490_nm/irradiance" ;
        irradiance:_Storage = "chunked" ;
        irradiance:_ChunkSizes = 1, 128, 1028 ;
        irradiance:_Shuffle = "true" ;
        irradiance:_DeflateLevel = 1 ;
        irradiance:_Endianness = "little" ;
    float irradiance_error(mirror_step, xtrack, spectral_channel) ;
        irradiance_error:units = "photons/s/cm^2/nm" ;
        irradiance_error:long_name = "irradiance error" ;
        irradiance_error:_FillValue = -9.96921e+36f ;
        irradiance_error:number_of_significant_digits = 2 ;
        irradiance_error:comment = "Root mean square of /frames/band_290_490_nm/irradiance_error" ;
        irradiance_error:_Storage = "chunked" ;
        irradiance_error:_ChunkSizes = 1, 128, 1028 ;
        irradiance_error:_Shuffle = "true" ;
        irradiance_error:_DeflateLevel = 1 ;
        irradiance_error:_Endianness = "little" ;
    float nominal_wavelength(xtrack, spectral_channel) ;
        nominal_wavelength:units = "nm" ;
        nominal_wavelength:valid_min = -9.96921e+36f ;
        nominal_wavelength:valid_max = 9.96921e+36f ;
        nominal_wavelength:_FillValue = 9.96921e+36f ;
        nominal_wavelength:_Storage = "contiguous" ;
        nominal_wavelength:_Endianness = "little" ;
    ushort pixel_quality_flag(mirror_step, xtrack, spectral_channel) ;
        pixel_quality_flag:long_name = "pixel quality flag" ;
        pixel_quality_flag:_FillValue = 65535US ;
        pixel_quality_flag:flag_masks = 1US, 2US, 4US, 8US, 16US, 32US, 64US, 128US, 256US, 512US, 1024US, 2048US ;
        pixel_quality_flag:flag_meanings = "missing_data bad_pixel processing_error transient_pixel rts_pixel saturated noise_underflow dark_corr_error offset_corr_error smear_corr_error straylight_corr_error nonlinear_range" ;
        pixel_quality_flag:comment = "Logical OR of /frames/band_290_490_nm/pixel_quality_flag" ;
        pixel_quality_flag:_Storage = "chunked" ;
        pixel_quality_flag:_ChunkSizes = 1, 128, 1028 ;
        pixel_quality_flag:_Shuffle = "true" ;
        pixel_quality_flag:_DeflateLevel = 1 ;
        pixel_quality_flag:_Endianness = "little" ;
    float wavecal_params(mirror_step, xtrack, /xtrack) ;
        wavecal_params:num_coefficients = 2 ;
        wavecal_params:start_spectral_channel = 0 ;
        wavecal_params:num_spectral_channels = 1028 ;
        wavecal_params:_Storage = "contiguous" ;
        wavecal_params:_Endianness = "little" ;
    float sf_hw1e(mirror_step, xtrack, spectral_channel) ;
        sf_hw1e:_Storage = "contiguous" ;
        sf_hw1e:_Endianness = "little" ;
    float sf_shape(mirror_step, xtrack, spectral_channel) ;
        sf_shape:_Storage = "contiguous" ;
        sf_shape:_Endianness = "little" ;
    float sf_asym(mirror_step, xtrack, spectral_channel) ;
        sf_asym:_Storage = "contiguous" ;
        sf_asym:_Endianness = "little" ;
    int wavecal_niter(mirror_step, xtrack) ;
        wavecal_niter:_Storage = "contiguous" ;
        wavecal_niter:_Endianness = "little" ;
    int wavecal_opt_status(mirror_step, xtrack) ;
        wavecal_opt_status:_Storage = "contiguous" ;
        wavecal_opt_status:_Endianness = "little" ;

  // group attributes:
  } // group band_290_490_nm

group: band_540_740_nm {
  dimensions:
    xtrack = 2048 ;
    spectral_channel = 1028 ;
    mirror_step = 1 ;
  variables:
    int xtrack(xtrack) ;
        xtrack:_Storage = "contiguous" ;
        xtrack:_Endianness = "little" ;
    float irradiance(mirror_step, xtrack, spectral_channel) ;
        irradiance:units = "photons/s/cm^2/nm" ;
        irradiance:ancillary_variables = "irradiance_error" ;
        irradiance:_FillValue = -9.96921e+36f ;
        irradiance:comment = "Average of /frames/band_540_740_nm/irradiance" ;
        irradiance:_Storage = "chunked" ;
        irradiance:_ChunkSizes = 1, 128, 1028 ;
        irradiance:_Shuffle = "true" ;
        irradiance:_DeflateLevel = 1 ;
        irradiance:_Endianness = "little" ;
    float irradiance_error(mirror_step, xtrack, spectral_channel) ;
        irradiance_error:units = "photons/s/cm^2/nm" ;
        irradiance_error:long_name = "irradiance error" ;
        irradiance_error:_FillValue = -9.96921e+36f ;
        irradiance_error:number_of_significant_digits = 2 ;
        irradiance_error:comment = "Root mean square of /frames/band_540_740_nm/irradiance_error" ;
        irradiance_error:_Storage = "chunked" ;
        irradiance_error:_ChunkSizes = 1, 128, 1028 ;
        irradiance_error:_Shuffle = "true" ;
        irradiance_error:_DeflateLevel = 1 ;
        irradiance_error:_Endianness = "little" ;
    float nominal_wavelength(xtrack, spectral_channel) ;
        nominal_wavelength:units = "nm" ;
        nominal_wavelength:valid_min = -9.96921e+36f ;
        nominal_wavelength:valid_max = 9.96921e+36f ;
        nominal_wavelength:_FillValue = 9.96921e+36f ;
        nominal_wavelength:_Storage = "contiguous" ;
        nominal_wavelength:_Endianness = "little" ;
    ushort pixel_quality_flag(mirror_step, xtrack, spectral_channel) ;
        pixel_quality_flag:long_name = "pixel quality flag" ;
        pixel_quality_flag:_FillValue = 65535US ;
        pixel_quality_flag:flag_masks = 1US, 2US, 4US, 8US, 16US, 32US, 64US, 128US, 256US, 512US, 1024US, 2048US ;
        pixel_quality_flag:flag_meanings = "missing_data bad_pixel processing_error transient_pixel rts_pixel saturated noise_underflow dark_corr_error offset_corr_error smear_corr_error straylight_corr_error nonlinear_range" ;
        pixel_quality_flag:comment = "Logical OR of /frames/band_540_740_nm/pixel_quality_flag" ;
        pixel_quality_flag:_Storage = "chunked" ;
        pixel_quality_flag:_ChunkSizes = 1, 128, 1028 ;
        pixel_quality_flag:_Shuffle = "true" ;
        pixel_quality_flag:_DeflateLevel = 1 ;
        pixel_quality_flag:_Endianness = "little" ;
    float wavecal_params(mirror_step, xtrack, /mirror_step) ;
        wavecal_params:num_coefficients = 3 ;
        wavecal_params:start_spectral_channel = 0 ;
        wavecal_params:num_spectral_channels = 1028 ;
        wavecal_params:_Storage = "contiguous" ;
        wavecal_params:_Endianness = "little" ;
    float sf_hw1e(mirror_step, xtrack, spectral_channel) ;
        sf_hw1e:_Storage = "contiguous" ;
        sf_hw1e:_Endianness = "little" ;
    float sf_shape(mirror_step, xtrack, spectral_channel) ;
        sf_shape:_Storage = "contiguous" ;
        sf_shape:_Endianness = "little" ;
    float sf_asym(mirror_step, xtrack, spectral_channel) ;
        sf_asym:_Storage = "contiguous" ;
        sf_asym:_Endianness = "little" ;
    int wavecal_niter(mirror_step, xtrack) ;
        wavecal_niter:_Storage = "contiguous" ;
        wavecal_niter:_Endianness = "little" ;
    int wavecal_opt_status(mirror_step, xtrack) ;
        wavecal_opt_status:_Storage = "contiguous" ;
        wavecal_opt_status:_Endianness = "little" ;

  // group attributes:
  } // group band_540_740_nm

group: frames {
  dimensions:
    mirror_step = 1 ;
  variables:
    int mirror_step(mirror_step) ;
        mirror_step:long_name = "scan mirror position index" ;
        mirror_step:_Storage = "contiguous" ;
        mirror_step:_Endianness = "little" ;
    double time(mirror_step) ;
        time:standard_name = "time" ;
        time:long_name = "exposure start time" ;
        time:calendar = "gregorian" ;
        time:units = "seconds since 1980-01-06T00:00:00Z" ;
        time:_Storage = "contiguous" ;
        time:_Endianness = "little" ;
    float exposure_time(mirror_step) ;
        exposure_time:units = "seconds" ;
        exposure_time:long_name = "exposure duration" ;
        exposure_time:valid_min = 0.f ;
        exposure_time:valid_max = 10.f ;
        exposure_time:_FillValue = 9.96921e+36f ;
        exposure_time:_Storage = "contiguous" ;
        exposure_time:_Endianness = "little" ;
    float solar_phi(mirror_step) ;
        solar_phi:units = "degrees" ;
        solar_phi:long_name = "solar boresight azimuthal angle" ;
        solar_phi:valid_min = -180.f ;
        solar_phi:valid_max = 180.f ;
        solar_phi:_FillValue = 9.96921e+36f ;
        solar_phi:_Storage = "contiguous" ;
        solar_phi:_Endianness = "little" ;
    float solar_theta(mirror_step) ;
        solar_theta:units = "degrees" ;
        solar_theta:long_name = "solar boresight polar angle" ;
        solar_theta:valid_min = 0.f ;
        solar_theta:valid_max = 180.f ;
        solar_theta:_FillValue = 9.96921e+36f ;
        solar_theta:_Storage = "contiguous" ;
        solar_theta:_Endianness = "little" ;
    float earth_sun_distance ;
        earth_sun_distance:units = "m" ;
        earth_sun_distance:long_name = "Earth-sun distance" ;
        earth_sun_distance:_Storage = "contiguous" ;
        earth_sun_distance:_Endianness = "little" ;

  // group attributes:
        :time_reference = "1980-01-06T00:00:00Z" ;
        :Conventions = "CF-1.6, ACDD-1.3" ;
        :format_version = 2 ;
        :product_type = "IRR" ;
        :processing_level = "1" ;
        :sdpc_version = "TEMPO_SDPC_v4.3.0" ;
        :time_coverage_start = "2024-03-14T04:18:08Z" ;
        :time_coverage_end = "2024-03-14T04:18:11Z" ;
        :processing_version = 2 ;
        :time_coverage_start_since_epoch = 1394425106.5535 ;
        :time_coverage_end_since_epoch = 1394425109.21705 ;

  group: band_290_490_nm {
    dimensions:
        xtrack = 2048 ;
        spectral_channel = 1028 ;
    variables:
        int xtrack(xtrack) ;
            xtrack:_Storage = "contiguous" ;
            xtrack:_Endianness = "little" ;
        float irradiance(mirror_step, xtrack, spectral_channel) ;
            irradiance:units = "photons/s/cm^2/nm" ;
            irradiance:ancillary_variables = "irradiance_error" ;
            irradiance:_FillValue = -9.96921e+36f ;
            irradiance:_Storage = "chunked" ;
            irradiance:_ChunkSizes = 1, 128, 1028 ;
            irradiance:_Shuffle = "true" ;
            irradiance:_DeflateLevel = 1 ;
            irradiance:_Endianness = "little" ;
        float irradiance_error(mirror_step, xtrack, spectral_channel) ;
            irradiance_error:units = "photons/s/cm^2/nm" ;
            irradiance_error:long_name = "irradiance error" ;
            irradiance_error:_FillValue = -9.96921e+36f ;
            irradiance_error:number_of_significant_digits = 2 ;
            irradiance_error:_Storage = "chunked" ;
            irradiance_error:_ChunkSizes = 1, 128, 1028 ;
            irradiance_error:_Shuffle = "true" ;
            irradiance_error:_DeflateLevel = 1 ;
            irradiance_error:_Endianness = "little" ;
        float nominal_wavelength(xtrack, spectral_channel) ;
            nominal_wavelength:units = "nm" ;
            nominal_wavelength:valid_min = -9.96921e+36f ;
            nominal_wavelength:valid_max = 9.96921e+36f ;
            nominal_wavelength:_FillValue = 9.96921e+36f ;
            nominal_wavelength:_Storage = "contiguous" ;
            nominal_wavelength:_Endianness = "little" ;
        ushort pixel_quality_flag(mirror_step, xtrack, spectral_channel) ;
            pixel_quality_flag:long_name = "pixel quality flag" ;
            pixel_quality_flag:_FillValue = 65535US ;
            pixel_quality_flag:flag_masks = 1US, 2US, 4US, 8US, 16US, 32US, 64US, 128US, 256US, 512US, 1024US, 2048US ;
            pixel_quality_flag:flag_meanings = "missing_data bad_pixel processing_error transient_pixel rts_pixel saturated noise_underflow dark_corr_error offset_corr_error smear_corr_error straylight_corr_error nonlinear_range" ;
            pixel_quality_flag:_Storage = "chunked" ;
            pixel_quality_flag:_ChunkSizes = 1, 128, 1028 ;
            pixel_quality_flag:_Shuffle = "true" ;
            pixel_quality_flag:_DeflateLevel = 1 ;
            pixel_quality_flag:_Endianness = "little" ;

    // group attributes:
    } // group band_290_490_nm

  group: band_540_740_nm {
    dimensions:
        xtrack = 2048 ;
        spectral_channel = 1028 ;
    variables:
        int xtrack(xtrack) ;
            xtrack:_Storage = "contiguous" ;
            xtrack:_Endianness = "little" ;
        float irradiance(mirror_step, xtrack, spectral_channel) ;
            irradiance:units = "photons/s/cm^2/nm" ;
            irradiance:ancillary_variables = "irradiance_error" ;
            irradiance:_FillValue = -9.96921e+36f ;
            irradiance:_Storage = "chunked" ;
            irradiance:_ChunkSizes = 1, 128, 1028 ;
            irradiance:_Shuffle = "true" ;
            irradiance:_DeflateLevel = 1 ;
            irradiance:_Endianness = "little" ;
        float irradiance_error(mirror_step, xtrack, spectral_channel) ;
            irradiance_error:units = "photons/s/cm^2/nm" ;
            irradiance_error:long_name = "irradiance error" ;
            irradiance_error:_FillValue = -9.96921e+36f ;
            irradiance_error:number_of_significant_digits = 2 ;
            irradiance_error:_Storage = "chunked" ;
            irradiance_error:_ChunkSizes = 1, 128, 1028 ;
            irradiance_error:_Shuffle = "true" ;
            irradiance_error:_DeflateLevel = 1 ;
            irradiance_error:_Endianness = "little" ;
        float nominal_wavelength(xtrack, spectral_channel) ;
            nominal_wavelength:units = "nm" ;
            nominal_wavelength:valid_min = -9.96921e+36f ;
            nominal_wavelength:valid_max = 9.96921e+36f ;
            nominal_wavelength:_FillValue = 9.96921e+36f ;
            nominal_wavelength:_Storage = "contiguous" ;
            nominal_wavelength:_Endianness = "little" ;
        ushort pixel_quality_flag(mirror_step, xtrack, spectral_channel) ;
            pixel_quality_flag:long_name = "pixel quality flag" ;
            pixel_quality_flag:_FillValue = 65535US ;
            pixel_quality_flag:flag_masks = 1US, 2US, 4US, 8US, 16US, 32US, 64US, 128US, 256US, 512US, 1024US, 2048US ;
            pixel_quality_flag:flag_meanings = "missing_data bad_pixel processing_error transient_pixel rts_pixel saturated noise_underflow dark_corr_error offset_corr_error smear_corr_error straylight_corr_error nonlinear_range" ;
            pixel_quality_flag:_Storage = "chunked" ;
            pixel_quality_flag:_ChunkSizes = 1, 128, 1028 ;
            pixel_quality_flag:_Shuffle = "true" ;
            pixel_quality_flag:_DeflateLevel = 1 ;
            pixel_quality_flag:_Endianness = "little" ;

    // group attributes:
    } // group band_540_740_nm
  } // group frames
}
danielfromearth commented 4 months ago

Thanks @dopplershift!

Hi @ajelenak, does the error message that @dopplershift encountered in his comment above —

NetCDF: Start+count exceeds dimension bound
Location: file ?; fcn ? line 478

— signify anything further to you about what might be causing the netCDF opening issue?

Sounds to me like the dimension information was perhaps recorded wrong somehow? Though I have no idea right now what next to check.

ajelenak commented 4 months ago

Wild guess because dimension bounds are mentioned... The spectral_channel netCDF dimensions in the file are HDF5 datasets without any data stored but declared as 1D arrays of certain size. This is valid HDF5, and valid for netCDF dimensions as far as I understand. Reading data from these HDF5 datasets will give all fill values (zeroes in this case).

I tried this file with a h5dump --enable-error-stack=2 ... command to check at the HDF5 level and there were no errors printed.

danielfromearth commented 4 months ago

Thanks @ajelenak.

The Location: file ?; fcn ? line 478 message is cryptic. I wonder if anyone here knows what that refers to — perhaps @DennisHeimbigner — e.g., line 478 in which file? And is there any way to determine which variable or dimension might be driving that error, so we can see whether it is the spectral_channel?

dopplershift commented 4 months ago

cc @WardF

WardF commented 4 months ago

Thanks, I'll take a look. This is not the first time something like this (with this error message) has come up.

DennisHeimbigner commented 4 months ago

After some checking, the error report appears to be coming from the function print_rows in the file ncdump/vardata.c line 478.

WardF commented 3 weeks ago

I am still poking at this to get specifics, thanks for your patience. I was able to use netCDF v4.6.1 to dump the binary file to .cdl, and then use ncgen to re-generate the netCDF binary file; this new, regenerated file does not produce the same error.