Netcdf/HDF5 errors when doing format conversion to netCDF4 with nccopy

[neumannd]

Environment Information

• What platform are you using? (please provide specific distribution/version in summary)
  • [x] Linux
  • [ ] Windows
  • [ ] OSX
  • [ ] Other
  • [ ] NA
• 32 and/or 64 bit?
  • [ ] 32-bit
  • [x] 64-bit
• What build system are you using?
  • [x] autotools (configure)
  • [ ] cmake
• Can you provide a sample netCDF file or C code to recreate the issue?
  • [x] Yes (please attach to this issue, thank you!)
  • [ ] No
  • [ ] Not at this time
• netCDF and HDF5 Versions for which the issue arises ([x]) and not arises ([ ]):
  • [ ] netCDF v4.3.3.1, HDF5 1.8.14
  • [ ] netCDF v4.4.0, HDF5 1.8.16
  • [x] netCDF v4.4.1.1, HDF5 1.8.18
  • [x] netCDF v4.5.0, HDF5 1.8.19
  • [x] netCDF v4.4.1.1, HDF5 1.10.0

Summary of Issue

Converting a netCDF 64bit offset file to netCDF-4 with nccopy causes a NetCDF or a HDF5 error depending on the target format (nc4 and nc7, respectively). The errors arise only if netCDF v4.4.1.1 with HDF5 1.8.18 an higher are used. At least netCDF v4.4.0 with HDF5 1.8.16 and lower does not yield the error. I am not sure whether HDF5 (looks a bit like it) or netCDF causes the error.

The errors are:

• conversion to netCDF-4: NetCDF: String match to name in use
• conversion to netCDF-4 classic: NetCDF: Can't open HDF5 attribute

It might be related to HDF5 v1.8 vs v1.10. But I don't think so. I did not create the netCDF file originally but got it as it is (and reduced its size to upload it). cdo tells me that it was created with netCDF v4.4.1 and HDF5 v1.10.0.

• I thought that with HDF5 >=1.10 created files can generally not opened by HDF5 <1.10 if netCDF-4/HDF5-compression was used. However, we have a "netCDF 64bit offset" file and no netCDF-4 file. Therefore, I would have expected no error.
• Using netCDF 4.4.1.1 with HDF5 1.10.0 yields the same error.

Steps to reproduce the behavior

The file is attached as testB.nc.txt. Please rename it. I was not allowed to upload a *.nc file.

> nccopy -k nc4 -d 3 testB.nc testC.nc
NetCDF: String match to name in use
Location: file nccopy.c; line 788

> nccopy -k nc7 -d 3 testB.nc testD.nc
NetCDF: Can't open HDF5 attribute
Location: file nccopy.c; line 788

The ncdump output of the file is:

netcdf testB {
dimensions:
        xt_ocean = 10 ;
        yt_ocean = 10 ;
        st_ocean = 134 ;
        time = UNLIMITED ; // (6 currently)
        bnds = 2 ;
variables:
        float geolon_t(yt_ocean, xt_ocean) ;
                geolon_t:standard_name = "longitude" ;
                geolon_t:long_name = "longitude" ;
                geolon_t:units = "degrees_east" ;
                geolon_t:_CoordinateAxisType = "Lon" ;
        float geolat_t(yt_ocean, xt_ocean) ;
                geolat_t:standard_name = "latitude" ;
                geolat_t:long_name = "latitude" ;
                geolat_t:units = "degrees_north" ;
                geolat_t:_CoordinateAxisType = "Lat" ;
        float xt_ocean(xt_ocean) ;
                xt_ocean:standard_name = "projection_x_coordinate" ;
                xt_ocean:long_name = "tcell longitude" ;
                xt_ocean:units = "degrees_E" ;
                xt_ocean:axis = "X" ;
        float yt_ocean(yt_ocean) ;
                yt_ocean:standard_name = "projection_y_coordinate" ;
                yt_ocean:long_name = "tcell latitude" ;
                yt_ocean:units = "degrees_N" ;
                yt_ocean:axis = "Y" ;
        double st_ocean(st_ocean) ;
                st_ocean:long_name = "tcell zstar depth" ;
                st_ocean:units = "meters" ;
                st_ocean:positive = "down" ;
                st_ocean:axis = "Z" ;
                st_ocean:cartesian_axis = "Z" ;
                st_ocean:edges = "st_edges_ocean" ;
        double time(time) ;
                time:standard_name = "time" ;
                time:long_name = "time" ;
                time:bounds = "time_bnds" ;
                time:units = "days since 1961-08-01 00:00:00" ;
                time:calendar = "standard" ;
                time:axis = "T" ;
        double time_bnds(time, bnds) ;
        float t_detp_5(time, st_ocean, yt_ocean, xt_ocean) ;
                t_detp_5:long_name = "phosphate in detritus fractions_5" ;
                t_detp_5:units = "mol/kg" ;
                t_detp_5:coordinates = "geolat_t geolon_t" ;
                t_detp_5:_FillValue = -1.e+20f ;
                t_detp_5:missing_value = -1.e+20f ;
                t_detp_5:cell_methods = "time: mean" ;
                t_detp_5:time_avg_info = "average_T1,average_T2,average_DT" ;

// global attributes:
                :CDI = "Climate Data Interface version 1.8.2 (http://mpimet.mpg.de/cdi)" ;
                :Conventions = "CF-1.6" ;
                :history = "Wed Jan 24 15:24:30 2018: cdo selindexbox,10,19,10,19 test.nc testB.nc\n",
                        "Wed Jan 24 15:10:40 2018: cdo selname,t_detp_5 ocean_day3d.nc /gfs2/work/mvkdneum/MOM5/testing/test.nc" ;
                :filename = "RUN.1975.05.20.00/ocean_day3d.nc" ;
                :title = "MOM4 Baltic Sea 1-3 n.m." ;
                :grid_type = "regular" ;
                :grid_tile = "N/A" ;
                :CDO = "Climate Data Operators version 1.8.2 (http://mpimet.mpg.de/cdo)" ;

[WardF]

64-bit offset files are created via the core library; HDF5 was not used to create it. They adhere to the classic model. Let me investigate and see what I can find out.

[DennisHeimbigner]

There is an error in nccopy at nccopy.c line 748. The conditional test there does not take all the possibilities into acct. If the input file is anything but netcdf classic, then the following nc_inq_var_filter will fail. So it needs to be modified to take into acct netcdf-64bit-offset and cdf5.

[WardF]

Going through back catalog of bugs; the issue here seems to be that it is trying to copy the global _NCProperties global attribute, which errors out with a 'name in use' error.

[WardF]

Ok, that wasn't the complete issue. I'm able to address this by modifying the history global attribute in the source file (ncdump > text > text editor > ncgen -o newfile.nc [edited text.cdl]) to remove the multiple strings.

Specifically,

:history = "Wed Jan 24 15:24:30 2018: cdo selindexbox,10,19,10,19 test.nc testB.nc\n", "Wed Jan 24 15:10:40 2018: cdo selname,t_detp_5 ocean_day3d.nc /gfs2/work/mvkdneum/MOM5/testing/test.nc" ;

becomes

:history = "Wed Jan 24 15:24:30 2018: cdo selindexbox,10,19,10,19 test.nc testB.nc\nWed Jan 24 15:10:40 2018: cdo selname,t_detp_5 ocean_day3d.nc /gfs2/work/mvkdneum/MOM5/testing/test.nc" ;

Why this issue started manifesting with more recent versions of netCDF and why that manifests as a collision of the _NCProperties attribute is TBD.

[WardF]

Ok. This ball of yarn is getting slowly unravelled. Should have a fix soon.

[AnushaM01]

Hi, Hi, I'm unable to convert this .h5 file to netcdf format using ncks command. I'm getting the following error on Ubuntu 18.04 Assertion 'var->varid == grp->vars.nelems' failed Aborted (core dumped)

Can someone help me with this issue.

Please rename the file by removing .txt Sample.h5.txt

[WardF]

@AnushaM01 I will take a look, although there is not a guarantee that you can convert every single type of .h5 file to a compatible netCDF files. Additionally, ncks is not a Unidata product. @czender, I will pursue this from the Unidata/libnetcdf side, but if you have a moment, does anything leap out at you here from the ncks side?

[czender]

I am copying this response from the NCO Help forum where I answered the same question here from (apparently) the same person this weekend:

This works for me. My guess is that your netCDF or NCO installation, or both, are out-of-date and need to be upgraded.

zender@spectral:~/Downloads$ ncks Sample.h5 Sample.nc
zender@spectral:~/Downloads$ ncks -m Sample.nc
netcdf Sample {
  group: G1_Product\ Identification {
  } // group /G1_Product\ Identification
  group: G2_PreProcessing {
    group: G21_Decluttering {
      group: G211_Type3\ Decluttering {
        group: G2111_Clutter\ Details-00 {
          dimensions:
            phony_dim_0 = 1 ;
            phony_dim_1 = 2 ;

          variables:
            ushort p011_GridCellPairs(phony_dim_0,phony_dim_1) ;

            float p012_ClutterHeight(phony_dim_0) ;

            float p013_SubtitutionHeight(phony_dim_0) ;

            float p014_ReplacementZVal(phony_dim_0) ;
        } // group /G2_PreProcessing/G21_Decluttering/G211_Type3\ Decluttering/G2111_Clutter\ Details-00
        group: G2111_Clutter\ Details-01 {
          dimensions:
            phony_dim_2 = 30935 ;
            phony_dim_3 = 2 ;

          variables:
            ushort p011_GridCellPairs(phony_dim_2,phony_dim_3) ;

            float p012_ClutterHeight(phony_dim_2) ;

            float p013_SubtitutionHeight(phony_dim_2) ;

            float p014_ReplacementZVal(phony_dim_2) ;
        } // group /G2_PreProcessing/G21_Decluttering/G211_Type3\ Decluttering/G2111_Clutter\ Details-01
      } // group /G2_PreProcessing/G21_Decluttering/G211_Type3\ Decluttering
    } // group /G2_PreProcessing/G21_Decluttering
    group: G22_Velocity\ Dealiasing {
    } // group /G2_PreProcessing/G22_Velocity\ Dealiasing
    group: G23_Rain\ Attenuation {
    } // group /G2_PreProcessing/G23_Rain\ Attenuation
    group: G24_Bright\ Band\ Correction {
    } // group /G2_PreProcessing/G24_Bright\ Band\ Correction
    group: G25_Preprocessing\ Filters {
      group: G251_Spike\ Filter {
      } // group /G2_PreProcessing/G25_Preprocessing\ Filters/G251_Spike\ Filter
      group: G252_Outlier\ Filter {
      } // group /G2_PreProcessing/G25_Preprocessing\ Filters/G252_Outlier\ Filter
      group: G253_Speckle\ Filter {
      } // group /G2_PreProcessing/G25_Preprocessing\ Filters/G253_Speckle\ Filter
      group: G254_Filters\ Applied\ Flag {
      } // group /G2_PreProcessing/G25_Preprocessing\ Filters/G254_Filters\ Applied\ Flag
    } // group /G2_PreProcessing/G25_Preprocessing\ Filters
    group: G26_Beam\ Blockage\ Correction {
    } // group /G2_PreProcessing/G26_Beam\ Blockage\ Correction
  } // group /G2_PreProcessing
  group: G3_Base\ Product\ Limits {
  } // group /G3_Base\ Product\ Limits
  group: G4_User_Inputs {
  } // group /G4_User_Inputs
  group: G5_Product\ Numeric\ Data {
    group: G51_Polar\ Array {
      dimensions:
        phony_dim_4 = 360 ;
        phony_dim_5 = 250 ;

      variables:
        float p001_Azimuth(phony_dim_4) ;

        float p002_SlantRange(phony_dim_5) ;

        float p003_Reflectivity(phony_dim_4,phony_dim_5) ;
    } // group /G5_Product\ Numeric\ Data/G51_Polar\ Array
    group: G52_Pixel\ Array {
      dimensions:
        phony_dim_6 = 677 ;
        phony_dim_7 = 677 ;

      variables:
        ushort CartesianBinaryArray(phony_dim_6,phony_dim_7) ;
    } // group /G5_Product\ Numeric\ Data/G52_Pixel\ Array
    group: G53_Annotation\ Parameters {
    } // group /G5_Product\ Numeric\ Data/G53_Annotation\ Parameters
  } // group /G5_Product\ Numeric\ Data
  group: G6_Plot\ Parameters {
    group: G61_Plot\ Canvas\ Layout {
    } // group /G6_Plot\ Parameters/G61_Plot\ Canvas\ Layout
    group: G62_Overlay\ Data {
      group: G621_MapAndPlaces {
        dimensions:
          phony_dim_8 = 6 ;

        variables:
          string p005_LocationName(phony_dim_8) ;
      } // group /G6_Plot\ Parameters/G62_Overlay\ Data/G621_MapAndPlaces
      group: G622_Grid\ Lines {
      } // group /G6_Plot\ Parameters/G62_Overlay\ Data/G622_Grid\ Lines
      group: G623_Font\ Parameters {
      } // group /G6_Plot\ Parameters/G62_Overlay\ Data/G623_Font\ Parameters
      group: G624_OffCenter\ Parameters {
      } // group /G6_Plot\ Parameters/G62_Overlay\ Data/G624_OffCenter\ Parameters
    } // group /G6_Plot\ Parameters/G62_Overlay\ Data
    group: G63_Color\ Code\ DDR\ and\ Post\ Filter {
      group: G631_Post_Filter\ Appiled {
      } // group /G6_Plot\ Parameters/G63_Color\ Code\ DDR\ and\ Post\ Filter/G631_Post_Filter\ Appiled
    } // group /G6_Plot\ Parameters/G63_Color\ Code\ DDR\ and\ Post\ Filter
  } // group /G6_Plot\ Parameters
  group: G7_Image {
    dimensions:
      phony_dim_10 = 986 ;
      phony_dim_11 = 3 ;
      phony_dim_9 = 745 ;

    variables:
      ubyte p001_Image(phony_dim_9,phony_dim_10,phony_dim_11) ;
        string p001_Image:IMAGE_VERSION = "1.2" ;
        p001_Image:IMAGE_MINMAXRANGE = 0ub, 255ub ;
        string p001_Image:IMAGE_SUBCLASS = "IMAGE_TRUECOLOR" ;
        string p001_Image:INTERLACE_MODE = "INTERLACE_PIXEL" ;
  } // group /G7_Image
} // group /
zender@spectral:~/Downloads$ 

[WardF]

Thanks @czender, that handles it :)