Why does this S5P dataset break ncks?

[czender]

This S5P L1B file breaks ncks, and the question is why. If the answer is fixable, then we'll do that later, for now we just need to know why:

zender@aerosol:~/data$ ncks -v spectral_channel_quality -m s5p.nc
ncks: WARNING File contains 6 non-atomic (e.g., compound, enum_t, opaque, vlen_t, and user-defined) variable types. NCO currently ignores non-atomic variables. ncks will process and copy to output only the atomic variables.
netcdf S5P_TEST_L1B_RA_BD1_20171119T041255_20171119T055342_00522_04_010000_20180401T080000 {
  group: BAND1_RADIANCE {
    group: STANDARD_MODE {
      dimensions:
        ground_pixel = 77 ;
        ncorner = 4 ;
        scanline = 63 ;
        spectral_channel = 497 ;
        time = 1 ;
      group: OBSERVATIONS {

        variables:
          int ground_pixel(ground_pixel) ;
            ground_pixel:comment = "This dimension variable defines the indices across track; index starts at 0" ;
            ground_pixel:long_name = "across track dimension index" ;
            ground_pixel:units = "1" ;

          int scanline(scanline) ;
            scanline:comment = "This dimension variable defines the indices along track; index starts at 0" ;
            scanline:long_name = "along track dimension index" ;
            scanline:units = "1" ;

Segmentation fault: 11

The warning clearly says there are non-atomic type (NAT, i.e., vlen_t) variables. Is that what breaks it? ncks was designed to ignore NAT variables, but maybe in some cases it does not. Or maybe it's a fixable bug in the ncks print or extraction code. Please run this through a debugger and see why it fails. I got this stacktrace but do not have time to follow through:

(gdb) where
#0  0x000015555472ce60 in nc_utf8proc_iterate () from /usr/local/lib/libnetcdf.so.13
#1  0x000015555472dfbf in nc_utf8proc_decompose_custom () from /usr/local/lib/libnetcdf.so.13
#2  0x000015555472e8c2 in nc_utf8proc_map_custom () from /usr/local/lib/libnetcdf.so.13
#3  0x000015555472e868 in nc_utf8proc_map () from /usr/local/lib/libnetcdf.so.13
#4  0x000015555472203b in nc_utf8_normalize () from /usr/local/lib/libnetcdf.so.13
#5  0x00001555547c7ea4 in nc4_normalize_name () from /usr/local/lib/libnetcdf.so.13
#6  0x00001555547cd067 in NC4_inq_varid () from /usr/local/lib/libnetcdf.so.13
#7  0x0000155554720f4a in nc_inq_varid () from /usr/local/lib/libnetcdf.so.13
#8  0x000015555509e2a6 in nco_inq_varid (nc_id=65539, 
    var_nm=0x13f61 <error: Cannot access memory at address 0x13f61>, var_id=var_id@entry=0x7ffffffe115c)
    at nco_netcdf.c:1807
#9  0x00001555550afc43 in nco_prn_cdl_trd (nc_id=nc_id@entry=65536, 
    grp_nm_fll=grp_nm_fll@entry=0xb80920 "/BAND1_RADIANCE/STANDARD_MODE/OBSERVATIONS", 
    prn_flg=prn_flg@entry=0x7ffffffe60b0, trv_tbl=trv_tbl@entry=0x61dd30) at nco_prn.c:3060
#10 0x00001555550b00bd in nco_prn_cdl_trd (nc_id=nc_id@entry=65536, 
    grp_nm_fll=grp_nm_fll@entry=0xb80b70 "/BAND1_RADIANCE/STANDARD_MODE", 
    prn_flg=prn_flg@entry=0x7ffffffe60b0, trv_tbl=trv_tbl@entry=0x61dd30) at nco_prn.c:3119
#11 0x00001555550b00bd in nco_prn_cdl_trd (nc_id=nc_id@entry=65536, 
    grp_nm_fll=grp_nm_fll@entry=0xb809b0 "/BAND1_RADIANCE", prn_flg=prn_flg@entry=0x7ffffffe60b0, 
    trv_tbl=trv_tbl@entry=0x61dd30) at nco_prn.c:3119
#12 0x00001555550b00bd in nco_prn_cdl_trd (nc_id=nc_id@entry=65536, 
    grp_nm_fll=grp_nm_fll@entry=0x7ffffffe5fe6 "/", prn_flg=prn_flg@entry=0x7ffffffe60b0, 
    trv_tbl=<optimized out>) at nco_prn.c:3119
#13 0x0000000000406561 in main (argc=<optimized out>, argv=<optimized out>) at ncks.c:1349

Thanks!

[hmb1]

@czender The below command is working for me with the latest repo version of NCO ncks -v spectral_channel_quality -m s5p.n

With the below command I am getting ncks -m s5p.nc

nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_inq_varid() nco_err_exit(): ERROR Error code is -59. Translation into English with nc_strerror(-59) is "NetCDF: Name contains illegal characters" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

[czender]

This fails for me on MacOS, yet works on Linux: ncks -v spectral_channel_quality -m s5p.nc This fails on both: ncks -m s5p.nc Please continue investigation into second command failure on Linux.

[hmb1]

@czender have done a fix to nco_prn_cdl_trd() so that COMPOUND/ENUM/VLEN/OPAQUE types are ignored and NOT marked for extraction. Maybe the traversal table should NOT be picking up these types. Let me think some more on this. Anyways my fix is in the branch hmb_fix_cdl

The command ncks -m s5p.nc now works

[czender]

Great! Please propagate the fix to XML and JSON printing if you have not already. We want to keep each output type working for all input classes. Also take time and think how this should be done with an eye towards allowing the easiest non-atomic types (enum and vlen?) to work with certain NCO features, starting with extraction and printing. 4.7.4 should include the fixes that allow printing of as much of the S5P datasets as possible with the minimal hassle.

[hmb1]

@czender OK I have a more thematic one line fix - will add fix to XML & JSON and then commit in branch hmb_fix_cdl

[hmb1]

ok json, xml, trd, cdl all fixed in branch hmb_fix_cdl

[czender]

Please merge to master then close issue. If you have ideas on extracting/printing non-atomic types, please open a new issue to discuss. This S5P dataset is a good one to use to prioritize changes that are useful to the community.

[hmb1]

merged