Make NetCDF output conform to spec

dholstius commented 2 years ago

Summary

The shared InMAP NetCDF content (e.g. ca_isrm.ncf) appears to be invalid
- ncvalidator states that it fails to conform with classic CDF-2 file format spec
- It contains 5 large fixed-size variables, while the CDF-2 format allows only one
Fresh NCO install on OS X won't read it (issue #4)
Fixing this would be nice

Details

Although ncdump gives up:

dholstius$ NC_PATH="/Users/dholstius/GitHub/BAAQMD/InMAP-SFAB/Data/GDrive/ca_isrm.nc"
dholstius$ ncdump -k $NC_PATH
ncdump: /Users/dholstius/GitHub/BAAQMD/InMAP-SFAB/Data/GDrive/ca_isrm.nc: NetCDF: One or more variable sizes violate format constraints

... we can fall back to reading the first four bytes:

dholstius$ od -An -c -N4 $NC_PATH
           C   D   F 002

Those indicate CDF-2 (64-bit offset) format.

After brew install pnetcdf:

dholstius$ ncvalidator $NC_PATH 
Error:
    Input file contains 5 large fixed-size variables
    CDF-2 format allows only one large fixed-size variable
    The 1st large fixed-size variable is PrimaryPM25
    The 2nd large fixed-size variable is SOA
    See: http://www.unidata.ucar.edu/software/netcdf/docs/file_structure_and_performance.html#offset_format_limitations
File "/Users/dholstius/GitHub/BAAQMD/InMAP-SFAB/Data/GDrive/ca_isrm.nc" fails to conform with classic CDF-2 file format specifications

Related Issues

4

dholstius commented 2 years ago

FYI @ctessum. I'm going to try converting to another NetCDF format on a different host with a different NCO stack, and see if that's a workaround. h/t @stephenreid65

ctessum commented 2 years ago

The issue is that the Go language netcdf library has an apparently invalid way of creating large files which worked fine with older versions of the NetCDF C library but doesn't work with the newest version. A workaround is to install a circa-2016 version of netcdf and then make a copy of the file to netcdf-4, which should then work with any version of the netcdf C library. (This might be the same thing as the workaround you posted above.) The only problem with this solution is that InMAP can't read NetCDF-4, so the new file won't be useable with the InMAP command line tools.

dholstius commented 2 years ago

Good to know. I was going bonkers trying to fix my netCDF stack, and envious that @stephenreid65 wasn't getting the same error on our cluster host (soma). Maybe the netCDF install is old enough on our cluster that it works! I'm going to have to take back my suggestion to Jeff (our sysadmin) to update it.

Update: on soma, nc-config –version indeed reports a circa-2016 version, 4.4.1.1 (2016-11-21).

BAAQMD / InMAP-SFAB