NetCDF4Classic fatal error in ncarg_6.5.0 but not in 6.4.0

[emanuelelombardi]

running a given script using ncl 6.5.0 I get a fatal error which does'nt occur running the same script in ncl 6.4.0 The error happens using setfileoption("nc", "Format", "NetCDF4Classic") To show the error I use a simple code named test.nc (at the end of this text) which creates the file test.nc I run test.nc both ncl 6.4.0 and 6.5.0 As you can see running it on 6.4.0 there are no errors, but running on 6.5.0 a fatal error is reported (but the file is properly created)

export NCARG_ROOT=/usr/local/ncarg_6.4 export PATH=$NCARG_ROOT/bin:$PATH ncl test.ncl Copyright (C) 1995-2017 - All Rights Reserved University Corporation for Atmospheric Research NCAR Command Language Version 6.4.0 The use of this software is governed by a License Agreement. See http://www.ncl.ucar.edu/ for more details. (0) done!

export NCARG_ROOT=/usr/local/ncarg_6.5.0 export PATH=$NCARG_ROOT/bin:$PATH ncl test.ncl Copyright (C) 1995-2018 - All Rights Reserved University Corporation for Atmospheric Research NCAR Command Language Version 6.5.0 The use of this software is governed by a License Agreement. See http://www.ncl.ucar.edu/ for more details. ncattput: ncid 65536: NetCDF: Attempt to define fill value when data already exists. fatal:Attempt to delete undefined attribute from variable (0) done!

;----------------------------------------------------------- load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl" load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl" load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl" ;----------------------------------------------------------- begin ofile = "test.nc"

;--- set dimensions --- nlon = 100 nlat = 200 nlev = 100

data = new((/ nlev, nlat, nlon /), "float", 1.0e20)

;--- create new file --- system("/bin/rm -f "+ofile) setfileoption("nc", "Format", "NetCDF4Classic") ncout = addfile(ofile, "c")

;--- define mode on --- setfileoption(ncout, "DefineMode", True)

;--- define dimensions --- dimNames = (/ "lev", "iy", "jx" /) dimSizes = (/ nlev, nlat, nlon /) dimUnlim = (/ False, False, False /) filedimdef(ncout, dimNames, dimSizes, dimUnlim)

;--- define variables --- filevardef(ncout, "lat", "float", (/ "iy", "jx" /)) filevardef(ncout, "lon", "float", (/ "iy", "jx" /)) filevardef(ncout, "lev" , "float", "lev") filevardef(ncout, "THETA", "float", (/ "lev", "iy", "jx" /))

;Define the compress level filevarcompressleveldef(ncout, "THETA", 9)

;--- exit file definition mode --- setfileoption(ncout, "DefineMode", False)

lat=0. lon=1. depth=100.

;--- fill data --- ncout->lat = (/ lat /) ncout->lon = (/ lon /) ncout->lev = (/ depth /) data(:,:,:) = -5. ncout->THETA(:,:,:) = (/ data(:,:,:) /) print ("done!") end

[rbrownrigg]

I speculate this change in behavior occurred in the underlying NetCDF 4.5.0 library; you can read more about it at:

https://github.com/Unidata/netcdf-c/issues/390

Note that this particular NetCDF ticket indicates that with the previous behavior, existing data was erased when a _FillValue was redefined. Is that the case for youo with NCL 6.4.0?

Also note that there's some discussion about introducing a _FillValue while in DefineMode. Perhaps in your situation, you'll need to add a _FillValue attribute (to THETA before exiting DefineMode (using filevarattsdef()) ?

[emanuelelombardi]

thanks a lot, rbrownrigg! Your suggestion works: I add the following lines before exiting DefineMode Now the code works both under 6.4.0 and 6.5.0

attrf=0.
attrf@_FillValue = 1.0e20 filevarattdef( ncout, "THETA", attrf ) delete(attrf)

[Dave-Allured]

I can confirm that the original test program above contains a protocol violation for netcdf-4 format, therefore it is not a valid example. Rick's suggested fix is valid because it defines a _FillValue attribute while THETA is in initial define mode. This satisfies the required protocol.

Rick pointed to the correct netcdf-C bug report. A bug in the underlying netcdf-C library, version 4.4.1, accepted the invalid sequence, and also corrupted any previously written data as a charming side effect. This bug was corrected in the next netcdf-C release.

As Rick suggested, this probably explains the change in behavior between NCL versions. The restriction on _FillValue in later NCL versions is correct behavior.

In my opinion, older scripts that trigger this error are not valid, and the proper fix is to correct _FillValue sequences in those scripts.

[Dave-Allured]

If a data array has no missing values, and you do not want a _FillValue attribute to be written to the output file, then you can take a different approach. Just suppress the _FillValue attribute in the first place. From the original test program above, change this line:

data = new((/ nlev, nlat, nlon /), "float", 1.0e20)

to this:

data = new((/ nlev, nlat, nlon /), "float", "No_FillValue")

Then you can skip this step:

attrf=0.
attrf@_FillValue = 1.0e20
filevarattdef( ncout, "THETA", attrf )
delete(attrf)

and just write the "data" variable directly to the output file, as shown in the original test program.

[Dave-Allured]

IMO this ticket is complete and should be closed.