Determine if delft3d file is CF compliant

[rsignell-usgs]

From: Jay Veeramony [jay.veeramony nrlssc.navy.mil] Sent: Monday, June 02, 2014 11:43 AM To: Eoin Howlett Cc: Andre Van der Westhuysen - NOAA Affiliate Subject: Re: Minutes: COMT PR/USVI call of May 21, 1-3 pm ET

Hi Eoin,

From: Jay Veeramony [mailto:jay.veeramony@nrlssc.navy.mil] Sent: Monday, June 02, 2014 11:43 AM To: Eoin Howlett Cc: Andre Van der Westhuysen - NOAA Affiliate Subject: Re: Minutes: COMT PR/USVI call of May 21, 1-3 pm ETI have placed the netcdf output file on our anonymous FTP server: ftp7320.nrlssc.navy.mil in pub/veeramon/trim-ike.nc

The file has 2-D model output every hour. I am having issues creating netcdf file for the time series output that is written to a different Delft3D output file. Once I have that figured out, I will put that file in the ftp server as well and send you an email

Let me know if you need any further details.

Thanks jay

[veeramony]

Digging deeper into the NEFIS to NETCDF conversion routine, in the netcdf file, the variables m & n are the unique values of the longitude (or x) and latitude (or y) if the grid is orthogonal and 1:length(x), 1:length(y) if the grid is non-orthogonal. The active cells during computation are (m(2:end-1),n(2:end-1)). Not sure why the conversion routine writes this field, since the grid latitudes and longitudes are also written to the netcdf file.

[rsignell-usgs]

And this is causing a problem for Iris (http://scitools.org.uk/iris/) , the British met office software I'm using as a test for CF compliance. It expects to find one longitude coordinate, but is finding two. I'll try removing the m and n variables via NcML and see if that helps.

[veeramony]

Ok. I will see if I can create a netcdf file with only lat, lon, time, water levels, u and v for now. What other variables are typically output? I can also include wind-stress, wind velocities etc.

[rsignell-usgs]

Ideally we would translate everything in a generic fashion so that the program could be used for any Delft3D NEFIS file. Is that not possible?

[veeramony]

Yes, It will be usable for any Delft3D NEFIS map file. Delft3D outputs two types of files: The map file outputs to a 3D (or 4D) grid, but at very coarse time steps (on the order of an output every hour or two). The history file creates time-series output at specified locations at much higher time resolution. So will have to write a conversion routine for each type of output.

[rsignell-usgs]

@ocefpaf, I'm trying to test this Delft3D dataset by reading and plotting data using Iris, but the plot is not appearing. A different dataset (NGOM) shows up fine. Can you see the problem? http://nbviewer.ipython.org/gist/rsignell-usgs/76ea6b62c1e375e4f14e

[ocefpaf]

@rsignell-usgs it was a matplotlib issue. The longitude and latitudes points had NaNs. I added a cf-checker report at the end and there are some errors and warnings, but the plot works now.

Someone (maybe me?) should write a version of the cf-checker script that does not require cdms.

http://nbviewer.ipython.org/gist/ocefpaf/eca3fb78e602e40f1b46

[rsignell-usgs]

So lon and lat had NaNs, but not missing_values? That would be an easy fix for @veeramony to make (or we could do it in NcML)

[rsignell-usgs]

@ocefpaf, see question above. What was the problem that the file caused for matplotlib?

[ocefpaf]

The NaNs. IPython notebook hides the proper error/warning messages with its figure display wrapper. I only found it because I had this problem before and the first thing I did was to run the .py in the IPython interpreter instead of the notebook.

[rsignell-usgs]

But why didn't netcdf4-python recognize the NaN as missing_value and create masked arrays like is usually does?

[ocefpaf]

I am not sure about the inner works of netcdf4-python, I have to check. But if I had to guess it is filling using what is specified below instead of masking.

longitude:_FillValue = NaNf ; longitude:missing_value = NaNf ;

[ocefpaf]

It should have have been fixed here https://github.com/Unidata/netcdf4-python/issues/168...

[rsignell-usgs]

Hmmm.. My netcdf4-python version in anaconda is v1.0.8 from Feb 7, 2014, quite a bit later than that release from March 2013. So should I raise an issue on https://github.com/Unidata/netcdf4-python ?

[ocefpaf]

Let me make a few more test first. A PR is better than raising an issue :smile:

[ocefpaf]

Found it the problem is iris and not netcdf4-python. Check cells 7-8.

http://nbviewer.ipython.org/gist/ocefpaf/911d6ff9ef954b211d01

I believe that is by choice and not a bug from iris devs. If the _FillValue is NaN you will get NaN.

[dpsnowden]

Before anyone writes any cf-checker code please check out ioos/compliance-checker and talk with @daf.

Sent from my iPhone

On Jun 10, 2014, at 4:45 PM, Filipe notifications@github.com wrote:

@rsignell-usgs https://github.com/rsignell-usgs it was a matplotlib issue. The longitude and latitudes points had NaNs. I added a cf-checker report at the end and there are some errors and warnings, but the plot works now.

Someone (maybe me?) should write a version of the cf-checker script that does not require cdms.

http://nbviewer.ipython.org/gist/ocefpaf/eca3fb78e602e40f1b46

— Reply to this email directly or view it on GitHub https://github.com/ioos/comt/issues/2#issuecomment-45669177.

[daf]

Yes, ioos/compliance-checker is nearing 1.0 with the completion of the CF checking suite (it does more than just CF), @DanielJMaher is currently comparing output with existing checkers to make sure we've got things correct. He's adding commits to ioos/compliance-checker#9, you can subscribe to that issue to see when we merge it (1.0 will follow not long after).

[daf]

@ocefpaf See above (I assume we cross posted). CF compliance portion is not finished, but is very close.

[ocefpaf]

Great! Thanks.

[rsignell-usgs]

@ocefpaf , " If the _FillValue is NaN you will get NaN." Why wouldn't we get a masked array? I'm sorry, I still don't understand.

[ocefpaf]

netcdf4-python masks the NaNs automatic and returns a MaskedArray, apparently iris returns a the filled array and leave the masking to the user. I think that is a design choice, but we have to ask them.

[rsignell-usgs]

Okay, I understand now. I will ask them.

[rsignell-usgs]

Closing this issue, since it doesn't seem to be a problem with the Delft3D file, and it plots okay with Iris: http://nbviewer.ipython.org/gist/rsignell-usgs/b246e59dc9f22cd1bd3c