NOAA-PMEL / Ferret

The Ferret program from NOAA/PMEL
https://ferret.pmel.noaa.gov/Ferret/
The Unlicense
55 stars 21 forks source link

change in default dimension order for USE (SET DATA/FORM=CDF) in 6-dimensional Ferret #1531

Open karlmsmith opened 6 years ago

karlmsmith commented 6 years ago

Reported by @AndrewWittenberg on 7 Apr 2015 22:33 UTC I hesitate to even report this, since it's probably more of a feature than a bug -- but it might be good to perhaps document this change in behavior in the user's guide and/or release notes. (One of my old scripts was failing due to this change, and it took me a while to figure out why.)

First, create a file "b.nc" which has two dimensions: TIME, plus another "unlabeled" dimension (COADSX) which has neither an "axis" attribute nor a "units" attribute. Also swap the dimension order.

    NOAA/PMEL TMAP
    FERRET v6.933 (beta)
    Linux 2.6.32-504.3.3.el6.x86_64 64-bit - 01/27/15
     7-Apr-15 18:03     

yes? use coads_climatology
yes? save/clob/file=a.nc sst[y=@max]
 LISTing to file a.nc
yes? sp ncatted -O -h -a axis,COADSX,d,, -a units,COADSX,d,, a.nc
yes? sp ncpdq -O -h -a COADSX,TIME a.nc b.nc

Now if we USE b.nc, there's no way for Ferret to know the direction of COADSX, so it somehow has to assign a default direction. This default appears to have changed with the introduction of the E & F axes in 6-dimensional Ferret.

In the old single-precision 4-d Ferret (v6.725), COADSX came in as an X-axis:

    NOAA/PMEL TMAP
    FERRET v6.725  
    Linux 2.6.18-274.12.1.el5 64-bit - 04/10/12
     7-Apr-15 18:08     

yes? use b.nc; sho dat
     currently SET data sets:
    1> ./b.nc  (default)
 name     title                             I         J         K         L
 SST      SEA SURFACE TEMPERATURE          1:180     ...       ...       1:12
       (Y=90S:90N@MAX)

But then in the earliest version of 6-d Ferret, it started coming in as an E-axis:

    NOAA/PMEL TMAP
    FERRET v6.8 (beta1, 6D)
    Linux 2.6.18-308.1.1.el5 64-bit - 05/08/12
     7-Apr-15 18:15     

yes? use b.nc; sho dat
     currently SET data sets:
    1> ./b.nc  (default)
 name     title                             I         J         K         L         M         N
 SST      SEA SURFACE TEMPERATURE          ...       ...       ...       1:12      1:180     ...
       (Y=90S:90N@MAX)

Presumably Ferret is first assigning axes with known directions (either from the "axis" or "units" attribute, or perhaps from the dimension name in some cases) to their specified directions. Then it somehow fills in the remaining spots with the "unlabeled" dimensions, based on the ordering in the NetCDF file.

Are the rules for assigning unlabeled axes documented somewhere?

As a workaround, note that the old behavior can be reproduced in 6-d Ferret, by explicitly specifiying the axis order:

    NOAA/PMEL TMAP
    FERRET v6.933 (beta)
    Linux 2.6.32-504.3.3.el6.x86_64 64-bit - 01/27/15
     7-Apr-15 18:24     

yes? use/ord=tx b.nc; sho dat
     currently SET data sets:
    1> ./b.nc  (default)
 name     title                             I         J         K         L         M         N
 SST      SEA SURFACE TEMPERATURE          1:180     ...       ...       1:12      ...       ...
       (Y=90S:90N@MAX)

This is the workaround that I'm using in my script.

So there may be nothing to do here, other than verifying that the behavior change was intended, and adding a line or two to the documentation.

Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/2259

karlmsmith commented 6 years ago

Comment by @AnsleyManke on 7 Apr 2015 23:14 UTC In addition to the change to 6 dimensions, we've been working to pull out some heuristics that used to be used to determine axis direction.

Just previous to 6-D ferret, this was the change: http://ferret.pmel.noaa.gov/Ferret/documentation/release-notes/version-6-7-release-notes#CF_compliance

This was in response to cases where there was a dimension like TSTATION which was being put in the T direction because its name stars with T, but which really belonged in the X direction; and because ambiguous axes like these were being given different directions in different grids in the same dataset.

Attributes in the dataset are probably our best and most consistent solution to this issue. For this particular example, here's the ncdump header:

> ncdump -h b.nc 
netcdf b {
dimensions:
        COADSX = UNLIMITED ; // (180 currently)
        TIME = 12 ;
variables:
        double COADSX(COADSX) ;
                COADSX:modulo = " " ;
                COADSX:point_spacing = "even" ;
                COADSX:standard_name = "longitude" ;
        double TIME(TIME) ;
                TIME:units = "hour since 0000-01-01 00:00:00" ;
                TIME:time_origin = "01-JAN-0000 00:00:00" ;
                TIME:modulo = " " ;
                TIME:axis = "T" ;
                TIME:standard_name = "time" ;
        float SST(COADSX, TIME) ;
                SST:missing_value = -1.e+34f ;
                SST:_FillValue = -1.e+34f ;
                SST:long_name = "SEA SURFACE TEMPERATURE" ;
                SST:history = "From /home/users/tmap/ferret/linux/fer_dsets/data/coads_climatology.cdf" ;
                SST:units = "Deg C" ;
                SST:long_name_mod = "Y=90S:90N@MAX" ;

// global attributes:
                :history = "FERRET V6.94    7-Apr-15" ;
                :Conventions = "CF-1.6" ;
}

The attribute COADSX:standard_name = "longitude" should determine the direction of COADSX. There is a bug in Ferret that means this doesn't happen(try the same example using COADSY and you will see it working correctly). Does your actual example have a standard_name attribute?

If there are grids which have coordinate variables with NO CLUES AT ALL, then the ordering seen in the file is used, according to the COARDS conventions: ferret.pmel.noaa.gov/Ferret/documentation/coards-netcdf-conventions/

in there is a section, "Order of dimensions", which says, "...those dimensions should appear in the relative order T, then Z, then Y, then X in the CDL definition corresponding to the file." I believe that because the ncdump of this file has this order:

 float SST(COADSX, TIME) ;

that what 4-D Ferret used to do was determine that TIME is in the T direction, and then put COADSX in the next available dimension, wrapping around to X. What 6-D Ferret does is to use the next available dimension, namely E.

All of this is not a perfect solution. Even USE/ORDER= is not adequate because it acts in a blanket way on all the grids in the dataset. We have been thinking there is a need for more control by the user on axis directions when initializing a dataset.