Change to NetCDF library behavior specifying read with strides

[karlmsmith]

Reported by @AnsleyManke on 14 Aug 2012 21:39 UTC For some OPeNDAP urls, a request with striding which worked with NetCDF versions previous to NetCDF4.2, fails when using NetCDF 4.2

The API for reading data with strides is to specify index values: start, count, and stride. When start+(stride*count) is larger than the highest index on the time axis, previously we were returned data at the start of each of those intervals. Now it seems to want to have two complete strided timesteps.

So if the data is say, daily data with 365 days, we could ask for sst[L=288:352:30] and get back data in mid-October, mid-November and mid-December. With NetCDF 4.2, it seems that the request works only if there is 30 days on the time axis AFTER the last requested timestep.

I've reported this to Unidata and they're working on it.

Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/1957

[karlmsmith]

Comment by @AnsleyManke on 11 Dec 2012 18:54 UTC An example is this. The data is monthly:

yes? use "http://data1.gfdl.noaa.gov:8380/thredds3/dodsC/dc_CM2.1_R1_Cntr-ITFblock_monthly_ocean_tripolar_01010101-02001231"

yes? sh grid so
    GRID GGU6
 name       axis              # pts   start                end
 LON       LONGITUDE          360mr   0.5E                 0.5W
 LAT       LATITUDE           200 i   81.5S                89.5N
 DEPTH     DEPTH (m)           50 i-  5                    5316.4
 TIME      TIME              1200 i   16-JAN-0101 12:00    16-DEC-0200 12:00
 normal    E
 normal    F

yes?  list so[i=300,j=140,k=1,L=1100:1200]
             VARIABLE : Salinity (1e-3)
             DATA SET : GFDL CM2.1, Closed ITF Experiment
             FILENAME : dc_CM2.1_R1_Cntr-ITFblock_monthly_ocean_tripolar_01010101-02001231
             FILEPATH : http://data1.gfdl.noaa.gov:8380/thredds3/dodsC/
             SUBSET   : 101 points (TIME)
             CALENDAR : NOLEAP
             LONGITUDE: 60.5W
             LATITUDE : 29.5N
             DEPTH (m): 5
                          60.5W 
                         300
 16-AUG-0192 12 / 1100:  37.08
 16-SEP-0192 00 / 1101:  37.14
 16-OCT-0192 12 / 1102:  37.13
 16-NOV-0192 00 / 1103:  37.07
 16-DEC-0192 12 / 1104:  36.99
 16-JAN-0193 12 / 1105:  36.90
 15-FEB-0193 00 / 1106:  36.80
...
 16-JUL-0200 12 / 1195:  36.96
 16-AUG-0200 12 / 1196:  37.05
 16-SEP-0200 00 / 1197:  37.08
 16-OCT-0200 12 / 1198:  37.04
 16-NOV-0200 00 / 1199:  36.94
 16-DEC-0200 12 / 1200:  36.90

If we want every August value, we'd like to be able to say this - and the library would sort out where to end the request, returning data from August 0192 through August 200. The netcdf4.1 library did let us do this.

yes?  list so[i=300,j=140,k=1,L=1100:1200:12]

but,

yes?  list so[i=300,j=140,k=1,L=1100:1200:12]
syntax error, unexpected WORD_STRING, expecting WORD_WORD
context: Error { code = 3; message = "Invalid Parameter Exception: DArrayDimension.setProjection: Bad Projection Request: stop >= size"^;};
 **netCDF error
             NetCDF: Malformed or inaccessible DAP DATADDS (OPeNDAP/netCDF Error code -73)
             Data set: http://data1.gfdl.noaa.gov:8380/thredds3/dodsC/dc_CM2.1_R1_Cntr-ITFblock_monthly_ocean_tripolar_01010101-02001231

Some requests return data, but the last point which should be valid data, is returned as zero:

es? list so[i=300,j=140,k=1,L=1100:1190:24]
             VARIABLE : Salinity (1e-3)
                        regrid: on T
             DATA SET : GFDL CM2.1, Closed ITF Experiment
             FILENAME : dc_CM2.1_R1_Cntr-ITFblock_monthly_ocean_tripolar_01010101-02001231
             FILEPATH : http://data1.gfdl.noaa.gov:8380/thredds3/dodsC/
             SUBSET   : 4 points (TIME)
             CALENDAR : NOLEAP
             LONGITUDE: 60.5W
             LATITUDE : 29.5N
             DEPTH (m): 5
                 60.5W 
                300
 AUG-0192 / 1:  37.08
 AUG-0194 / 2:  36.88
 AUG-0196 / 3:  36.84
 AUG-0198 / 4:   0.00

[karlmsmith]

Comment by steven.c.hankin on 11 Dec 2012 19:06 UTC This trac ticket is 4 months old. At the time of the ticket the problem had been reported to Unidata. Haven't they released a net netCDF library since that time? Have we heard back from them?

[karlmsmith]

Comment by @AnsleyManke on 11 Dec 2012 20:44 UTC The last release from Unidata was August 3. In Sept, in an exchange with Dennis he said he has a fix for this. I've prodded him a couple times, and just wrote him again.

[karlmsmith]

Comment by @AnsleyManke on 18 Dec 2012 21:52 UTC Unidata does have a fix for this in their code base. I verified it using their daily snapshot releases. Dennis Heimbigner says,

... we are going thru the process of releasing netcdf 4.3, but it may be a couple of weeks. but it may be a couple of weeks. We are teaching our newest team member (Ward Fisher) how to do it and he will, of course, make some errors along the way.

Nice to hear they're getting some more people on board.