NcML Aggregations don't issue error when dimension size changes

dopplershift commented 9 years ago

Taken from esupport (EWU-865844)

''' If TDS is asked to "joinExisting" aggregate 2 .nc files that are identical (e.g., same dimension and variable names), except that one has twice as many lat and lon values (2x the resolution) and thus 4x as many data values, TDS will aggregate the files (as far as I know, without throwing an error message) and present the dataset to users.

At least right now, you can see this behavior with this dataset http://oceanwatch.pfeg.noaa.gov/thredds/dodsC/satellite/QS/ux10/14day.html The .dds reflects the data files with the larger lat and lon dimension sizes:

Dataset { Float64 altitude[altitude = 1]; Float64 lat[lat = 1201]; Float64 lon[lon = 2881]; Float64 time[time = 2454]; Grid { ARRAY: Float32 QSux10[time = 2454][altitude = 1][lat = 1201][lon = 2881]; MAPS: Float64 time[time = 2454]; Float64 altitude[altitude = 1]; Float64 lat[lat = 1201]; Float64 lon[lon = 2881]; } QSux10; } satellite/QS/ux10/14day;

If you request data from the last time point http://oceanwatch.pfeg.noaa.gov/thredds/dodsC/satellite/QS/ux10/14day.ascii?QSux10[2453][0:1:0][0:1200:1200][0:2880:2880] it succeeds because the file for that time point has 1201 lat values and 2881 lon values, which is also what the .dds indicates.

If you request data from the first time point http://oceanwatch.pfeg.noaa.gov/thredds/dodsC/satellite/QS/ux10/14day.ascii?QSux10[0][0:1:0][0:1200:1200][0:2880:2880] it fails with the message Error { code = 500; message = "NcSDArray InvalidRangeException=Illegal Range for dimension 2: last requested 1200 > max 600"; };

If you want 2 different-sized sample files from this dataset, let me know and I can email them to you (or make them available to you some other way if email fails).

* Caveats *

I didn't see a note/warning about this behavior in

https://www.unidata.ucar.edu/software/thredds/current/netcdf-java/ncml/Aggregation.html If I missed it, I'm sorry.

Is there a switch in the TDS setup that allows me to turn on/ turn off strict checking of files to be aggregated?
Maybe TDS does output an error message to a log file when it does the aggregation. I didn't see one. (I confess I didn't look super hard.) In any case, TDS successfully creates the aggregation and displays it to users.
Maybe you think this is appropriate behavior ("It's not a bug; it's a feature!"). Perhaps there are use cases where it is useful for TDS to be able aggregated different-sized files. Fine. If so, please just tell me that this is a feature (not a bug), and I will accept that.

Thank you for looking into this. '''

dopplershift commented 9 years ago

There's also the open question of whether FMRC can be used to perform the task of time aggregation of netcdf files.

JohnLCaron commented 9 years ago

NcML error checking, in general, sucks.

FMRC can be used if the file is recognized as a Grid.

On Thu, Sep 3, 2015 at 10:52 AM, Ryan May notifications@github.com wrote:

There's also the open question of whether FMRC can be used to perform the task of time aggregation of netcdf files.

— Reply to this email directly or view it on GitHub https://github.com/Unidata/thredds/issues/168#issuecomment-137509489.

Unidata / thredds

NcML Aggregations don't issue error when dimension size changes #168