v1 migration issue - Time axis appears unordered when aggregating across multiple ncdf files

[julian1]

Hi. We have been experimenting with the new version of ncwms and testing it against our current configuration and data. We noticed the following issue, which affects the many of our datasets.

With a dataset config path,

 <dataset id="ACORN_SAG_nonQC" location="/mnt/opendap/1/IMOS/opendap/ACORN/gridded_1h-avg-current-map_non-QC/SAG/????/??/??/*.nc" ... />

We get the following exception,

java.lang.IllegalArgumentException: Coordinate values must increase or decrease monotonically
        at uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.checkAscending(AbstractIrregularAxis.java:121)
        at uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.init(AbstractIrregularAxis.java:102)
        at uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.<init>(AbstractIrregularAxis.java:63)
        at uk.ac.rdg.resc.edal.grid.TimeAxisImpl.<init>(TimeAxisImpl.java:51)
        at uk.ac.rdg.resc.edal.util.cdm.CdmUtils.createTimeAxis(CdmUtils.java:296)
        at uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.createDataset(CdmGridDatasetFactory.java:216)
        at uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.createDataset(CdmGridDatasetFactory.java:85)
        at uk.ac.rdg.resc.edal.catalogue.jaxb.DatasetConfig.createDataset(DatasetConfig.java:254)
        at uk.ac.rdg.resc.edal.catalogue.jaxb.DatasetConfig.refresh(DatasetConfig.java:213)
        at uk.ac.rdg.resc.edal.catalogue.jaxb.CatalogueConfig$1.run(CatalogueConfig.java:154)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

Putting a trace in AbstractIrregularAxis.checkAscending() shows the time data is not ordered,

value 2014-01-01T05:30:00.000Z
value 2014-01-01T18:30:00.000Z
value 2014-01-01T17:30:00.000Z
value 2014-01-01T16:30:00.000Z
...

In fact, the order appears to be arbitrary and matches whatever was returned by the underlying jni filesystem api.

# find  /mnt/opendap/1/IMOS/opendap/ACORN/gridded_1h-avg-current-map_non-QC/SAG/2014/01/01/ -type f
/mnt/opendap/1/IMOS/opendap/ACORN/gridded_1h-avg-current-map_non-QC/SAG/2014/01/01/IMOS_ACORN_V_20140101T053000Z_SAG_FV00_1-hour-avg.nc
/mnt/opendap/1/IMOS/opendap/ACORN/gridded_1h-avg-current-map_non-QC/SAG/2014/01/01/IMOS_ACORN_V_20140101T183000Z_SAG_FV00_1-hour-avg.nc
/mnt/opendap/1/IMOS/opendap/ACORN/gridded_1h-avg-current-map_non-QC/SAG/2014/01/01/IMOS_ACORN_V_20140101T173000Z_SAG_FV00_1-hour-avg.nc
/mnt/opendap/1/IMOS/opendap/ACORN/gridded_1h-avg-current-map_non-QC/SAG/2014/01/01/IMOS_ACORN_V_20140101T163000Z_SAG_FV00_1-hour-avg.nc
/

I believe the correct (ncwms v1) behavior would be to open each netcdf, record the time scalar, and then sort to construct the ordered axis of values.

Attempting a fix by sorting the values in place in AbstractIrregularAxis, does get rid of the exception.

However, I don't think this solution is valid, since the local ordering in AbstractIrregularAxis will be inconsistent with other parts of the dataset catalogue.

[guygriffiths]

Fixed in develop branch. EDAL was ordering the files by the values of their time axes, but was reading the time values as longs. When these were all on the same day, no reordering was applied. Fixed by reading time values as floats

[jonblower]

I don't quite understand this - aren't the longs accurate to millisecond precision?

[guygriffiths]

We read the time values directly from the NetCDF file, so the precision is whatever the units of the time axis are. In Julian's example, the units were "days since...", the values were stored as floats, and all of the files were on the same day.

[jonblower]

Oh I see. Can we just automatically convert the axis to java Date objects (I think this is what is done by a CDM API method)?

Really I think it's bad practice to use non-integer values for the time values - instead, data providers should probably use a smaller time unit, like seconds. But perhaps it was unavoidable in this case.

[guygriffiths]

I've not found an automatic method to convert the values to Date objects (or similar), but it's not really necessary - this only applies to sorting out which order files should appear in in the NcML aggregation. Since we're using "joinExisting" they all need to have the same units anyway (as far as I can tell).

[chcuong]

Hi. Everything is ok when we used NCWMS 1.1. Nonetheless, we have been experimenting with the new version of ncwms and testing it against our current configuration and data and I see the following error (We tried both code from master branch and develop branch)

Error report

Stack trace: java.lang.IllegalArgumentException :Coordinate values must increase or decrease monotonically uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.checkAscending(AbstractIrregularAxis.java:113) uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.init(AbstractIrregularAxis.java:102) uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.(AbstractIrregularAxis.java:63) uk.ac.rdg.resc.edal.grid.TimeAxisImpl.(TimeAxisImpl.java:51) uk.ac.rdg.resc.edal.util.cdm.CdmUtils.createTimeAxis(CdmUtils.java:296) uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.createDataset(CdmGridDatasetFactory.java:213) uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.createDataset(CdmGridDatasetFactory.java:82) uk.ac.rdg.resc.edal.catalogue.jaxb.DatasetConfig.createDataset(DatasetConfig.java:254) uk.ac.rdg.resc.edal.catalogue.jaxb.DatasetConfig.refresh(DatasetConfig.java:213) uk.ac.rdg.resc.edal.catalogue.jaxb.CatalogueConfig$1.run(CatalogueConfig.java:154) java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) java.util.concurrent.FutureTask.runAndReset(Unknown Source) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.lang.Thread.run(Unknown Source)

I tried to use the @guygriffiths solution by reading time values as floats . I applied to function getDataset of the file NetcdfDatasetAggregator.java of current developer branch as belows:

//Collections.sort(times); // We made comment to this function and try to reading time values as floats. final String aggDimName = timeDimName; Collections.sort(files, new Comparator() { @Override public int compare(File ncFile1, File ncFile2) { NetcdfFile nc1 = null; NetcdfFile nc2 = null; try { nc1 = NetcdfFile.open(ncFile1.getAbsolutePath()); nc2 = NetcdfFile.open(ncFile2.getAbsolutePath()); Variable timeVar1 = nc1.findVariable(aggDimName); Variable timeVar2 = nc2.findVariable(aggDimName); float time1 = timeVar1.read().getFloat(0); float time2 = timeVar2.read().getFloat(0); return Float.compare(time1, time2); } catch (Exception e) { / * There was a problem reading the data. Sort * alphanumerically by filename and hope for the * best... * * This catches all exceptions because however * it fails this is still our best option. * * If the error is a genuine problem, it'll show * up as soon as we try and aggregate. / return ncFile1.getAbsolutePath().compareTo( ncFile2.getAbsolutePath()); } finally { if (nc1 != null) { try { nc1.close(); } catch (IOException e) { log.error("Problem closing netcdf file", e); } } if (nc2 != null) { try { nc2.close(); } catch (IOException e) { log.error("Problem closing netcdf file", e); } }

    }
}

});

Please help us to fix this issue. We appriciate your help.

[guygriffiths]

Have you got any test data you can give us to reproduce this error?

[chcuong]

We have found that this is our data error. Thanks.

[tomLandry]

Hello everyone. We are experimenting this very same error right now, with NcWMS v2.0.4. We will troubleshoot our datasets, but any guidance is appreciated!

[guygriffiths]

@tomLandry - Is this when aggregating multiple files into a single dataset? Do you have any data you could share so that I can reproduce the error?

[tomLandry]

Thanks for the lightning quick answer! We are simultaneously trying to upgrade of NcWMS version from 2.02 to latest, and massaging data. I'll see if we can send you a sample that is breaking the thing.

[docgregt]

Hi, I just downloaded the latest ncWMS server (2.4.1)...just ran it with java to test it. All ECMWF files I try get this error (java.lang.IllegalArgumentException: Coordinate values must increase or decrease monotonically). These files load up into geoserver with no problem. However, geoserver has a bug in rendering wind vectors or barbs at global scale with wrapx so I cant use it. Is there something I need to configure to get ncWMS to work with the ECMWF files out of the box? Or some transform to get them to work? I can certainly post a file but they are 150MB in size.