NCWMS2 GetCapabilities returns java.lang.IllegalArgumentException: Coordinate values must increase or decrease monotonically

[Renaud009]

NCWMS2 retourne ces erreurs pour plusieurs datasets présents sur les oiseaux: java.lang.IllegalArgumentException: Coordinate values must increase or decrease monotonically uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.checkAscending(AbstractIrregularAxis.java:113) uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.init(AbstractIrregularAxis.java:102) uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.<init>(AbstractIrregularAxis.java:63) uk.ac.rdg.resc.edal.grid.TimeAxisImpl.<init>(TimeAxisImpl.java:51) uk.ac.rdg.resc.edal.util.cdm.CdmUtils.createTimeAxis(CdmUtils.java:296) uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.createDataset(CdmGridDatasetFactory.java:213) uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.createDataset(CdmGridDatasetFactory.java:82) uk.ac.rdg.resc.edal.ncwms.NcwmsCatalogue.getDatasetFromId(NcwmsCatalogue.java:170) uk.ac.rdg.resc.edal.wms.WmsServlet.getCapabilities(WmsServlet.java:538) uk.ac.rdg.resc.edal.wms.WmsServlet.dispatchWmsRequest(WmsServlet.java:311) uk.ac.rdg.resc.edal.ncwms.NcwmsServlet.dispatchWmsRequest(NcwmsServlet.java:176) uk.ac.rdg.resc.edal.wms.WmsServlet.doGet(WmsServlet.java:245) javax.servlet.http.HttpServlet.service(HttpServlet.java:624) javax.servlet.http.HttpServlet.service(HttpServlet.java:731) com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:168) com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:233) org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

Exemples: https://outarde.crim.ca/twitcher/ows/proxy/ncWMS2/wms?SERVICE=WMS&REQUEST=GetCapabilities&VERSION=1.3.0&DATASET=outputs/CMIP5/CCCMA/CanESM2/historical/day/atmos/r1i1p1/tasmax/tasmax_day_CanESM2_historical_r1i1p1_19790101-20051231.nc

https://outarde.crim.ca/twitcher/ows/proxy/ncWMS2/wms?SERVICE=WMS&REQUEST=GetCapabilities&VERSION=1.3.0&DATASET=outputs/CMIP5/CCCMA/CanESM2/rcp85/day/atmos/r1i1p1/pr/pr_day_CanESM2_rcp85_r1i1p1_20060101-21001231.nc

[tomLandry]

@cehbrecht : I think we had this error in the past and Blaise (not with us anymore) solved it. I vaguely recall this to be linked to broken data. Seen that before?

[tomLandry]

NcWMS source code

/**

• Checks that the axis values ascend or descend
• monotonically, throwing an IllegalArgumentException if not. */ private void checkAscending() { double prevVal = axisValues[0]; for (int i = 1; i < this.axisValues.length; i++) { if (this.axisValues[i] <= prevVal) { throw new IllegalArgumentException("Coordinate values must increase or decrease monotonically"); } prevVal = this.axisValues[i]; } }

[tomLandry]

Good read on GitHub on this very problem right here: https://github.com/Reading-eScience-Centre/edal-java/issues/26

[tomLandry]

guygriffiths, 14 sept 2015: "Fixed in develop branch. EDAL was ordering the files by the values of their time axes, but was reading the time values as longs. When these were all on the same day, no reordering was applied. Fixed by reading time values as floats."

NcWMS is now at 2.4.1 and we're at 2.0.4. I suggest a version upgrade on one of the smaller birds. @dbyrns how difficult would that be?

[dbyrns]

Near impossible... @fmigneault try to rebuild a docker image in the past and failed. We could try it again but I highly doubt that we could achieve this for the demo.

[Zeitsperre]

As was mentioned in the standup meeting, for reference, the 'values not monotonically increasing' error will happen if any Dimensions in a NetCDF that are float values are not read with enough precision (ie: -179.1, -179.0, -178.9 is good; -179, -179, -178 is bad).

[tomLandry]

Then @Zeitsperre we have to massage the data so NcWMS won't complain it seems.

[cehbrecht]

I haven't looked at NcWMS for quite a while. I can update the conda package and the Dockerfile in birdhouse. But if you only need the docker image maybe there is a better maintained one available on dockerhub.

[dbyrns]

I agree with you @cehbrecht we should switch to the latest on dockerhub but then we will still need to setup the configuration so that it acts like the one you package for birdhouse in the past.

[tomLandry]

Ok, thanks for input Carsten. This Docker image is built by the same guy, green from a couple of minutes ago. https://hub.docker.com/r/guygriffiths/ncwms/builds/

[cehbrecht]

In the long run I would prefer to use "officially" maintained docker images. But, yes, configuration can be an issue ... this is probably different for each maintainer.

I will look in updating ncwms in birdhouse (conda, docker). You can choose which one works best for you :)

[dbyrns]

I'm looking into what should be done with the official image so that data are fetch and serve like the old one. It may be pretty easy...

[tomLandry]

@Zeitsperre : maintainer from NcWMS is ready to help. He would need our seemingly broken data to reproduce. Maybe an OpenDAP link or THREDDS?

[Zeitsperre]

Sure. The URL for that exact file directly from ESGF-CoG is as follows:

http://esgfcog.cccma.ec.gc.ca/thredds/catalog/esgcet/12/cmip5.output.CCCma.CanESM2.rcp85.day.atmos.r1i1p1.v20130331.html?dataset=cmip5.output.CCCma.CanESM2.rcp85.day.atmos.r1i1p1.v20130331.tasmax_day_CanESM2_rcp85_r1i1p1_20060101-21001231.nc

In order to download it, they need to register a free account for research/education purposes.

[tomLandry]

Following our discussion, here is the file from our OPEnDAP server

Catalog http://outarde.crim.ca:8083/thredds/catalog/birdhouse/CMIP5/CCCMA/CanESM2/rcp85/day/atmos/r1i1p1/tasmax/catalog.html Dataset: tasmax/tasmax_day_CanESM2_rcp85_r1i1p1_20060101-21001231.nc Data size: 1.137 Gbytes ID: birdhouse/CMIP5/CCCMA/CanESM2/rcp85/day/atmos/r1i1p1/tasmax/tasmax_day_CanESM2_rcp85_r1i1p1_20060101-21001231.nc Access: HTTPServer: /twitcher/ows/proxy/thredds/fileServer/birdhouse/CMIP5/CCCMA/CanESM2/rcp85/day/atmos/r1i1p1/tasmax/tasmax_day_CanESM2_rcp85_r1i1p1_20060101-21001231.nc OPENDAP: /twitcher/ows/proxy/thredds/dodsC/birdhouse/CMIP5/CCCMA/CanESM2/rcp85/day/atmos/r1i1p1/tasmax/tasmax_day_CanESM2_rcp85_r1i1p1_20060101-21001231.nc Dates: 2018-05-08T19:25:50Z (modified) Viewers: NetCDF-Java ToolsUI (webstart)

[Zeitsperre]

Just copied over that exact file from crim-internal server. Visualizing it raises no errors.

[cehbrecht]

There is an attempt of community maintained docker images: http://geocontainers.org/

But looks outdated ...

[tomLandry]

Updated in Birdhouse. Thanks Carsten ! https://github.com/bird-house/pyramid-phoenix/issues/156#event-1627753824

[cehbrecht]

... well not really ... updated the conda ncwms2 package and the phoenix buildout installation. Not sure if the default ncwms config is still working ... it least it shows a map. I currently want to update the ncwms docker image ... but these are all outdated.

Need to have better maintained images: https://github.com/bird-house/birdhouse-docker-images/issues/1 https://hub.docker.com/r/birdhouse/ncwms2/

[dbyrns]

I look at the latest ncwms2 image : https://hub.docker.com/r/guygriffiths/ncwms/ And I think that all we have to do is mount the following files into it : https://github.com/bird-house/birdhousebuilder.recipe.ncwms/blob/master/birdhousebuilder/recipe/ncwms/config.xml to /usr/local/tomcat/.ncWMS2/config.xml

https://github.com/bird-house/birdhousebuilder.recipe.ncwms/blob/master/birdhousebuilder/recipe/ncwms/ncWMS2.xml to /usr/local/tomcat/conf/Catalina/localhost/ncWMS.xml

https://github.com/Ouranosinc/PAVICS/blob/master/birdhouse/config/ncwms2/server.xml to /usr/local/tomcat/conf/server.xml

Template values for PAVICS are almost all the default ones specify here : https://github.com/bird-house/birdhousebuilder.recipe.ncwms We only override the ncwms_password and the data_dir data_dir have been replaced in the template by dynamic-services and should look like this : "dynamic-services = outputs=/pavics-data"

[cehbrecht]

I tried to update the ncwms image using the buildout: https://hub.docker.com/r/birdhouse/ncwms2/

But it is still not working :(

If you can use the docker image by guygriffiths it would be the better choice.

My ncwms Dockerfile is living here: https://github.com/bird-house/birdhouse-docker-images/tree/master/ncwms2

Would be nice to have community maintained images for those services we just use and configure: https://github.com/bird-house/birdhouse-docker-images/issues/1

[Zvax]

I downloaded the tasmin_day_CanESM2_historical_r1i1p1_19790101-20051231.nc dataset from outarde.

The lat lon values are monotonical, even when cast to integers (there are jumps of 3 between values). The time values however, have drastic jumps:

    54476.5, 54477.5, 54478.5, 54479.5, 54480.5, 54481.5, 54482.5, 54483.5, 
    54484.5, 54485.5, 54486.5, 54487.5, 54488.5, 54489.5, 54490.5, 54491.5, 
    54492.5, 54493.5, 54494.5, 54495.5, 54496.5, 54497.5, 54498.5, 54499.5, 
    54500.5, 54501.5, 54502.5, 54503.5, 3.99806390033924e-47, 
    2.88969763368863e-38, 3.61181940230661e-35, 1.16677822260747e-33, 
    4.80531465108474e-37, 3.83547555141028e-44, 1.66790546265913e-34, 
    1.16026731006209e-34, 5.70793526006812e-35, 8.45809382859479e-37, 

The jump from "54503.5" to "3.99806390033924e-47" would trigger the exception of non monotonical values: 54502.5 to 54503.5 is ascending, 54503.5 to 3.99806390033924e-47 looks descending.

Just in case I missed something: here are the lat lon axis:

 lat = -87.8638013437108, -85.0965294927955, -82.3129154523645, 
    -79.5256090771376, -76.7369021848465, -73.9475176584678, 
    -71.1577545160655, -68.3677586127913, -65.577609515306, 
    -62.7873543034412, -59.9970226129695, -57.2066340321214, 
    -54.4162020305644, -51.6257361794164, -48.8352434707287, 
    -46.0447291355798, -43.2541971698291, -40.4636506825932, 
    -37.6730921335235, -34.8825234982516, -32.0919463862222, 
    -29.3013621262409, -26.5107718296892, -23.7201764380129, 
    -20.9295767589677, -18.1389734947175, -15.3483672639697, 
    -12.5577586197089, -9.76714806367372, -6.97653605842678, 
    -4.18592303766731, -1.39530941529766, 1.39530941529766, 4.18592303766731, 
    6.97653605842678, 9.76714806367372, 12.5577586197089, 15.3483672639697, 
    18.1389734947175, 20.9295767589677, 23.7201764380129, 26.5107718296892, 
    29.3013621262409, 32.0919463862222, 34.8825234982516, 37.6730921335235, 
    40.4636506825932, 43.2541971698291, 46.0447291355798, 48.8352434707287, 
    51.6257361794164, 54.4162020305644, 57.2066340321214, 59.9970226129695, 
    62.7873543034412, 65.577609515306, 68.3677586127913, 71.1577545160655, 
    73.9475176584678, 76.7369021848465, 79.5256090771376, 82.3129154523645, 
    85.0965294927955, 87.8638013437108 ;

 lon = 0, 2.8125, 5.625, 8.4375, 11.25, 14.0625, 16.875, 19.6875, 22.5, 
    25.3125, 28.125, 30.9375, 33.75, 36.5625, 39.375, 42.1875, 45, 47.8125, 
    50.625, 53.4375, 56.25, 59.0625, 61.875, 64.6875, 67.5, 70.3125, 73.125, 
    75.9375, 78.75, 81.5625, 84.375, 87.1875, 90, 92.8125, 95.625, 98.4375, 
    101.25, 104.0625, 106.875, 109.6875, 112.5, 115.3125, 118.125, 120.9375, 
    123.75, 126.5625, 129.375, 132.1875, 135, 137.8125, 140.625, 143.4375, 
    146.25, 149.0625, 151.875, 154.6875, 157.5, 160.3125, 163.125, 165.9375, 
    168.75, 171.5625, 174.375, 177.1875, 180, 182.8125, 185.625, 188.4375, 
    191.25, 194.0625, 196.875, 199.6875, 202.5, 205.3125, 208.125, 210.9375, 
    213.75, 216.5625, 219.375, 222.1875, 225, 227.8125, 230.625, 233.4375, 
    236.25, 239.0625, 241.875, 244.6875, 247.5, 250.3125, 253.125, 255.9375, 
    258.75, 261.5625, 264.375, 267.1875, 270, 272.8125, 275.625, 278.4375, 
    281.25, 284.0625, 286.875, 289.6875, 292.5, 295.3125, 298.125, 300.9375, 
    303.75, 306.5625, 309.375, 312.1875, 315, 317.8125, 320.625, 323.4375, 
    326.25, 329.0625, 331.875, 334.6875, 337.5, 340.3125, 343.125, 345.9375, 
    348.75, 351.5625, 354.375, 357.1875 ;

[Zeitsperre]

@Zvax That doesn't look right at all. I just downloaded and checked the 'time' and I can see the error on my end as well. Looking at ncdump, it seems as though this came directly from CCCMA and history shows nothing notable, so I have no idea how/when the time dimension got corrupted. I say disregard this file and I'll add a new/different version.

[ldperron]

I'm replacing all CCCMA files on Outarde. Almost all of them has corruption.

[ldperron]

CCCMA done on Outarde, hirondelle and pluvier.

[ldperron]

And colibri

[Zeitsperre]

@huard AFAIK, no problems to be found with the CMIP5 data at Ouranos. Not sure how this corruption happened. Should I copy over another experiment?

[huard]

No, let's focus on the demo.

Are you able to browse the geoserver shapefiles from QGIS ?

On Thu, May 17, 2018 at 4:00 PM Trevor James Smith notifications@github.com wrote:

@huard https://github.com/huard AFAIK, no problems to be found with the CMIP5 data at Ouranos. Not sure how this corruption happened. Should I copy over another experiment?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/Ouranosinc/pavics-sdi/issues/55#issuecomment-389990922, or mute the thread https://github.com/notifications/unsubscribe-auth/AAE9QyguBEjIV0oRZ1UEBLxl8Wft1IwLks5tzdbqgaJpZM4T_sks .

[Zeitsperre]

[dbyrns]

It's maybe unrelated, but at least a recent version of ncwms is now available : https://github.com/Ouranosinc/PAVICS/pull/82 @cehbrecht you can take a look if you want... it's not well packaged like what you did (3 static files mount by the docker-compose) but at least we are able to use the latest ncwms version the same way as before.

[tomLandry]

@dbyrns, can we close this? I get from comments that this problem is solved, at least on our end.