Dataset not recognized by WCS

cpaulik commented 8 years ago

Hi,

I have a dataset (ERA-LAND modelled data). It was converted from a grib file using CDO and NCO. Putting it into Thredds it works fine with WMS and OpenDAP. When I try to access it via WCS I get the Backtrace at the bottom. It also does not have any <ContentMetadata/> in the WCS GetCapabilities.

I've tried adding some metadata fields that thredds might expect but it did not change anything. Could you please point me to documentation about which metadata attributes thredds expects for a dataset to work with WCS? According to the CF checker it is conform with the conventions.

The file can also be found here in a zip file.

Dataset details

dimensions:
    lat = 256 ;
    lon = 512 ;
    time = UNLIMITED ; // (1 currently)
variables:
    double lat(lat) ;
        lat:standard_name = "latitude" ;
        lat:long_name = "latitude" ;
        lat:units = "degrees_north" ;
        lat:_CoordinateAxisType = "Lat" ;
        lat:valid_range = -90.f, 90.f ;
    double lon(lon) ;
        lon:standard_name = "longitude" ;
        lon:long_name = "longitude" ;
        lon:units = "degrees_east" ;
        lon:_CoordinateAxisType = "Lon" ;
        lon:valid_range = -180.f, 180.f ;
    double time(time) ;
        time:standard_name = "time" ;
        time:units = "hours since 1984-1-2 18:00:00" ;
        time:calendar = "proleptic_gregorian" ;
        time:_CoordinateAxisType = "Time" ;
    float var39(time, lat, lon) ;
        var39:cell_methods = "depth: mean" ;
        var39:long_name = "soil moisture" ;
        var39:_CoordinateAxes = "time lat lon" ;

// global attributes:
        :CDI = "Climate Data Interface version 1.7.0 (http://mpimet.mpg.de/cdi)" ;
        :Conventions = "CF-1.4" ;
        :institution = "European Centre for Medium-Range Weather Forecasts" ;
        :CDO = "Climate Data Operators version 1.7.0 (http://mpimet.mpg.de/cdo)" ;
        :nco_openmp_thread_number = 1 ;
        :NCO = "4.4.2" ;
        :geospatial_lat_min = "-90.0" ;
        :geospatial_lat_max = "90.0" ;
        :geospatial_lon_min = "-180.0" ;
        :geospatial_lon_max = "180.0" ;
        :history = "Fri Apr 15 17:47:01 2016: ncatted -a cdm_data_type,global,o,c,Grid 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:45:50 2016: ncatted -a geospatial_lon_max,global,o,c,180.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:45:42 2016: ncatted -a geospatial_lon_min,global,o,c,-180.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:45:27 2016: ncatted -a geospatial_lat_max,global,o,c,90.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:45:19 2016: ncatted -a geospatial_lat_min,global,o,c,-90.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:43:15 2016: ncatted -a _CoordinateAxes,var39,o,c,time lat lon 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:42:20 2016: ncatted -a grid_type,var39,d,f,90.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:42:12 2016: ncatted -a table,var39,d,f,90.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:40:48 2016: ncatted -a valid_range,lat,a,f,90.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:40:43 2016: ncatted -a valid_range,lat,a,f,-90.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:40:36 2016: ncatted -a valid_range,lon,a,f,180.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:40:32 2016: ncatted -a valid_range,lon,o,f,-180.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:39:52 2016: ncatted -a valid_range,lon,a,f,90.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:39:48 2016: ncatted -a valid_range,lon,a,f,-90.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:39:26 2016: ncatted -a valid_range,lon,a,f,180.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:39:21 2016: ncatted -a valid_range,lon,a,f,-180.0 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:36:41 2016: ncatted -a long_name,var39,o,c,soil moisture 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:35:47 2016: ncatted -a axis,lon,d,c,lat lon time 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:35:40 2016: ncatted -a axis,lat,d,c,lat lon time 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:35:34 2016: ncatted -a axis,time,d,c,lat lon time 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:35:16 2016: ncatted -a _CoordinateAxes,var39,o,c,lat lon time 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:34:44 2016: ncatted -a _CoordinateAxisType,time,o,c,Time 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:34:33 2016: ncatted -a _CoordinateAxisType,lat,o,c,Lat 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:34:22 2016: ncatted -a _CoordinateAxisType,lon,o,c,Lon 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:14:31 2016: ncks -x -v depth,depth_bnds 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg.nc 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc\nFri Apr 15 17:08:45 2016: ncwa -a depth 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular.nc 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg.nc\nFri Apr 15 17:06:00 2016: cdo -f nc4 -z zip4 copy 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular.grb 39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular.nc" ;
        :cdm_data_type = "Grid" ;
}

Output of WCS GetCapabilities

<WCS_Capabilities xmlns="http://www.opengis.net/wcs" xmlns:gml="http://www.opengis.net/gml" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.0.0">
<Service>
<fees>NONE</fees>
<accessConstraints>NONE</accessConstraints>
</Service>
<Capability>
<Request>
<GetCapabilities>
<DCPType>
<HTTP>
<Get>
<OnlineResource xlink:href="http://cpa:8080/thredds/wcs/testAll/39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc"/>
</Get>
</HTTP>
</DCPType>
</GetCapabilities>
<DescribeCoverage>
<DCPType>
<HTTP>
<Get>
<OnlineResource xlink:href="http://cpa:8080/thredds/wcs/testAll/39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc"/>
</Get>
</HTTP>
</DCPType>
</DescribeCoverage>
<GetCoverage>
<DCPType>
<HTTP>
<Get>
<OnlineResource xlink:href="http://cpa:8080/thredds/wcs/testAll/39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc"/>
</Get>
</HTTP>
</DCPType>
</GetCoverage>
</Request>
<Exception>
<Format>application/vnd.ogc.se_xml</Format>
</Exception>
</Capability>
<ContentMetadata/>
</WCS_Capabilities>

Backtrace

2016-04-15T11:19:37.416 -0400 [   1057546][     190] INFO  - threddsServlet - Remote host: 10.0.2.2 - Request: "GET /thredds/wcs/testAll/39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc?service=WCS&version=1.0.0&request=DescribeCoverage HTTP/1.1"
2016-04-15T11:19:37.431 -0400 [   1057561][     190] ERROR - thredds.server.wcs.v1_0_0_1.WcsHandler - Unknown problem.
java.lang.IllegalArgumentException: Coverage list must contain at least one ID [0].
    at thredds.wcs.v1_0_0_1.DescribeCoverage.<init>(DescribeCoverage.java:77) ~[clcommon-4.6.4.jar:4.6.4]
    at thredds.server.wcs.v1_0_0_1.WcsRequestParser.parseRequest(WcsRequestParser.java:144) ~[classes/:4.6.4]
    at thredds.server.wcs.v1_0_0_1.WcsHandler.handleKVP(WcsHandler.java:97) [classes/:4.6.4]
    at thredds.server.wcs.WCSController.doGet(WCSController.java:224) [classes/:4.6.4]
    at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_73]
    at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_73]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) [spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) [spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:776) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) [servlet-api.jar:?]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [servlet-api.jar:?]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292) [catalina.jar:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.32]
    at thredds.servlet.filter.RequestQueryFilter.doFilter(RequestQueryFilter.java:118) [classes/:4.6.4]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.32]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.32]
    at thredds.servlet.filter.RequestCORSFilter.doFilterInternal(RequestCORSFilter.java:49) [classes/:4.6.4]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344) [spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261) [spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.32]
    at thredds.servlet.filter.RequestPathFilter.doFilter(RequestPathFilter.java:94) [classes/:4.6.4]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.32]
    at thredds.server.RequestBracketingLogMessageFilter.doFilter(RequestBracketingLogMessageFilter.java:81) [classes/:4.6.4]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.32]
    at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71) [log4j-web-2.2.jar:2.2]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.32]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) [catalina.jar:8.0.32]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [catalina.jar:8.0.32]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [catalina.jar:8.0.32]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [catalina.jar:8.0.32]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [catalina.jar:8.0.32]
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616) [catalina.jar:8.0.32]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [catalina.jar:8.0.32]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522) [catalina.jar:8.0.32]
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095) [tomcat-coyote.jar:8.0.32]
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672) [tomcat-coyote.jar:8.0.32]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500) [tomcat-coyote.jar:8.0.32]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456) [tomcat-coyote.jar:8.0.32]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_73]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_73]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.0.32]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_73]

kdruken commented 6 years ago

@cpaulik @lesserwhirls Did you ever resolve this issue? I'm seeing the same issue with the empty field with several datasets that I am helping to format...for life of me can't seem to line up attributes to get WCS to recognise the data contents, curious if you sorted out problem?

lesserwhirls commented 6 years ago

Greetings! I finally had a chance to investigate this one, and it looks like there are two issues at hand here.

The metadata for longitude specifies the valid_range as:
```
lon:valid_range = -180.f, 180.f ;
```
except the actual longitude values used in the file are from 0 - 360. When netCDF-Java opens this file, it interprets half of the longitude values and invalid. This can be fixed by using NcML to change the valid_range attribute.
The latitude variable does not appear to be "regular" (i.e. regularly spaced), or at least it's not passing our check for being regular (which is needed to use WCS). We do the following to check if the grid spacing is regular:
- compute the "expected" spacing given the range of the grid variable
- make sure no individual delta in values is different than the expected spacing by more than 0.005
The 0.005 tolerance is supposed to help account for errors in floating point math. For the example dataset attached to this ticket, the computed spacing is 0.701669188773156, but the range of individual deltas is from 8.38639E-05 to 0.00579897303795818, which is just outside of the tolerance level of 0.005. This threshold is only reached for the delta between the first and second values, and the delta between the second-to-last and last values.

I think this issue really comes down to how the latitude variable was computed, and how it was written to the netCDF file. I noticed that while lat and lon are of type double, their valid_range attributes are of type float, which makes me think something wasn't quite right when it was generated. Thankfully, this too can be addressed using NcML.

Here is what I did with the example file to make things happy:

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" 
    location="./39.128_RD_OPER_gbg4_AN_N128_19840102_1800_0-regular-avg-removed_depth.nc">
  <variable name="lon">
    <attribute name="valid_range" type="double" value="0 360" />
  </variable>
  <variable name="lat">
    <values start="-89.296875" increment="0.703125" />
  </variable>
</netcdf>

Just update the location attribute in the netcdf element and you should be good to go.

Unidata / thredds