search

aodn / aodn-portal

AODN Open Geospatial Portal
https://portal.aodn.org.au/
GNU General Public License v3.0
21 stars 13 forks source link

Change date on multi ncWMS collection throws error #2385

Closed kereid closed 7 years ago

kereid commented 7 years ago

On systest:

        App version: 4.27.0
        Build date: Wed Nov 02 09:41:17 AEDT 2016
        Build number: 80
        Git ref: 9d6330b9af9699b605b61b7a42e01251431bf545

Steps to reproduce:

Add cars collection ie Australian weekly. Proceed to layer tabe and change to a different product, go back to subset tab and change date.

What should happen:

Collection loads.

What does happen:

See red error triangle and sometimes collection only loads half of extent on the map.

kereid commented 7 years ago

Might not be just related to date change as intermittently getting error even when just adding collection

jonescc commented 7 years ago

Possibly an issue with thredds 4.6.6 we don't see with the version we are using. From threddsServlet.log on thredds-systest:

2016-11-04T14:24:17.946 +1100 [ 179664070][   38471] INFO  - threddsServlet - Remote host: 127.0.0.1 - Request: "GET /thredds/wms/CSIRO/Climatology/CARS/2009/AODN-product/CARS2009_Australia_weekly.nc?TIME=2009-1
2-25T23%3A04%3A36.923Z&BBOX=-69.2578125%2C133.2421875%2C-43.2421875%2C159.2578125&CRS=EPSG%3A4326&SERVICE=ncwms&HEIGHT=296&REQUEST=GetMap&STYLES=&WIDTH=296&EXCEPTIONS=application%2Fvnd.ogc.se_xml&TRANSPARENT=TRU
E&VERSION=1.3.0&FORMAT=image%2Fpng&QUERYABLE=true&LAYERS=SLC2 HTTP/1.1"
2016-11-04T14:24:17.955 +1100 [ 179664079][   38469] ERROR - thredds.server.wms.ThreddsWmsController - dispatchWmsRequest(): IOException: 
java.io.IOException: java.lang.IllegalStateException: DataBTree doesnt start with TREE
        at thredds.server.wms.ThreddsScalarLayer.readHorizontalPoints(ThreddsScalarLayer.java:176) ~[ThreddsScalarLayer.class:?]
        at uk.ac.rdg.resc.ncwms.controller.AbstractWmsController.readDataGrid(AbstractWmsController.java:1387) ~[ncwms-1.2.tds.4.6.6.jar:1.2.tds.4.6.6]
        at uk.ac.rdg.resc.ncwms.controller.AbstractWmsController.getMap(AbstractWmsController.java:532) ~[ncwms-1.2.tds.4.6.6.jar:1.2.tds.4.6.6]
        at thredds.server.wms.ThreddsWmsController.dispatchWmsRequest(ThreddsWmsController.java:179) [ThreddsWmsController.class:?]
        at uk.ac.rdg.resc.ncwms.controller.AbstractWmsController.handleRequestInternal(AbstractWmsController.java:200) [ncwms-1.2.tds.4.6.6.jar:1.2.tds.4.6.6]
        at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:146) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
ghost commented 7 years ago

Interesting... doesn't happen here: https://thredds-systest-eb.aodn.org.au/thredds/wms/CSIRO/Climatology/CARS/2009/AODN-product/CARS2009_Australia_weekly.nc?TIME=2009-12-25T23%3A04%3A36.923Z&BBOX=-69.2578125%2C133.2421875%2C-43.2421875%2C159.2578125&CRS=EPSG%3A4326&SERVICE=ncwms&HEIGHT=296&REQUEST=GetMap&STYLES=&WIDTH=296&EXCEPTIONS=application%2Fvnd.ogc.se_xml&TRANSPARENT=TRUE&VERSION=1.3.0&FORMAT=image%2Fpng&QUERYABLE=true&LAYERS=SLC2

jonescc commented 7 years ago

Problem is intermittent. Just flagging it here for @pblain to consider when considering whether or not to release this to production. Needs further investigation to work out whether this is the problem.

bhasin85 commented 7 years ago

Look like it is related to thredds issue #518

The fix is in commit

jonescc commented 7 years ago

Installed 4.6.7-snapshot (Thredds master branch) and tested locally in vagrant - still missing tiles occasionally although a different error is reported in the logs when this occurs (RAF cache off):

2016-11-09T15:54:50.501 +1100 [   8900983][    8954] ERROR - thredds.server.wms.ThreddsWmsController - dispatchWmsRequest(): IOException: 
java.io.IOException: java.lang.NullPointerException
        at thredds.server.wms.ThreddsScalarLayer.readHorizontalPoints(ThreddsScalarLayer.java:176) ~[ThreddsScalarLayer.class:?]
        at uk.ac.rdg.resc.ncwms.controller.AbstractWmsController.readDataGrid(AbstractWmsController.java:1387) ~[ncwms-1.2.tds.4.6.6.jar:1.2.tds.4.6.6]
        at uk.ac.rdg.resc.ncwms.controller.AbstractWmsController.getMap(AbstractWmsController.java:532) ~[ncwms-1.2.tds.4.6.6.jar:1.2.tds.4.6.6]
        at thredds.server.wms.ThreddsWmsController.dispatchWmsRequest(ThreddsWmsController.java:179) [ThreddsWmsController.class:?]
        at uk.ac.rdg.resc.ncwms.controller.AbstractWmsController.handleRequestInternal(AbstractWmsController.java:200) [ncwms-1.2.tds.4.6.6.jar:1.2.tds.4.6.6]
        at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:146) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) [servlet-api.jar:?]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) [servlet-api.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) [catalina.jar:7.0.61]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.61]
        at thredds.servlet.filter.RequestQueryFilter.doFilter(RequestQueryFilter.java:118) [RequestQueryFilter.class:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.61]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.61]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat7-websocket.jar:7.0.61]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.61]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.61]
        at thredds.servlet.filter.RequestCORSFilter.doFilterInternal(RequestCORSFilter.java:49) [RequestCORSFilter.class:?]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344) [spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261) [spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.61]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.61]
        at thredds.servlet.filter.RequestPathFilter.doFilter(RequestPathFilter.java:94) [RequestPathFilter.class:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.61]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.61]
        at thredds.server.RequestBracketingLogMessageFilter.doFilter(RequestBracketingLogMessageFilter.java:81) [RequestBracketingLogMessageFilter.class:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.61]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.61]
        at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71) [log4j-web-2.2.jar:2.2]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.61]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.61]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) [catalina.jar:7.0.61]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) [catalina.jar:7.0.61]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) [catalina.jar:7.0.61]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) [catalina.jar:7.0.61]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) [catalina.jar:7.0.61]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [catalina.jar:7.0.61]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) [catalina.jar:7.0.61]
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079) [tomcat-coyote.jar:7.0.61]
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620) [tomcat-coyote.jar:7.0.61]
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) [tomcat-coyote.jar:7.0.61]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_91]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_91]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-coyote.jar:7.0.61]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_91]
Caused by: java.lang.NullPointerException
jonescc commented 7 years ago

Also getting:

2016-11-09T16:02:35.020 +1100 [   9365502][        ] ERROR - threddsServlet - doFilter(): Yikes! Response is already committed (Heiko's bug?).

Don't get either of these errors or occasional missing tiles in production.

jonescc commented 7 years ago

Same errors occur using our current production thredds version when RAF is off.

jonescc commented 7 years ago

Looks like fix above should resolve the 'DataBTree doesnt start with TREE' issue - thanks @bhasin85. Pull request to upgrade to version of thredds 4.6.x including these fixes at https://github.com/aodn/thredds/pull/8

There still looks to be an issue with some tiles not being able to be rendered which applies to both production and the latest 4.6 branch.