search

Reading-eScience-Centre / ncwms

ncWMS - A Web Map Service for displaying environmental data over the web
Other
62 stars 30 forks source link

Problem when using Dynamic Services #76

Closed nuds-zhao closed 3 years ago

nuds-zhao commented 3 years ago

Hi, I'm not sure where to ask the question. I try to publish a netCDF data using ncWMS, and it works when I publish it using static way. But when it's in a large collection of datasets and published using Dynamic Services, it returns a error:

_<ServiceExceptionReport version="1.1.1" xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wms/1.1.1/exceptions_1_1_1.xsd">

The layer LDASOUT/M_2020120101.LDASOUT_DOMAIN1/SOIL_M was not found on this server

_

But I'm sure the data is there. So I checked the server log:

uk.ac.rdg.resc.edal.exceptions.EdalException: Problem creating dataset LDASOUT/M_2020120101.LDASOUT_DOMAIN1 at /home/hzhao/data/output/analysis//M_2020120101.LDASOUT_DOMAIN1 at uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory.createDataset(CdmDatasetFactory.java:115) at uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory.createDataset(CdmDatasetFactory.java:72) at uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory.createDataset(CdmDatasetFactory.java:1) 2020-12-16 21:04:50 WARN WmsServlet:2639 - Wms Exception caught: "The layer LDASOUT/M_2020120101.LDASOUT_DOMAIN1/SOIL_M was not found on this server" from:uk.ac.rdg.resc.edal.wms.util.WmsUtils:242 at uk.ac.rdg.resc.edal.ncwms.NcwmsCatalogue.getDatasetFromId(NcwmsCatalogue.java:228) at uk.ac.rdg.resc.edal.wms.util.WmsUtils.getVariableMetadataFromLayerName(WmsUtils.java:240) at uk.ac.rdg.resc.edal.wms.util.WmsUtils.getLayerMetadata(WmsUtils.java:284) at uk.ac.rdg.resc.edal.wms.GetMapStyleParams.(GetMapStyleParams.java:182) at uk.ac.rdg.resc.edal.wms.GetMapParameters.(GetMapParameters.java:97) at uk.ac.rdg.resc.edal.wms.WmsServlet.getMap(WmsServlet.java:413) at uk.ac.rdg.resc.edal.wms.WmsServlet.dispatchWmsRequest(WmsServlet.java:368) at uk.ac.rdg.resc.edal.ncwms.NcwmsServlet.dispatchWmsRequest(NcwmsServlet.java:212) at uk.ac.rdg.resc.edal.wms.WmsServlet.doGet(WmsServlet.java:303) at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:156) at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:233) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalArgumentException: Coordinate values must increase or decrease monotonically at uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.checkAscending(AbstractIrregularAxis.java:114) at uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.init(AbstractIrregularAxis.java:103) at uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.(AbstractIrregularAxis.java:64) at uk.ac.rdg.resc.edal.grid.ReferenceableAxisImpl.(ReferenceableAxisImpl.java:66) at uk.ac.rdg.resc.edal.util.cdm.CdmUtils.createReferenceableAxis(CdmUtils.java:417) at uk.ac.rdg.resc.edal.util.cdm.CdmUtils.createReferenceableAxis(CdmUtils.java:350) at uk.ac.rdg.resc.edal.util.cdm.CdmUtils.createHorizontalGrid(CdmUtils.java:205) at uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.getNonStaggeredGriddedVariableMetadata(CdmGridDatasetFactory.java:148) at uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.generateGridDataset(CdmGridDatasetFactory.java:134) at uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.generateDataset(CdmGridDatasetFactory.java:124) at uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory.createDataset(CdmDatasetFactory.java:88) ... 39 more

Seems EDAL gives an error that "Coordinate values must increase or decrease monotonically", then I go check the coordinate values, which do increase monotonically. So far I don't know what's the real problem with this, and hope you can give some hits to me. Thanks!

Btw, other datasets under the same folder share the same coordinate values and they work just good.

guygriffiths commented 3 years ago

Interesting, do you have a copy of the data I could download to take a look at?

nuds-zhao commented 3 years ago

M_2020120404.zip

Hi, Here is one sample dataset.

guygriffiths commented 3 years ago

Thanks for that, but if I load it in as a dynamic dataset it works fine for both the SOIL_M and QFX layers. Is this definitely a problematic file? Are you able to make 2020120101.LDASOUT_DOMAIN1 (the file in the stack trace) available?

nuds-zhao commented 3 years ago

M_2020120101.zip

Sure. Why I didn't give you this file is in the next day I tried to load the file and it worked but the one I gave you had the same problem. I know it seems too werid to be true, but this is the problem I'm facing. There are always some of these datasedts can not be loaded on my map app. This time both files work fine, so I also provide another problematic file here together with its error log:

M_2020120501.zip

2020-12-18 19:55:06 WARN WmsServlet:2639 - Wms Exception caught: "The layer LDASOUT/M_2020120501.LDASOUT_DOMAIN1/SOIL_M was not found on this server" from:uk.ac.rdg.resc.edal.wms.util.WmsUtils:242 uk.ac.rdg.resc.edal.exceptions.EdalException: Problem creating dataset LDASOUT/M_2020120501.LDASOUT_DOMAIN1 at /home/hzhao/data/output/analysis//M_2020120501.LDASOUT_DOMAIN1 at uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory.createDataset(CdmDatasetFactory.java:115) at uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory.createDataset(CdmDatasetFactory.java:72) at uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory.createDataset(CdmDatasetFactory.java:1) at uk.ac.rdg.resc.edal.ncwms.NcwmsCatalogue.getDatasetFromId(NcwmsCatalogue.java:228) at uk.ac.rdg.resc.edal.wms.util.WmsUtils.getVariableMetadataFromLayerName(WmsUtils.java:240) at uk.ac.rdg.resc.edal.wms.util.WmsUtils.getLayerMetadata(WmsUtils.java:284) at uk.ac.rdg.resc.edal.wms.GetMapStyleParams.(GetMapStyleParams.java:182) at uk.ac.rdg.resc.edal.wms.GetMapParameters.(GetMapParameters.java:97) at uk.ac.rdg.resc.edal.wms.WmsServlet.getMap(WmsServlet.java:413) at uk.ac.rdg.resc.edal.wms.WmsServlet.dispatchWmsRequest(WmsServlet.java:368) at uk.ac.rdg.resc.edal.ncwms.NcwmsServlet.dispatchWmsRequest(NcwmsServlet.java:212) at uk.ac.rdg.resc.edal.wms.WmsServlet.doGet(WmsServlet.java:303) at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:156) at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:233) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalArgumentException: Coordinate values must increase or decrease monotonically at uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.checkAscending(AbstractIrregularAxis.java:114) at uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.init(AbstractIrregularAxis.java:103) at uk.ac.rdg.resc.edal.grid.AbstractIrregularAxis.(AbstractIrregularAxis.java:64) at uk.ac.rdg.resc.edal.grid.ReferenceableAxisImpl.(ReferenceableAxisImpl.java:66) at uk.ac.rdg.resc.edal.util.cdm.CdmUtils.createReferenceableAxis(CdmUtils.java:417) at uk.ac.rdg.resc.edal.util.cdm.CdmUtils.createReferenceableAxis(CdmUtils.java:350) at uk.ac.rdg.resc.edal.util.cdm.CdmUtils.createHorizontalGrid(CdmUtils.java:206) at uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.getNonStaggeredGriddedVariableMetadata(CdmGridDatasetFactory.java:148) at uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.generateGridDataset(CdmGridDatasetFactory.java:134) at uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.generateDataset(CdmGridDatasetFactory.java:124) at uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory.createDataset(CdmDatasetFactory.java:88) ... 39 more 2020-12-18 19:55:06 WARN WmsServlet:2639 - Wms Exception caught: "The layer LDASOUT/M_2020120501.LDASOUT_DOMAIN1/SOIL_M was not found on this server" from:uk.ac.rdg.resc.edal.wms.util.WmsUtils:242

guygriffiths commented 3 years ago

That's rather unusual. I wonder if it could be related to the way dynamic dataset caching works. There was a bug about this which I recently fixed (https://github.com/Reading-eScience-Centre/ncwms/issues/51), but that fix has not had a full release yet. Can you try this version and see if it fixes your issue: https://github.com/Reading-eScience-Centre/ncwms/releases/tag/1.5.1-SNAPSHOT-dynamic-dataset-fix

Hopefully that's what's causing this issue, if not let me know and I will look into it next year when I return from leave.

nuds-zhao commented 3 years ago

Hi,

Hope you had a great holiday!

I tryied the developer version, but it's not working. Same error appears in both static and dynamic publishing:

uk.ac.rdg.resc.edal.exceptions.EdalException: Problem creating dataset LDASOUT/M_2021010420.LDASOUT_DOMAIN1 at /home/hzhao/data/output/analysis//M_2021010420.LDASOUT_DOMAIN1 at uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory.createDataset(CdmDatasetFactory.java:133) 2021-01-05 01:51:38 WARN WmsServlet:2639 - Wms Exception caught: "The layer LDASOUT/M_2021010420.LDASOUT_DOMAIN1/SOIL_M was not found on this server" from:uk.ac.rdg.resc.edal.wms.util.WmsUtils:242 at uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory.createDataset(CdmDatasetFactory.java:81) at uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory.createDataset(CdmDatasetFactory.java:75) at uk.ac.rdg.resc.edal.ncwms.NcwmsCatalogue.getDatasetFromId(NcwmsCatalogue.java:222) at uk.ac.rdg.resc.edal.wms.util.WmsUtils.getVariableMetadataFromLayerName(WmsUtils.java:240) at uk.ac.rdg.resc.edal.wms.util.WmsUtils.getLayerMetadata(WmsUtils.java:284) at uk.ac.rdg.resc.edal.wms.GetMapStyleParams.(GetMapStyleParams.java:182) at uk.ac.rdg.resc.edal.wms.GetMapParameters.(GetMapParameters.java:97) at uk.ac.rdg.resc.edal.wms.WmsServlet.getMap(WmsServlet.java:413) at uk.ac.rdg.resc.edal.wms.WmsServlet.dispatchWmsRequest(WmsServlet.java:368) at uk.ac.rdg.resc.edal.ncwms.NcwmsServlet.dispatchWmsRequest(NcwmsServlet.java:212) at uk.ac.rdg.resc.edal.wms.WmsServlet.doGet(WmsServlet.java:303) at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:156) at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:233) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.util.ServiceConfigurationError: org.opengis.referencing.operation.CoordinateOperationFactory: Provider org.geotoolkit.referencing.operation.CachingCoordinateOperationFactory could not be instantiated at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:581) at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:803) at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:721) at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1394) at org.apache.sis.internal.system.DefaultFactories.forClass(DefaultFactories.java:94) at org.apache.sis.internal.system.DefaultFactories.forBuildin(DefaultFactories.java:136) at org.apache.sis.referencing.factory.sql.EPSGFactory.factory(EPSGFactory.java:291) at org.apache.sis.referencing.factory.sql.EPSGFactory.(EPSGFactory.java:281) at org.apache.sis.referencing.AuthorityFactories.EPSG(AuthorityFactories.java:122) at org.apache.sis.referencing.CommonCRS.factory(CommonCRS.java:1815) at org.apache.sis.referencing.CommonCRS.geographic(CommonCRS.java:635) at org.apache.sis.referencing.CommonCRS.normalizedGeographic(CommonCRS.java:592) at org.apache.sis.referencing.CommonCRS.defaultGeographic(CommonCRS.java:560) at uk.ac.rdg.resc.edal.util.GISUtils.defaultGeographicCRS(GISUtils.java:122) at uk.ac.rdg.resc.edal.grid.cdm.CdmTransformedGrid.(CdmTransformedGrid.java:163) at uk.ac.rdg.resc.edal.util.cdm.CdmUtils.createHorizontalGrid(CdmUtils.java:224) at uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.getNonStaggeredGriddedVariableMetadata(CdmGridDatasetFactory.java:148) at uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.generateGridDataset(CdmGridDatasetFactory.java:134) at uk.ac.rdg.resc.edal.dataset.cdm.CdmGridDatasetFactory.generateDataset(CdmGridDatasetFactory.java:124) at uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory.createDataset(CdmDatasetFactory.java:97) ... 39 more Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.geotoolkit.referencing.operation.CachingCoordinateOperationFactory at jdk.internal.reflect.GeneratedConstructorAccessor20.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:779) ... 57 more

nuds-zhao commented 3 years ago

Sorry, I just rebuilt the server with the edal-examples dependency enabled. It works.

And the loading failure disappeared. It seems the bug you fixed is the real reason causing the issue.

guygriffiths commented 3 years ago

Great, glad to hear it's working

nuds-zhao commented 3 years ago

Sorry to comment here again, but I encounter the same problem today. This time the sever log only displays the very facile error: " The layer LDASOUT/M_2021011217.LDASOUT_DOMAIN1/SOIL_M was not found on this server".

I think the peoblem may still be there. Seems the developer version only solved part of the issue. By the way, among those layers which can be loaded successfully, there are still some of them failed to load the first several tiles. I guess this might result from the same bug.

Could you please look into this issue again? Thanks!