Updates to GetFeatureInfo, time parameters, ehcache config, and CRS config

yosoyjay commented 7 years ago

Updates summarized as:

text/html output type for GetFeatureInfo requests, to support some web-base clients that assume this will exists
Support for time parameters using the ISO8601 + style format , ie. +00:00 and +08:00.
Move ehcache.xml config to a configurable file defined via JVM runtime arg. Previous defaults still used if ehcache.xml is not defined
Allow the customization of which CRS codes are advertised in the GetCapabilities document.
Changes to allow for datasest to be refreshed through ncWMS
Updates to README and development files (.dockerignore etc.)

kwilcox commented 7 years ago

@guygriffiths any chance for a merge? I'd like to base some builds off of the develop branch

guygriffiths commented 7 years ago

@kwilcox - Sorry, I thought I'd already done it...

guygriffiths commented 7 years ago

Just discovered that these changes cause issues when running multiple instances of edal-wms on the same Tomcat server (which I need to do fairly regularly). Any chance someone could have a look into this?

Stack trace:

SEVERE: Servlet [NcwmsApplicationServlet] in web application [/ncWMS2] threw load() exception
javax.management.InstanceAlreadyExistsException: net.sf.ehcache:type=CacheManager,name=EDAL-CacheManager
    at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
    at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
    at net.sf.ehcache.management.ManagementService.registerCacheManager(ManagementService.java:265)
    at net.sf.ehcache.management.ManagementService.init(ManagementService.java:233)
    at net.sf.ehcache.management.ManagementService.registerMBeans(ManagementService.java:193)
    at net.sf.ehcache.management.ManagementService.registerMBeans(ManagementService.java:218)
    at uk.ac.rdg.resc.edal.catalogue.DataCatalogue.<init>(DataCatalogue.java:200)

yosoyjay commented 7 years ago

Yeah, I'll take a took at it.

On Tue, Apr 25, 2017 at 8:19 AM, Guy Griffiths notifications@github.com wrote:

Just discovered that these changes cause issues when running multiple instances of edal-wms on the same Tomcat server (which I need to do fairly regularly). Any chance someone could have a look into this?

Stack trace:

SEVERE: Servlet [NcwmsApplicationServlet] in web application [/ncWMS2] threw load() exception javax.management.InstanceAlreadyExistsException: net.sf.ehcache:type=CacheManager,name=EDAL-CacheManager at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) at net.sf.ehcache.management.ManagementService.registerCacheManager(ManagementService.java:265) at net.sf.ehcache.management.ManagementService.init(ManagementService.java:233) at net.sf.ehcache.management.ManagementService.registerMBeans(ManagementService.java:193) at net.sf.ehcache.management.ManagementService.registerMBeans(ManagementService.java:218) at uk.ac.rdg.resc.edal.catalogue.DataCatalogue.(DataCatalogue.java:200)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Reading-eScience-Centre/edal-java/pull/83#issuecomment-297064906, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJ6N-a3-IDU4cuoZ8LscP9O0J4ziNQTks5rzg78gaJpZM4Mr3Rp .

yosoyjay commented 7 years ago

Would you mind detailing how exactly you run multiple edal-wms instances on a single Tomcat server?

I'm trying to test the proposed fix, but I'm encountering lots of issues in my test case using ncWMS.

On Tue, Apr 25, 2017 at 8:23 AM, Jesse Lopez yosoyjay@gmail.com wrote:

Yeah, I'll take a took at it.

On Tue, Apr 25, 2017 at 8:19 AM, Guy Griffiths notifications@github.com wrote:

Just discovered that these changes cause issues when running multiple instances of edal-wms on the same Tomcat server (which I need to do fairly regularly). Any chance someone could have a look into this?

Stack trace:

SEVERE: Servlet [NcwmsApplicationServlet] in web application [/ncWMS2] threw load() exception javax.management.InstanceAlreadyExistsException: net.sf.ehcache:type=CacheManager,name=EDAL-CacheManager at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) at net.sf.ehcache.management.ManagementService.registerCacheManager(ManagementService.java:265) at net.sf.ehcache.management.ManagementService.init(ManagementService.java:233) at net.sf.ehcache.management.ManagementService.registerMBeans(ManagementService.java:193) at net.sf.ehcache.management.ManagementService.registerMBeans(ManagementService.java:218) at uk.ac.rdg.resc.edal.catalogue.DataCatalogue.(DataCatalogue.java:200)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Reading-eScience-Centre/edal-java/pull/83#issuecomment-297064906, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJ6N-a3-IDU4cuoZ8LscP9O0J4ziNQTks5rzg78gaJpZM4Mr3Rp .

yosoyjay commented 7 years ago

I pushed the proposed changed here:

https://github.com/axiom-data-science/edal-java/commit/c6a13333947017c794ad2af4815c9d8723949055

On Tue, Apr 25, 2017 at 11:39 AM, Jesse Lopez yosoyjay@gmail.com wrote:

Would you mind detailing how exactly you run multiple edal-wms instances on a single Tomcat server?

I'm trying to test the proposed fix, but I'm encountering lots of issues in my test case using ncWMS.

On Tue, Apr 25, 2017 at 8:23 AM, Jesse Lopez yosoyjay@gmail.com wrote:

Yeah, I'll take a took at it.

On Tue, Apr 25, 2017 at 8:19 AM, Guy Griffiths notifications@github.com wrote:

Just discovered that these changes cause issues when running multiple instances of edal-wms on the same Tomcat server (which I need to do fairly regularly). Any chance someone could have a look into this?

Stack trace:

SEVERE: Servlet [NcwmsApplicationServlet] in web application [/ncWMS2] threw load() exception javax.management.InstanceAlreadyExistsException: net.sf.ehcache:type=CacheManager,name=EDAL-CacheManager at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) at net.sf.ehcache.management.ManagementService.registerCacheManager(ManagementService.java:265) at net.sf.ehcache.management.ManagementService.init(ManagementService.java:233) at net.sf.ehcache.management.ManagementService.registerMBeans(ManagementService.java:193) at net.sf.ehcache.management.ManagementService.registerMBeans(ManagementService.java:218) at uk.ac.rdg.resc.edal.catalogue.DataCatalogue.(DataCatalogue.java:200)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Reading-eScience-Centre/edal-java/pull/83#issuecomment-297064906, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJ6N-a3-IDU4cuoZ8LscP9O0J4ziNQTks5rzg78gaJpZM4Mr3Rp .

guygriffiths commented 7 years ago

@yosoyjay I often use edal-wms or even ncWMS as a basis for another custom application. Quite a few projects I work on need a data WMS + specific custom behaviour, so I use edal-wms in exactly the same way that ncWMS does, and add functionality on top of it.

Thanks for the changes, I'll take a look at them later today and see if it fixes the issue.

guygriffiths commented 7 years ago

@yosoyjay - I pulled those changes and it fixed the multiple instances in the same VM bug, but I'm still getting the same error if I undeploy then redeploy ncWMS in a running instance of Tomcat. I'm currently working with a group where I don't have the ability to restart Tomcat and this is causing difficulties.

yosoyjay commented 7 years ago

@guygriffiths Okay, I'll test that case and work on a fix.

yosoyjay commented 7 years ago

@guygriffiths I'm seeing a lot of warnings and errors when stopping or reloading an instance in Tomcat and it's unworkable.

I've found other applications that had similar problems with ehcache when reloading in Tomcat. The only solutions I'm finding so far involving forcing Tomcat to shutdown, so I'm inclined now to register the statistics reporting mBean as an option.

Have you run into the SldTemplateStyleCatalogue error?


11-May-2017 08:40:21.876 WARNING [http-nio-8080-exec-7] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ncWMS2] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
11-May-2017 08:40:21.877 WARNING [http-nio-8080-exec-7] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ncWMS2] appears to have started a thread named [EDAL-CacheManager] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.util.TimerThread.mainLoop(Timer.java:552)
 java.util.TimerThread.run(Timer.java:505)
11-May-2017 08:40:21.878 WARNING [http-nio-8080-exec-7] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ncWMS2] appears to have started a thread named [Statistics Thread-EDAL-CacheManager-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 java.lang.Thread.run(Thread.java:745)
11-May-2017 08:40:21.879 WARNING [http-nio-8080-exec-7] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ncWMS2] appears to have started a thread named [H2 File Lock Watchdog /Users/jesse/.ncWMS2/.h2/epsg.db.lock.db] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Thread.sleep(Native Method)
 org.h2.store.FileLock.run(FileLock.java:493)
 java.lang.Thread.run(Thread.java:745)
11-May-2017 08:40:21.880 WARNING [http-nio-8080-exec-7] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ncWMS2] appears to have started a thread named [H2 TCP Server (tcp://192.168.1.3:59148)] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.net.PlainSocketImpl.socketAccept(Native Method)
 java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
 java.net.ServerSocket.implAccept(ServerSocket.java:545)
 java.net.ServerSocket.accept(ServerSocket.java:513)
 org.h2.server.TcpServer.listen(TcpServer.java:244)
 org.h2.tools.Server.run(Server.java:549)
 java.lang.Thread.run(Thread.java:745)
11-May-2017 08:40:21.880 WARNING [http-nio-8080-exec-7] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ncWMS2] appears to have started a thread named [H2 Log Writer EPSG.DB] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 org.h2.store.WriterThread.run(WriterThread.java:105)
 java.lang.Thread.run(Thread.java:745)
11-May-2017 08:40:24.786 WARNING [http-nio-8080-exec-7] org.apache.catalina.startup.ContextConfig.validateSecurityRoles Security role name ncWMS-admin used in an <auth-constraint> without being defined in a <security-role>
11-May-2017 08:40:24.833 INFO [http-nio-8080-exec-7] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
log4j:WARN File option not set for appender [file].
log4j:WARN Are you using FileAppender instead of ConsoleAppender?
log4j:ERROR No output stream or file set for the appender named [file].
2017-05-11 08:40:25 ERROR SldTemplateStyleCatalogue:202 - Problem processing styles in edal-wms module
java.nio.file.FileSystemAlreadyExistsException
    at com.sun.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:113)
    at java.nio.file.FileSystems.newFileSystem(FileSystems.java:326)
    at java.nio.file.FileSystems.newFileSystem(FileSystems.java:276)
    at uk.ac.rdg.resc.edal.graphics.utils.SldTemplateStyleCatalogue.<init>(SldTemplateStyleCatalogue.java:175)
    at uk.ac.rdg.resc.edal.graphics.utils.SldTemplateStyleCatalogue.<clinit>(SldTemplateStyleCatalogue.java:107)
    at uk.ac.rdg.resc.edal.ncwms.NcwmsApplicationServlet.init(NcwmsApplicationServlet.java:216)
    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1227)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1140)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1027)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5038)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5348)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
    at org.apache.catalina.core.StandardContext.reload(StandardContext.java:3869)
    at org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:997)
    at org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:607)
    at org.apache.catalina.manager.HTMLManagerServlet.doPost(HTMLManagerServlet.java:211)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
    at org.apache.catalina.filters.CsrfPreventionFilter.doFilter(CsrfPreventionFilter.java:136)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
    at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:616)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:509)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1104)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1524)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1480)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
11-May-2017 08:40:25.667 INFO [http-nio-8080-exec-7] org.apache.catalina.core.StandardContext.reload Reloading Context with name [/ncWMS2] is completed
11-May-2017 08:40:25.947 INFO [H2 TCP Server (tcp://192.168.1.3:59148) thread] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.h2.mvstore.DataUtils]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
 java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.h2.mvstore.DataUtils]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1353)
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1341)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1206)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1167)
    at org.h2.value.Transfer.readBytes(Transfer.java:292)
    at org.h2.server.TcpServerThread.run(TcpServerThread.java:125)
    at java.lang.Thread.run(Thread.java:745)```

guygriffiths commented 7 years ago

Yes, I came across that one as well. A quick fix is to change line 177 of SldTemplateStyleCatalogue to:

try {
     fileSystem = FileSystems.newFileSystem(uri, Collections.<String, Object> emptyMap());
} catch (FileSystemAlreadyExistsException e) {
    fileSystem = FileSystems.getFileSystem(uri);
}

It gets rid of the error and the styles work, but I haven't tested how robust it is yet.

guygriffiths commented 7 years ago

Oops, just seen your other pull request - that's the danger with going through emails chronologically I guess!

yosoyjay commented 7 years ago

No problem!

Reading-eScience-Centre / edal-java

Updates to GetFeatureInfo, time parameters, ehcache config, and CRS config #83