search

opengeospatial / ets-security-client10

Public Repo for D112 Secure Client Tests
Other
1 stars 1 forks source link

Servlet-API for Jetty conflicts with older version in Tomcat #6

Closed openfirmware closed 6 years ago

openfirmware commented 6 years ago

I created a local fork of teamengine-docker and added a sub-project for this test suite, and this has made it easier for me to deploy and test changes. A recent change in my commits caused the test session in TEAM Engine to fail after starting the test session; this only occurs with TEAM Engine and not when running the test suite under an IDE or as an executable JAR.

The console running the Docker container with TEAM Engine 5.3 has some clues:

INFO: Starting Servlet Engine: Apache Tomcat/7.0.88
Jul 24, 2018 9:23:19 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /usr/local/tomcat/webapps/examples
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /usr/local/tomcat/webapps/examples has finished in 391 ms
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /usr/local/tomcat/webapps/ROOT
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /usr/local/tomcat/webapps/ROOT has finished in 163 ms
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /usr/local/tomcat/webapps/docs
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /usr/local/tomcat/webapps/docs has finished in 182 ms
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /usr/local/tomcat/webapps/host-manager
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /usr/local/tomcat/webapps/host-manager has finished in 160 ms
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /usr/local/tomcat/webapps/manager
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /usr/local/tomcat/webapps/manager has finished in 170 ms
Jul 24, 2018 9:23:20 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /usr/local/tomcat/webapps/teamengine
Jul 24, 2018 9:23:20 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/usr/local/tomcat/webapps/teamengine/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Jul 24, 2018 9:23:22 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /usr/local/tomcat/webapps/teamengine has finished in 1,763 ms
Jul 24, 2018 9:23:22 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-apr-8080"]
Jul 24, 2018 9:23:22 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-apr-8009"]
Jul 24, 2018 9:23:22 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 2856 ms
Jul 24, 2018 9:24:10 PM org.apache.jasper.compiler.TldLocationsCache tldScanJar
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Fatal Error: Error in call to extension function {public javax.xml.transform.Source org.opengis.cite.securityclient10.TestNGController.doTestRun(org.w3c.dom.Document) throws java.lang.Exception}: Exception in extension function java.lang.NoClassDefFoundError: javax/servlet/http/HttpSessionIdListener; SystemID: file:/root/te_base/work/_root_te_base_scripts_security-client10_1.0_ctl_/security-client10-suite/tns$run-ets-security-client10.fn; Line#: 50; Column#: -1
Fatal Error: Error in call to extension function {public java.lang.Object com.occamlab.te.TECore.callFunction(net.sf.saxon.expr.XPathContext,java.lang.String,java.lang.String,net.sf.saxon.om.NodeInfo) throws java.lang.Exception}: Exception in extension function net.sf.saxon.s9api.SaxonApiException: Error in call to extension function {public javax.xml.transform.Source org.opengis.cite.securityclient10.TestNGController.doTestRun(org.w3c.dom.Document) throws java.lang.Exception}: Exception in extension function java.lang.NoClassDefFoundError: javax/servlet/http/HttpSessionIdListener; SystemID: file:/root/te_base/work/_root_te_base_scripts_security-client10_1.0_ctl_/security-client10-suite/tns$Main.test; Line#: 99; Column#: -1
Jul 24, 2018 9:26:14 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [jsp] in context with path [/teamengine] threw exception [An exception occurred processing JSP page /viewSessionLog.jsp at line 185

182:    resDir = new File(userLog.toString() + System.getProperty("file.separator") + sessionId );
183: }
184: if(resDir.exists()){
185: core.earlHtmlReport(resultdir.toString());
186: earlHtml = new  File(resultdir + System.getProperty("file.separator") + "result" + System.getProperty("file.separator") + "index.html");
187: 
188: }

Stacktrace:] with root cause
java.lang.ArrayIndexOutOfBoundsException: 0
    at com.occamlab.te.html.EarlToHtmlTransformation.findEarlResultFile(EarlToHtmlTransformation.java:66)
    at com.occamlab.te.html.EarlToHtmlTransformation.earlHtmlReport(EarlToHtmlTransformation.java:36)
    at com.occamlab.te.TECore.earlHtmlReport(TECore.java:2486)
    at org.apache.jsp.viewSessionLog_jsp._jspService(viewSessionLog_jsp.java:342)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:439)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:607)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1139)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
    at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2555)
    at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2544)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

The main two items are:

INFO: validateJarFile(/usr/local/tomcat/webapps/teamengine/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class

and

Exception in extension function java.lang.NoClassDefFoundError: javax/servlet/http/HttpSessionIdListener

These are both caused by servlet-api 2.5 being loaded instead of servlet-api 3.1. The version of Jetty I am embedding (9.4.11.v20180605) corresponds to that version of the servlet-api. Tomcat however loads servlet-api 2.5; I was able to copy the tomcat directory out of the Docker container and found this version of the servlet-api in tomcat's lib directory.

From what I found online regarding this first message, the common solution is to set the servlet-api as "provided" in Maven to have the existing version of the library to be used instead. The second error message is a very common problem, although I haven't found anyone else with the "embedded server inside a Tomcat WAR" scenario.

The real problem is the mismatch in versions providing different APIs to servlet. I have a few potential solutions which I would like to share, as I may be missing something that Java might be able to do to handle this. (Can Java handle two versions of a library? Node.js does this, but that is a way different implementation.)

  1. Downgrade Jetty to have matching servlet-api
  2. Force upgrade Tomcat in docker-teamengine (see compatibility table)
  3. When running in TEAM Engine, use Tomcat servlets instead of Jetty servlets

The first one is probably easiest to get running. Downside is that this is an old and unsupported version, and the API may change and affect the ETS code.

The second is outside of this ETS and I am not sure about how strong of requirements we can have for TEAM Engine/Tomcat. It may also require different ETS versions for different Tomcat versions, each with a different servlet-api.

The third one will require more coding and more testing as there are more code paths.

I will be working on the ETS implementation for the JAR/IDE interfaces first and once those are stable I will come back and fix the TEAM Engine interface (and this issue). If anyone has any suggestions, please let me know!

openfirmware commented 6 years ago

I have been working on this issue this week and have learned some things.

Option A: Downgrade Jetty

I don't think this is a workable solution as it would be fragile and require you to use specific Jetty versions (and its according servlet API version) for different versions of Tomcat (or whichever container runner application). For example, there would have to be a version for Tomcat 6 (servlet API 2.5), Tomcat 7 (servlet API 3.0), Tomcat 8/8.5 (servlet API 3.1), Tomcat 9.0 (servlet API 4.0). This adds a lot of overhead for the system administrator who wants to install this ETS in their TEAM Engine.

Additionally, older versions of Jetty have different levels of TLS and cipher suite security (i.e. deprecated cipher lists). This could complicate testing modern secure clients which may not connect to older versions of Jetty.

Option B: Upgrade Tomcat

This also will depend on the system administrator, and has the same issue of needing different Jetty versions for different Tomcat versions. There is no Jetty version currently released that uses Servlet API 4.0, so does that mean the ETS cannot be used with Tomcat 9?

Option C: Use TEAM Engine Servlets

I think there might be something in MonitorServlet or similar classes that could be used to create custom test session endpoints for secure clients. However I can't find any documentation on how to use these.

Option D: Dynamic Class Loading

There is another potential solution I alluded to, which is using dynamic class loaders in Java to load Jetty's servlet API for this ETS, and then restoring the class loader when finished. I found some references to this being possible:

I am not experienced enough with Java to fully understand this, although I am giving it a shot as I think it will be the fastest way to get TEAM Engine support working.

Summary

I am working on the last option, as I don't think I have enough time to learn the internals of TEAM Engine for option C. If I can't get it working then I will have to explain in the ER what the current shortcomings in TEAM Engine and ETS are for client testing suites, and how we can potentially move forward with improving support.

openfirmware commented 6 years ago

Two more options:

Option E: Use OSGi

There was a thread on using this somewhere in the TEAM Engine repository. Maybe this could be implemented in just this ETS, and not need changes in TE. I haven't used OSGi so I don't know how much work this is. Kind of similar to Option D, but is more formalized.

Option F: Run in separate process

Could create a separate java process for the test server, and use some sort of inter-process communication to setup and teardown the test server and endpoints for test sessions. Could be a lot of work. There might be security implications in running a separate process, as some OS security systems may need special configuration (I am thinking of AppArmor).

openfirmware commented 6 years ago

Option G: Maven Shade Plugin

The Maven Shade plugin offers class renaming and relocation, which might be enough to avoid the servlet api version conflict.

openfirmware commented 6 years ago

The Maven Shade plugin seems to be working, I have some code on a branch.

I forked teamengine-docker and created a new entry for ets-security-client10 and have been using that to test locally. I had to modify the Dockerfile to add an extra task which extracts the sample Java Keystore file, so that the ETS has one. A better procedure for the TEAM Engine system administrator will be developed where they can set up the keystore details when they install the ETS.

The Shade plugin method works as far as running a new test session as WMS 1.1.1, and the Docker TEAM Engine console lists the test session endpoint (should be exposed in the browser though). I modified the docker-run command to open port 10080 for the ETS. Connecting to that endpoint using curl fails though with an SSL handshake error:

*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 10080 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:10080 
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:10080

I am looking into this now. Maybe the Java version in the Docker image is too old or doesn't have the right security ciphers?

openfirmware commented 6 years ago

I fixed the test run properties to run the Jetty server on 0.0.0.0 host, and tried to connect to that server inside the Docker container by starting bash instead of Tomcat, then starting Tomcat in the background and connecting with curl. Still fails, because the Jetty test server isn't actually started?

openfirmware commented 6 years ago

I brought up this issue during our CITE teleconference today, and Torsten asked me to clarify the versions being used here.

When I am testing the JAR or IDE, I am using:

Running as JAR/IDE is no problem. (If I have time, I would like to use Docker to automate testing the JAR process under different JDK versions, but I have to get this and a few other major items done first.)

For testing under TEAM Engine:

Right now when I run the ETS from TEAM Engine in Docker, the Docker console will print out the same test information as the JAR. This information contains the HTTP(S) endpoint for the test session, which I am trying to connect to from a test client (curl).

There are a few considerations for connecting to the ETS running in a Docker container, especially on a MacOS host, as the Docker container is running under a Linux VM which is running in MacOS. I was able to open ports to access TEAM Engine on port 8080, but accessing the ETS endpoint on port 10080 (the sample port I use in the test run properties) fails.

I have tried creating a second Docker container and connecting to the first container, and used nmap to scan the ports that were open; only the two Tomcat ports were available and port 10080 was not open.

I also tried launching the TEAM Engine docker with an interactive bash session instead of the Catalina script, and then ran the Catalina script in the background. Then I could set up a test session, and try using curl in the same container as the ETS, which should eliminate any firewall issues. This also fails and is what I am debugging now.

openfirmware commented 6 years ago

Found the issue. In the CTL script for TEAM Engine, wms-111 was being passed in as a test run property instead of wms111. This caused the embedded Jetty server to issue a NPE, which was caught in a try/catch handler and no output was sent to the console.

I had set up Tomcat 7 with Eclipse and JDK 8 under MacOS, and was able to use that to debug the TestServer class in the ETS. (I will write up a short guide on how to set up that test environment, as there doesn't seem to be an existing guide.)

I also tested this fix with TEAM Engine running under Docker, and when the host is set to 0.0.0.0 and port 10080 is published by Docker, I could access the testing endpoint from the host machine.

There are a few things to clean up before I will merge in the fix to the master repository:

Our lab has set up a cloud instance that we are using for another Testbed 14 deliverable, and I will deploy TEAM Engine to that instance for demonstrating how to use a free HTTPS certificate from Let's Encrypt.

openfirmware commented 6 years ago

Using Maven Shade to "rename" a set of packages into namespaces that do not conflict with earlier versions of those libraries loaded by Tomcat is a working solution.