Configuring Jetty-12 work module

a1730 commented 3 months ago

Jetty Version INFO :oejs.Server:main: jetty-12.0.10; built: 2024-05-30T04:40:36.563Z; git: 26106dfc84a03ddb6216062fe33b047fc332d0ce; jvm 22.0.1+8

Jetty Environment

core,ee8

Java Version openjdk 22.0.1 2024-04-16 OpenJDK Runtime Environment Temurin-22.0.1+8 (build 22.0.1+8) OpenJDK 64-Bit Server VM Temurin-22.0.1+8 (build 22.0.1+8, mixed mode)

Question Jetty-12 is not cleaning up the tmp directory after itself so we decided to use Jetty work module that worked for us with Jetty-10 but Jetty seems to be ignoring our configuration after we added work module to Jetty-12. So, how do we configure Jetty to use the work module/directory?

joakime commented 3 months ago

Just create a directory called work in the ${jetty.base} directory. Aka ${jetty.base}/work/

a1730 commented 3 months ago

That worked. Thank you @joakime !

a1730 commented 3 months ago

My apologies, that did not really work. Jetty uses work directory after setting -Djava.io.tmpdir=./work on the java command line otherwise Jetty uses whatever java.io.tmpdir is set to. It may just be a documentation issue.

java -Djetty.home=${JETTY_HOME} -Djetty.base=${JETTY_BASE} -jar $JETTY_HOME/start.jar --list-modules says: work - Creates the $JETTY_BASE/work directory as a persistent temp directory.

Add work module while setting java.io.tmpdir to anything but work. The work directory is not created and jetty uses java.io.tmpdir.
Okay, create work directory as instructed by @joakime while leaving java.io.tmpdir unchanged. Jetty uses tmpdir instead of work directory.
Change java.io.tmpdir=./work = life is good😄

janbartel commented 3 months ago

@a1730 I can't replicate your problem. If I enable the work module, then a writeable ${jetty.base}/work directory is created and used, irrespective of whether you set java.io.tmpdir. This is exactly what I would expect to happen from looking at the code. Here's an example:

mkdir base; cd base
java -jar ../jetty-home/start.jar --add-modules=http,server,deploy,ee8-webapp,ee8-demo-simple,work
java -Djava.io.tmpdir="/tmp" -jar ../jetty-home/start.jar --list-config

Results in:

[39] java -Djava.io.tmpdir="/tmp" -jar ../jetty-home/start.jar --list-config

Enabled Modules:
----------------
  0) resources                 transitive provider of resources for logging-jetty
  1) logging/slf4j             transitive provider of logging/slf4j for logging-jetty
                               dynamic dependency of logging-jetty
  2) logging-jetty             transitive provider of logging for threadpool
                               transitive provider of logging for bytebufferpool
                               transitive provider of logging for server
  3) bytebufferpool            transitive provider of bytebufferpool for server
                               ini template available with --add-modules=bytebufferpool
  4) threadpool                transitive provider of threadpool for server
                               ini template available with --add-modules=threadpool
  5) server                    ${jetty.base}/start.d/server.ini
  6) deploy                    ${jetty.base}/start.d/deploy.ini
  7) sessions                  transitive provider of sessions for ee8-servlet
                               ini template available with --add-modules=sessions
  8) ee8-servlet               transitive provider of ee8-servlet for ee8-webapp
                               transitive provider of ee8-servlet for ee8-security
  9) security                  transitive provider of security for ee8-security
 10) ee8-security              transitive provider of ee8-security for ee8-webapp
 11) ee-webapp                 transitive provider of ee-webapp for ee8-webapp
                               ini template available with --add-modules=ee-webapp
 12) ee8-webapp                ${jetty.base}/start.d/ee8-webapp.ini
 13) ee8-deploy                transitive provider of ee8-deploy for ee8-demo-simple
                               ini template available with --add-modules=ee8-deploy
 14) ee8-demo-simple           ${jetty.base}/start.d/ee8-demo-simple.ini
 15) http                      ${jetty.base}/start.d/http.ini
 16) work                      ${jetty.base}/start.d/work.ini

JVM Version & Properties:
-------------------------
 java.home = /home/janb/java/jvm/jdk-19.0.2
 java.vm.vendor = Oracle Corporation
 java.vm.version = 19.0.2+7-44
 java.vm.name = OpenJDK 64-Bit Server VM
 java.vm.info = mixed mode, sharing
 java.runtime.name = OpenJDK Runtime Environment
 java.runtime.version = 19.0.2+7-44
 java.io.tmpdir = /tmp
 user.dir = /home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/base
 user.language = en
 user.country = AU

Jetty Version & Properties:
---------------------------
 jetty.version = 12.0.11-SNAPSHOT
 jetty.tag.version = jetty-12.0.11-SNAPSHOT
 jetty.build = ff3ce62eb4bb84370fc5d07362a9b98dae715e41
 jetty.home = /home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/jetty-home
 jetty.base = /home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/base

Config Search Order:
--------------------
 <command-line>
 ${jetty.base} -> /home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/base
 ${jetty.home} -> /home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/jetty-home

System Properties:
------------------
 (no system properties specified)

Properties: Jetty
-----------------
 java.version = 19.0.2
 java.version.major = 19
 java.version.micro = 2
 java.version.minor = 0
 java.version.platform = 19
 jetty.base = /home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/base
 jetty.base.uri = file:///home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/base
 jetty.home = /home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/jetty-home
 jetty.home.uri = file:///home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/jetty-home
 jetty.webapp.addHiddenClasses = org.eclipse.jetty.logging.,${jetty.home.uri}/lib/logging/,org.slf4j.
 runtime.feature.alpn = true
 slf4j.version = 2.0.12

Classpath: Jetty
----------------
Version Information on 12 entries in the classpath.
Note: order presented here is how they would appear on the classpath.
      changes to the --module=name command line options will be reflected here.
 0:                    (dir) | ${jetty.base}/resources
 1:                   2.0.12 | ${jetty.home}/lib/logging/slf4j-api-2.0.12.jar
 2:         12.0.11-SNAPSHOT | ${jetty.home}/lib/logging/jetty-slf4j-impl-12.0.11-SNAPSHOT.jar
 3:         12.0.11-SNAPSHOT | ${jetty.home}/lib/jetty-http-12.0.11-SNAPSHOT.jar
 4:         12.0.11-SNAPSHOT | ${jetty.home}/lib/jetty-server-12.0.11-SNAPSHOT.jar
 5:         12.0.11-SNAPSHOT | ${jetty.home}/lib/jetty-xml-12.0.11-SNAPSHOT.jar
 6:         12.0.11-SNAPSHOT | ${jetty.home}/lib/jetty-util-12.0.11-SNAPSHOT.jar
 7:         12.0.11-SNAPSHOT | ${jetty.home}/lib/jetty-io-12.0.11-SNAPSHOT.jar
 8:         12.0.11-SNAPSHOT | ${jetty.home}/lib/jetty-deploy-12.0.11-SNAPSHOT.jar
 9:         12.0.11-SNAPSHOT | ${jetty.home}/lib/jetty-session-12.0.11-SNAPSHOT.jar
10:         12.0.11-SNAPSHOT | ${jetty.home}/lib/jetty-security-12.0.11-SNAPSHOT.jar
11:         12.0.11-SNAPSHOT | ${jetty.home}/lib/jetty-ee-12.0.11-SNAPSHOT.jar

Active XMLs: Jetty
------------------
 ${jetty.home}/etc/jetty-bytebufferpool.xml
 ${jetty.home}/etc/jetty-threadpool.xml
 ${jetty.home}/etc/jetty.xml
 ${jetty.home}/etc/jetty-deploy.xml
 ${jetty.home}/etc/sessions/id-manager.xml
 ${jetty.home}/etc/jetty-ee-webapp.xml
 ${jetty.home}/etc/jetty-http.xml

Properties: ee8
---------------
 contextHandlerClass = org.eclipse.jetty.ee8.webapp.WebAppContext

Classpath: ee8
--------------
Version Information on 5 entries in the classpath.
Note: order presented here is how they would appear on the classpath.
      changes to the --module=name command line options will be reflected here.
 0:                    4.0.6 | ${jetty.home}/lib/jetty-servlet-api-4.0.6.jar
 1:         12.0.11-SNAPSHOT | ${jetty.home}/lib/jetty-ee8-nested-12.0.11-SNAPSHOT.jar
 2:         12.0.11-SNAPSHOT | ${jetty.home}/lib/jetty-ee8-servlet-12.0.11-SNAPSHOT.jar
 3:         12.0.11-SNAPSHOT | ${jetty.home}/lib/jetty-ee8-security-12.0.11-SNAPSHOT.jar
 4:         12.0.11-SNAPSHOT | ${jetty.home}/lib/jetty-ee8-webapp-12.0.11-SNAPSHOT.jar

Active XMLs: ee8
----------------
 ${jetty.home}/etc/jetty-ee8-webapp.xml
 ${jetty.home}/etc/jetty-ee8-deploy.xml

And actually running it results in:

[40] java -Djava.io.tmpdir="/tmp" -jar ../jetty-home/start.jar 2024-06-18 03:33:22.339:WARN :oejx.XmlConfiguration:main: Deprecated method public static void org.eclipse.jetty.ee.WebAppClassLoading.addHiddenClasses(org.eclipse.jetty.util.Attributes,java.lang.String[]) in file:///home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/jetty-home/etc/jetty-ee-webapp.xml[40] java -Djava.io.tmpdir="/tmp" -jar ../jetty-home/start.jar 2024-06-18 03:33:22.339:WARN :oejx.XmlConfiguration:main: Deprecated method public static void org.eclipse.jetty.ee.WebAppClassLoading.addHiddenClasses(org.eclipse.jetty.util.Attributes,java.lang.String[]) in file:///home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/jetty-home/etc/jetty-ee-webapp.xml 2024-06-18 03:33:22.413:INFO :oejs.Server:main: jetty-12.0.11-SNAPSHOT; built: 2024-06-17T00:17:05.583Z; git: ff3ce62eb4bb84370fc5d07362a9b98dae715e41; jvm 19.0.2+7-44 2024-06-18 03:33:22.440:INFO :oejdp.ScanningAppProvider:main: Deployment monitor ee8 in [file:///home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/base/webapps/] at intervals 0s 2024-06-18 03:33:22.453:INFO :oejd.DeploymentManager:main: addApp: App@5552768b[ee8,null,/home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/base/webapps/ee8-demo-simple.war] 2024-06-18 03:33:22.633:INFO :oejew.StandardDescriptorProcessor:main: NO JSP Support for /ee8-demo-simple, did not find org.eclipse.jetty.ee8.jsp.JettyJspServlet > 2024-06-18 03:33:22.673:INFO :oejsh.ContextHandler:main: Started oeje8n.ContextHandler$CoreContextHandler@4a07d605{EE8 Demo Simple WebApp,/ee8-demo-simple,b=file:///home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/base/work/jetty-0_0_0_0-8080-ee8-demo-simple_war-_ee8-demo-simple-any-/webapp/,a=AVAILABLE,h=oeje8n.ContextHandler$CoreContextHandler$CoreToNestedHandler@74287ea3{STARTED}} 2024-06-18 03:33:22.676:INFO :oejs.DefaultSessionIdManager:main: Session workerName=node0 2024-06-18 03:33:22.687:INFO :oejs.AbstractConnector:main: Started ServerConnector@24bb6584{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} 2024-06-18 03:33:22.708:INFO :oejs.Server:main: Started oejs.Server@7770f470{STARTING}[12.0.11-SNAPSHOT,sto=5000] @1278ms 2024-06-18 03:33:22.413:INFO :oejs.Server:main: jetty-12.0.11-SNAPSHOT; built: 2024-06-17T00:17:05.583Z; git: ff3ce62eb4bb84370fc5d07362a9b98dae715e41; jvm 19.0.2+7-44 2024-06-18 03:33:22.440:INFO :oejdp.ScanningAppProvider:main: Deployment monitor ee8 in [file:///home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/base/webapps/] at intervals 0s 2024-06-18 03:33:22.453:INFO :oejd.DeploymentManager:main: addApp: App@5552768b[ee8,null,/home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/base/webapps/ee8-demo-simple.war] 2024-06-18 03:33:22.633:INFO :oejew.StandardDescriptorProcessor:main: NO JSP Support for /ee8-demo-simple, did not find org.eclipse.jetty.ee8.jsp.JettyJspServlet 2024-06-18 03:33:22.673:INFO :oejsh.ContextHandler:main: Started oeje8n.ContextHandler$CoreContextHandler@4a07d605{EE8 Demo Simple WebApp,/ee8-demo-simple,b=file:///home/janb/src/jetty-eclipse/jetty-12.0.x/jetty-home/target/base/work/jetty-0_0_0_0-8080-ee8-demo-simple_war-_ee8-demo-simple-any-/webapp/,a=AVAILABLE,h=oeje8n.ContextHandler$CoreContextHandler$CoreToNestedHandler@74287ea3{STARTED}} 2024-06-18 03:33:22.676:INFO :oejs.DefaultSessionIdManager:main: Session workerName=node0 2024-06-18 03:33:22.687:INFO :oejs.AbstractConnector:main: Started ServerConnector@24bb6584{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} 2024-06-18 03:33:22.708:INFO :oejs.Server:main: Started oejs.Server@7770f470{STARTING}[12.0.11-SNAPSHOT,sto=5000] @1278ms

Notice that the line in bold shows that the webapp was deployed from $[jetty.base}/work directory.

If you're seeing other behaviour, then let us know exactly what you're doing and how it differs from the simple example above.

a1730 commented 3 months ago

Hi @janbartel , Your results do not match mine as per the attached. We added work module which created the work directory but the runtime had other ideas.

Enabled Modules:
----------------
  0) resources                 ${jetty.base}/start.d/resources.ini
  1) logging/slf4j             transitive provider of logging/slf4j for logging-jetty
                               dynamic dependency of logging-jetty
  2) logging-jetty             transitive provider of logging for threadpool
                               transitive provider of logging for bytebufferpool
                               transitive provider of logging for server
  3) bytebufferpool            transitive provider of bytebufferpool for server
                               ini template available with --add-modules=bytebufferpool
  4) ext                       ${jetty.base}/start.d/ext.ini
  5) threadpool                transitive provider of threadpool for server
                               ini template available with --add-modules=threadpool
  6) server                    transitive provider of server for http
                               transitive provider of server for ee8-plus
                               transitive provider of server for state-tracking
                               transitive provider of server for deploy
                               transitive provider of server for plus
                               transitive provider of server for security
                               transitive provider of server for sessions
                               transitive provider of server for ee8-servlet
                               transitive provider of server for ee8-security
                               transitive provider of server for requestlog
                               transitive provider of server for jndi
                               transitive provider of server for gzip
                               ini template available with --add-modules=server
  7) deploy                    transitive provider of deploy for core-deploy
                               transitive provider of deploy for ee8-deploy
  8) core-deploy               ${jetty.base}/start.d/core-deploy.ini
  9) sessions                  transitive provider of sessions for ee8-servlet
                               ini template available with --add-modules=sessions
 10) ee8-servlet               ${jetty.base}/start.d/ee8-servlet.ini
 11) security                  transitive provider of security for ee8-security
 12) ee8-security              transitive provider of ee8-security for ee8-plus
                               transitive provider of ee8-security for ee8-webapp
 13) ee-webapp                 transitive provider of ee-webapp for ee8-webapp
                               ini template available with --add-modules=ee-webapp
 14) ee8-webapp                transitive provider of ee8-webapp for ee8-plus
                               transitive provider of ee8-webapp for ee8-deploy
                               ini template available with --add-modules=ee8-webapp
 15) plus                      transitive provider of plus for ee8-annotations
                               transitive provider of plus for ee8-plus
                               transitive provider of plus for jndi
 16) jndi                      ${jetty.base}/start.d/jndi.ini
 17) ee8-plus                  transitive provider of ee8-plus for ee8-annotations
 18) ee8-annotations           transitive provider of ee8-annotations for ee8-jsp
 19) ee8-apache-jsp            transitive provider of ee8-apache-jsp for ee8-jsp
 20) ee8-deploy                ${jetty.base}/start.d/ee8-deploy.ini
 21) ee8-glassfish-jstl        transitive provider of ee8-glassfish-jstl for ee8-jstl
 22) ee8-jndi                  ${jetty.base}/start.d/ee8-jndi.ini
 23) ee8-jsp                   transitive provider of ee8-jsp for ee8-jstl
 24) ee8-jstl                  ${jetty.base}/start.d/ee8-jstl.ini
 25) ee8-servlets              ${jetty.base}/start.d/ee8-servlets.ini
 26) state-tracking            ${jetty.base}/start.d/state-tracking.ini
 27) gzip                      ${jetty.base}/start.d/gzip.ini
 28) http                      ${jetty.base}/start.d/http.ini
 29) http2c                    ${jetty.base}/start.d/http2c.ini
 30) http-forwarded            ${jetty.base}/start.d/http-forwarded.ini
 31) jdbc                      ${jetty.base}/start.d/jdbc.ini
 32) proxy-protocol            ${jetty.base}/start.d/proxy-protocol.ini
 33) requestlog                ${jetty.base}/start.d/requestlog.ini
 34) work                      ${jetty.base}/start.d/work.ini

And actually running it results in:

2024-06-17 23:32:54.826:WARN :oejx.XmlConfiguration:main: Deprecated method public static void org.eclipse.jetty.ee.WebAppClassLoading.addHiddenClasses(org.eclipse.jetty.util.Attributes,java.lang.String[]) in file:///home/cglcons/vendor/jetty-home-12.0.10/etc/jetty-ee-webapp.xml
2024-06-17 23:32:54.963:INFO :oejs.Server:main: jetty-12.0.10; built: 2024-05-30T04:40:36.563Z; git: 26106dfc84a03ddb6216062fe33b047fc332d0ce; jvm 22.0.1+8
2024-06-17 23:32:54.981:INFO :oejdp.ScanningAppProvider:main: Deployment monitor core in [file:///home/cglcons/jetty/JETTY12.base/webapps/] at intervals 0s
2024-06-17 23:32:54.987:WARN :oejdp.ScanningAppProvider:main: class org.eclipse.jetty.deploy.providers.ContextProvider@8c3619e[file:///home/cglcons/jetty/JETTY12.base/webapps/] no environment for App@4331d187[ee8,null,/home/cglcons/jetty/JETTY12.base/webapps/hello.war], ignoring
2024-06-17 23:32:54.988:INFO :oejdp.ScanningAppProvider:main: Deployment monitor ee8 in [file:///home/cglcons/jetty/JETTY12.base/webapps/] at intervals 0s
2024-06-17 23:32:54.988:INFO :oejd.DeploymentManager:main: addApp: App@29ca3d04[ee8,null,/home/cglcons/jetty/JETTY12.base/webapps/hello.war]
2024-06-17 23:32:55.041:INFO :oejs.DefaultSessionIdManager:main: Session workerName=node0
2024-06-17 23:32:55.662:INFO :oejsh.ContextHandler:main: Started oeje8n.ContextHandler$CoreContextHandler@660e9100{hello,/hello,b=file:///tmp/jetty-services_-8080-hello_war-_hello-any-16050621075839022921/webapp/,a=AVAILABLE,h=oeje8n.ContextHandler$CoreContextHandler$CoreToNestedHandler@69f63d95{STARTED}}
2024-06-17 23:32:55.669:INFO :oejs.AbstractConnector:main: Started ServerConnector@514eedd8{[proxy], ([proxy], http/1.1, h2c)}{services.:8080}
2024-06-17 23:32:55.683:INFO :oejs.Server:main: Started oejs.Server@481ba2cf{STARTING}[12.0.10,sto=5000] @1801ms
2024-06-17 23:34:44.005:INFO :oejs.RequestLog:qtp928294079-32: 192.168.120.100 - - "OPTIONS /check HTTP/2.0" - 404 0 "-"
2024-06-17 23:34:44.005:INFO :oejs.RequestLog:qtp928294079-28: 192.168.120.100 - - "OPTIONS /check HTTP/2.0" - 404 0 "-"
^C2024-06-17 23:35:55.403:INFO :oejs.Server:JettyShutdownThread: Stopped oejs.Server@481ba2cf{STOPPING}[12.0.10,sto=5000]
2024-06-17 23:35:55.403:INFO :oejs.Server:JettyShutdownThread: Shutdown oejs.Server@481ba2cf{STOPPING}[12.0.10,sto=5000]
2024-06-17 23:35:55.413:INFO :oejs.AbstractConnector:JettyShutdownThread: Stopped ServerConnector@514eedd8{[proxy], ([proxy], http/1.1, h2c)}{services.:8080}

I removed java.io.tmpdir from the picture and the app was deployed to /tmp.

Please note that we are happy with the current settings as Jetty is working as configured, and this is not even an annoyance for us. I just thought I should report it, in case something was off. Thank you very much for taking a look.

joakime commented 3 months ago

I removed java.io.tmpdir from the picture and the app was deployed to /tmp.

Your app is not deployed there, that is just a temporary directory use for unpacked war files, a subdirectory of which is used as a the ServletContext temporary directory.

2024-06-17 23:32:55.662:INFO :oejsh.ContextHandler:main: Started oeje8n.ContextHandler$CoreContextHandler@660e9100{hello,/hello,b=file:///tmp/jetty-services_-8080-hello_war-_hello-any-16050621075839022921/webapp/,a=AVAILABLE,h=oeje8n.ContextHandler$CoreContextHandler$CoreToNestedHandler@69f63d95{STARTED}}

That can also be a result of ...

A Server.setTemporaryDirectory setting
A WebAppContext.setTemporaryDirectory setting
A Servlet Context temporary directory attribute

Check those possible sources too.

Can we also see your entire --list-config output, not just the list of modules?

Also, try the steps that @janbartel did, but in a new jetty.base directory. (just to eliminate possible configurations attempts you have from the past)

a1730 commented 3 months ago

Thank you @joakime. I followed the instructions from @janbartel in a new jetty.base and all is well. We now have to figure out what was wrong with the other jetty.base but this is on us😄 Thank you all.

jetty / jetty.project

Configuring Jetty-12 work module #11924