Open jmehrens opened 5 months ago
@basil If you are willing and able to:
angus-activation-2.0.2.jar
from your machine.jakarta.activation.debug=true
. This will send output to standard out file.How is the byte size of angus-activation-2.0.2.jar
relevant?
It is a way to see if that jar was deployed without asking if that jar was deployed :) The idea is force examination of the system instead of examination of ones memory of what was deployed. I trust you did it correctly but want to verify it. We already have 2 tickets of users falling in to this trap (which I need to fix) and I want to rule this out as a problem.
If that jar is missing there are some silent failures that are only shown with debugging on. As an example:
MimetypesFileTypeMap: load HOME
Cannot find or load an implementation for MimeTypeRegistryProvider.MimeTypeRegistry: can't load /home/jmehrens/.mime.types; Exception: java.lang.IllegalStateException: Provider for jakarta.activation.spi.MimeTypeRegistryProvider cannot be found
MimetypesFileTypeMap: load SYS
Cannot find or load an implementation for MimeTypeRegistryProvider.MimeTypeRegistry: can't load /usr/lib/jvm/jdk-21-oracle-x64/conf/mime.types; Exception: java.lang.IllegalStateException: Provider for jakarta.activation.spi.MimeTypeRegistryProvider cannot be found
MimetypesFileTypeMap: load JAR
MimetypesFileTypeMap: !anyLoaded
MimetypesFileTypeMap: not loading mime types file: /META-INF/mime.types
MimetypesFileTypeMap: load DEF
Cannot find or load an implementation for MimeTypeRegistryProvider.MimeTypeRegistry: can't load /META-INF/mimetypes.default; Exception: java.lang.IllegalStateException: Provider for jakarta.activation.spi.MimeTypeRegistryProvider cannot be found
The idea is force examination of the system instead of examination of ones memory of what was deployed.
I support users in an open-source project, so I can understand this perspective. This also means we can skip the indirection and I can explain to you how to reproduce the problem from scratch.
Download the latest Jenkins weekly release, then start it with Java 11, 17, or 21 with java -jar jenkins.war
. The first time you start it, complete the setup wizard following the defaults and install the suggested plugins. Among other things, this will install three interesting plugins: Email Extension, Jakarta Mail API, and Jakarta Activation API.
Once Jenkins is up and running, go to Manage Jenkins, System. Under "System Admin e-mail address", set an address like Basil Crow <basilcrow@example.com>
. Under "Extended E-mail Notification", configure your SMTP server. If your SMTP server requires authentication, configure that under Advanced. Also check Enable Debug Mode as that can be quite handy.
Once Jenkins is configured, create a new Freestyle project. As an example, I often create a test project that clones https://github.com/basil/simple-maven-project-with-tests.git and runs mvn clean verify
. Under Post-build Actions add Editable Email Notification. Under Project Recipient List add your email address. Under Advanced Settings, Triggers remove any existing triggers and add an Always trigger that sends to Recipient List. This means every time you run the job, it will send an email to the Recipient List.
Now run the job in Jenkins, and at the end it should send you an email. This should work before proceeding to the next step, so if it fails, do whatever it takes to get it to work, or contact me for help. Now, once it is working, you can break it by removing my workaround as follows.
Clone https://github.com/jenkinsci/jakarta-mail-api-plugin and remove the workaround with rm -rf src/main/java
. Build it with Java 11, 17, or 21 with mvn clean verify -Pquick-build
, which will generate a target/jakarta-mail-api.hpi
file. Replace ${JENKINS_HOME}/plugins/jakarta-mail-api.jpi
with this file and restart Jenkins. Re-run the job. An email should no longer be sent and in the console log you should see a stack trace about how some mime type is missing because the mailcap is empty. Attach a debugger to the Java process and you can verify the gory details of my assessment. The stack trace:
12:23:33 DEBUG SMTP: IOException while sending, closing, THROW:
12:23:33 jakarta.activation.UnsupportedDataTypeException: no object DCH for MIME type multipart/mixed;
12:23:33 boundary="----=_Part_0_1965321476.1709583812613"
12:23:33 at jakarta.activation.ObjectDataContentHandler.writeTo(DataHandler.java:872)
12:23:33 at jakarta.activation.DataHandler.writeTo(DataHandler.java:291)
12:23:33 at jakarta.mail.internet.MimeBodyPart.writeTo(MimeBodyPart.java:1683)
12:23:33 at jakarta.mail.internet.MimeMessage.writeTo(MimeMessage.java:1913)
12:23:33 at org.eclipse.angus.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:1305)
12:23:33 at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:554)
12:23:33 at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:484)
12:23:33 at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:385)
12:23:33 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
12:23:33 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
12:23:33 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:767)
12:23:33 at hudson.model.Build$BuildExecution.cleanUp(Build.java:189)
12:23:33 at hudson.model.Run.execute(Run.java:1936)
12:23:33 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
12:23:33 at hudson.model.ResourceController.execute(ResourceController.java:101)
12:23:33 at hudson.model.Executor.run(Executor.java:442)
12:23:33 MessagingException message: IOException while sending message
12:23:33 Next UnsupportedDataTypeException message: no object DCH for MIME type multipart/mixed;
12:23:33 boundary="----=_Part_0_1965321476.1709583812613"
@basil Thanks for the reproducer! This will save me a ton of time. You are a gentleman and a scholar.
For my own notes this is the project console output of the basic test to get a weekly build working with no changes.
Looking for: jmehrens@localhost
starting at: 0
firstFoundIdx: 0
firstFoundIdx-substring: jmehrens@localhost
=> found type: 0
Successfully created MimeMessage
Sending email to: jmehrens@localhost
DEBUG: getProvider() returning jakarta.mail.Provider[TRANSPORT,smtp,org.eclipse.angus.mail.smtp.SMTPTransport,Oracle]
DEBUG SMTP: useEhlo true, useAuth false
DEBUG SMTP: trying to connect to host "localhost", port 2525, isSSL false
220 jmehrens-VirtualBox ESMTP SubEthaSMTP null
DEBUG SMTP: connected to host "localhost", port: 2525
EHLO jmehrens-VirtualBox
250-jmehrens-VirtualBox
250-8BITMIME
250-AUTH LOGIN
250 Ok
DEBUG SMTP: Found extension "8BITMIME", arg ""
DEBUG SMTP: Found extension "AUTH", arg "LOGIN"
DEBUG SMTP: Found extension "Ok", arg ""
DEBUG SMTP: use8bit false
MAIL FROM:<nobody@nowhere>
250 Ok
DEBUG SMTP: sendPartial set
RCPT TO:<jmehrens@localhost>
250 Ok
DEBUG SMTP: Verified Addresses
DEBUG SMTP: jmehrens@localhost
DATA
354 End data with <CR><LF>.<CR><LF>
Date: Sat, 9 Mar 2024 21:58:24 -0600 (CST)
From: address not configured yet <nobody@nowhere>
To: jmehrens@localhost
Message-ID: <1521760439.1.1710043105326@jmehrens-VirtualBox>
Subject: simple-maven-project - Build # 7 - Successful!
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="----=_Part_0_26769737.1710043105033"
X-Jenkins-Job: simple-maven-project
X-Jenkins-Result: SUCCESS
List-ID:
------=_Part_0_26769737.1710043105033
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
simple-maven-project - Build # 7 - Successful:
Check console output at http://localhost:8080/job/simple-maven-project/7/ to view the results.
------=_Part_0_26769737.1710043105033--
.
250 Ok
DEBUG SMTP: message successfully delivered to mail server
QUIT
221 Bye
Finished: SUCCESS
Here is the launch console output for jenkins.war running with activation debugging for the above console log:
jmehrens@jmehrens-VirtualBox:~/jenkins$ java -Djakarta.activation.debug=true -Dangus.activation.debug=true -jar jenkins.war
Running from: /home/jmehrens/jenkins/jenkins.war
webroot: /home/jmehrens/.jenkins/war
2024-03-10 03:55:40.405+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
2024-03-10 03:55:40.733+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2024-03-10 03:55:40.941+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-10.0.20; built: 2024-01-29T20:46:45.278Z; git: 3a745c71c23682146f262b99f4ddc4c1bc41630c; jvm 21.0.1+12-LTS-29
2024-03-10 03:55:41.716+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2024-03-10 03:55:41.874+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: Session workerName=node0
2024-03-10 03:55:43.204+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /home/jmehrens/.jenkins found at: $user.home/.jenkins
2024-03-10 03:55:43.553+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@7adbd080{Jenkins v2.448,/,file:///home/jmehrens/.jenkins/war/,AVAILABLE}{/home/jmehrens/.jenkins/war}
2024-03-10 03:55:43.585+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@45afc369{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2024-03-10 03:55:43.631+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started Server@f0f2775{STARTING}[10.0.20,sto=0] @4472ms
2024-03-10 03:55:43.634+0000 [id=40] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2024-03-10 03:55:44.275+0000 [id=46] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
2024-03-10 03:55:44.717+0000 [id=53] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2024-03-10 03:55:54.000+0000 [id=58] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
MimetypesFileTypeMap: load HOME
MailcapCommandMap: load HOME
MimeTypeRegistry: can't load from file - /home/jmehrens/.mime.types; Exception: java.io.FileNotFoundException: /home/jmehrens/.mime.types (No such file or directory)
MimetypesFileTypeMap: load SYS
MimeTypeRegistry: can't load from file - /usr/lib/jvm/jdk-21-oracle-x64/conf/mime.types; Exception: java.io.FileNotFoundException: /usr/lib/jvm/jdk-21-oracle-x64/conf/mime.types (No such file or directory)
MimetypesFileTypeMap: load JAR
MimetypesFileTypeMap: !anyLoaded
new MailcapFile: file /home/jmehrens/.mailcap
MailcapRegistry: can't load from file - /home/jmehrens/.mailcap; Exception: java.io.FileNotFoundException: /home/jmehrens/.mailcap (No such file or directory)
MailcapCommandMap: load SYS
MimetypesFileTypeMap: not loading mime types file: /META-INF/mime.types
MimetypesFileTypeMap: load DEF
new MailcapFile: file /usr/lib/jvm/jdk-21-oracle-x64/conf/mailcap
MailcapRegistry: can't load from file - /usr/lib/jvm/jdk-21-oracle-x64/conf/mailcap; Exception: java.io.FileNotFoundException: /usr/lib/jvm/jdk-21-oracle-x64/conf/mailcap (No such file or directory)
MailcapCommandMap: load JAR
MailcapCommandMap: getResources
MailcapCommandMap: URL jar:file:/home/jmehrens/.jenkins/plugins/jakarta-mail-api/WEB-INF/lib/angus-mail-2.0.3.jar!/META-INF/mailcap
Added: MIMETypeEntry: text/html, html
Added: MIMETypeEntry: text/html, htm
Added: MIMETypeEntry: text/html, HTML
Added: MIMETypeEntry: text/html, HTM
Added: MIMETypeEntry: text/plain, txt
Added: MIMETypeEntry: text/plain, text
Added: MIMETypeEntry: text/plain, TXT
Added: MIMETypeEntry: text/plain, TEXT
Added: MIMETypeEntry: image/gif, gif
Added: MIMETypeEntry: image/gif, GIF
Added: MIMETypeEntry: image/ief, ief
Added: MIMETypeEntry: image/jpeg, jpeg
Added: MIMETypeEntry: image/jpeg, jpg
Added: MIMETypeEntry: image/jpeg, jpe
Added: MIMETypeEntry: image/jpeg, JPG
Added: MIMETypeEntry: image/tiff, tiff
Added: MIMETypeEntry: image/tiff, tif
Added: MIMETypeEntry: image/png, png
Added: MIMETypeEntry: image/png, PNG
Added: MIMETypeEntry: image/x-xwindowdump, xwd
Added: MIMETypeEntry: application/postscript, ai
Added: MIMETypeEntry: application/postscript, eps
Added: MIMETypeEntry: application/postscript, ps
Added: MIMETypeEntry: application/rtf, rtf
Added: MIMETypeEntry: application/x-tex, tex
Added: MIMETypeEntry: application/x-texinfo, texinfo
Added: MIMETypeEntry: application/x-texinfo, texi
Added: MIMETypeEntry: application/x-troff, t
Added: MIMETypeEntry: application/x-troff, tr
Added: MIMETypeEntry: application/x-troff, roff
Added: MIMETypeEntry: audio/basic, au
Added: MIMETypeEntry: audio/midi, midi
Added: MIMETypeEntry: audio/midi, mid
new MailcapFile: InputStream
Added: MIMETypeEntry: audio/x-aifc, aifc
Added: MIMETypeEntry: audio/x-aiff, aif
Added: MIMETypeEntry: audio/x-aiff, aiff
Added: MIMETypeEntry: audio/x-mpeg, mpeg
Added: MIMETypeEntry: audio/x-mpeg, mpg
Added: MIMETypeEntry: audio/x-wav, wav
Added: MIMETypeEntry: video/mpeg, mpeg
Added: MIMETypeEntry: video/mpeg, mpg
Added: MIMETypeEntry: video/mpeg, mpe
Added: MIMETypeEntry: video/quicktime, qt
Added: MIMETypeEntry: video/quicktime, mov
Added: MIMETypeEntry: video/x-msvideo, avi
parse: text/plain;; x-java-content-handler=org.eclipse.angus.mail.handlers.text_plain
Type: text/plain
Command: content-handler, Class: org.eclipse.angus.mail.handlers.text_plain
MimetypesFileTypeMap: successfully loaded mime types file: /META-INF/mimetypes.default
parse: text/html;; x-java-content-handler=org.eclipse.angus.mail.handlers.text_html
Type: text/html
Command: content-handler, Class: org.eclipse.angus.mail.handlers.text_html
parse: text/xml;; x-java-content-handler=org.eclipse.angus.mail.handlers.text_xml
Type: text/xml
Command: content-handler, Class: org.eclipse.angus.mail.handlers.text_xml
parse: multipart/*;; x-java-content-handler=org.eclipse.angus.mail.handlers.multipart_mixed; x-java-fallback-entry=true
Type: multipart/*
Command: content-handler, Class: org.eclipse.angus.mail.handlers.multipart_mixed
parse: message/rfc822;; x-java-content-handler=org.eclipse.angus.mail.handlers.message_rfc822
Type: message/rfc822
Command: content-handler, Class: org.eclipse.angus.mail.handlers.message_rfc822
MailcapCommandMap: successfully loaded mailcap file from URL: jar:file:/home/jmehrens/.jenkins/plugins/jakarta-mail-api/WEB-INF/lib/angus-mail-2.0.3.jar!/META-INF/mailcap
MailcapCommandMap: load DEF
new MailcapFile: InputStream
MailcapCommandMap: successfully loaded mailcap file: /META-INF/mailcap.default
2024-03-10 03:55:54.064+0000 [id=55] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
2024-03-10 03:55:54.097+0000 [id=58] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2024-03-10 03:55:54.761+0000 [id=50] INFO h.p.b.g.GlobalTimeOutConfiguration#load: global timeout not set
2024-03-10 03:55:56.174+0000 [id=58] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded
2024-03-10 03:55:56.175+0000 [id=58] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted
2024-03-10 03:55:57.028+0000 [id=51] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2024-03-10 03:55:57.057+0000 [id=49] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2024-03-10 03:55:57.162+0000 [id=59] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
2024-03-10 03:55:57.444+0000 [id=39] INFO hudson.lifecycle.Lifecycle#onReady: Jenkins is fully up and running
DEBUG: setDebug: Jakarta Mail version 2.1.3
MailcapCommandMap: createDataContentHandler for text/plain
search DB #1
got content-handler
class org.eclipse.angus.mail.handlers.text_plain
MailcapCommandMap: createDataContentHandler for multipart/mixed
search DB #1
search DB #2
search fallback DB #1
got content-handler
class org.eclipse.angus.mail.handlers.multipart_mixed
TODO notes for myself:
UnsupportedDataTypeException: no object DCH for MIME type multipart/mixed;
message. Needs additional details in the message to aid debugging.Notes to myself:
Jenkins Dependencies and Class Loading shows a classloader tree where for example JAF-API is not able to locate JakartaMail (plug-in C) mailcap.file resolved from JAF-API (plugin A) because the context classloader is
Jenkins core
. See: https://github.com/jakartaee/jaf-api/pull/145#issuecomment-1981960181Mail uses JAF but Mail jar contains the mailcap resource file.
More thought needs to be put in to this but there are some options: