Exception in thread "IdxTxnThread" java.lang.NoClassDefFoundError: javax/activation/DataSource

[sunmingtao]

DLC Indexer fails to send error email notification in Dev/UAT, potentially Prod.

However, it works locally

See below log:

11:40:00.057 java[30774]: 2020-10-21 11:40:00,057: DEBUG: au.gov.nla.dl.services.TransactionIndexingService - Staged: for indexing: 0, for deletion: 0, for closing: 011:40:00.062 java[30774]: Exception in thread "IdxTxnThread" java.lang.NoClassDefFoundError: javax/activation/DataSource11:40:00.062 java[30774]:       at 
org.apache.commons.mail.Email.createMimeMessage(Email.java:1901)11:40:00.062 java[30774]:       at 
org.apache.commons.mail.Email.buildMimeMessage(Email.java:1326)11:40:00.062 java[30774]:       at 
org.apache.commons.mail.Email.send(Email.java:1495)11:40:00.062 java[30774]:       at 
au.gov.nla.dl.services.EmailService.sendNotification(EmailService.java:24)11:40:00.062 java[30774]:       at 
au.gov.nla.dl.services.TransactionIndexingService.notifyIfErrors(TransactionIndexingService.java:660)11:40:00.062 java[30774]:       at 
au.gov.nla.dl.services.TransactionIndexingService.stageLatestTransactionsForIndexing(TransactionIndexingService.java:262)11:40:00.062 java[30774]:       at 
au.gov.nla.dl.controllers.TransactionIndexerController$1.run(TransactionIndexerController.java:45)11:40:00.062 java[30774]: 
Caused by: java.lang.ClassNotFoundException: javax.activation.DataSource11:40:00.062 java[30774]:       at 
java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)11:40:00.062 java[30774]:       at 
java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)11:40:00.062 java[30774]:       at 
java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)11:40:00.062 java[30774]:       ... 7 more

[sunmingtao]

The exception is thrown on the line "new MimeMessage(session)" (at Email.java 1901)

javax.activation.DataSource is on the classpath

[msun@hammer dl-indexer-service]$ jar tf /apps/dl-indexer-service/ROOT/WEB-INF/lib/activation-1.1.jar | grep javax/activation/DataSource.class
javax/activation/DataSource.class

I have put the following code to locate the jar from which the classes are loaded.

CodeSource src = MimeMessage.class.getProtectionDomain().getCodeSource();
if (src != null){      
    URL jar = src.getLocation();      
    System.out.println("MimeMessage class locates at " + jar.toString()); 
}
src = DataSource.class.getProtectionDomain().getCodeSource();
if (src != null)
    URL jar = src.getLocation();     
    System.out.println("DataSource class locates at " + jar.toString()); 
}

Surprisingly, the MimeMessage class is loaded from the jar in jetty container rather than the application lib folder (WEB-INF/lib)

10:50:00.044 java[3176]: MimeMessage class locates at *file:/var/cache/jvmctl/container/jetty-9.4.21.v20190926/lib/mail/javax.mail.glassfish-1.4.1.v201005082020.jar*
10:50:00.044 java[3176]: DataSource class locates at *file:/apps/dl-indexer-service/ROOT/WEB-INF/lib/activation-1.1.jar*

The jar containing MimeMessage is on the classpath as well (A maven dependency)

[msun@hammer dl-indexer-service]$ find /apps/dl-indexer-service/ROOT/WEB-INF/lib/ -name "javax.mail*jar"
/apps/dl-indexer-service/ROOT/WEB-INF/lib/javax.mail-api-1.6.2.jar
[msun@hammer dl-indexer-service]$ jar tf /apps/dl-indexer-service/ROOT/WEB-INF/lib/javax.mail-api-1.6.2.jar | grep MimeMessage.class
javax/mail/internet/MimeMessage.class

These observations bring a lot of questions.

1. When there is class conflict, why do Jetty libraries take precedence over application libraries?
2. Even if JVM cannot find the class (javax.activation.DataSource) in the jetty container, why stop there? Why not search the application classpath?
3. Can we get rid of the javax.mail.glassfish.jar from Jetty container to force JVM to load class from application classpath?
4. If 3 isn't possible (maybe other applications need it), can we add activation.jar, which contains the missing class javax.activation.DataSource, to the jetty library?

[sunmingtao]

Here is the answer to all the questions and it provides a neat solution as well.

https://github.com/eclipse/jetty.docker/issues/10

Jetty libraries do take precedence over the jar on class path, which is super counter-intuitive.