Stripping AWS Mail component with TrueZIP blocks packaging for docker images

[poikilotherm]

Dear devs, Dear @bsilverstein as the one who wrote the code for #3921,

while working on #4172, I am crafting a new Maven profile to generate smallest possible Docker images based on the payara 5 server-full image using https://dmp.fabric8.io.

As I want to make things really small, I don't want to bundle the dataverse dependencies within the WAR file, but create a small WAR without all deps, containing only classes and ressources. The deps get moved in by copying them the /lib of the Glassfish/Payara domain (so they are kind of "provided" scoped in Maven terminology). (For those interested in Docker: this should provide us with smaller layer changes and smaller hub transfers!)

The stripping of the mailing stuff from the AWS SDK makes this hard to achieve in two ways: a) I would to fiddle around with this, too and b) the TrueZIP plugin simply fails as I don't build the WAR with deps inside and it is not easy to skip it in a profile without making a mess in the pom.xml...

Back in #3921 where all of this was introduced, there has been a pointer to the AWS dev docs about using individual components.

Did you guys ever try to get that rolling? If not: maybe this would be the perfect time to do so now... Willing to contribute on this. :smile:

---

This is related to a question I made on IRC (see logs).

EDIT: @pdurbin suggested some clarification on IRC.

The pom.xml plugin section I would like to see removed:

-            <!-- v4.8: The truezip-maven-plugin below deletes two copies of a file that the AWS SDK bundle
-            includes called javamail.providers which breaks system emails. TODO: make our own AWS SDK without Amazon SES.
-            - bsilverstein 8/8/2017 -->
-            <plugin>
-                <groupId>org.codehaus.mojo</groupId>
-                <artifactId>truezip-maven-plugin</artifactId>
-                <version>1.1</version>
-                <executions>
-                    <execution>
-                        <id>remove-javamail-providers-from-exploded</id>
-                        <goals>
-                            <goal>remove</goal>
-                        </goals>
-                        <phase>package</phase>
-                        <configuration>
-                            <fileset>
-                                <directory>target/dataverse-${project.version}/WEB-INF/lib/aws-java-sdk-bundle-${aws.version}.jar/META-INF</directory>
-                                <includes>
-                                    <include>javamail.providers</include>
-                                </includes>
-                            </fileset>
-                        </configuration>
-                    </execution>
-                    <execution>
-                        <id>remove-javamail-providers-from-war</id>
-                        <goals>
-                            <goal>remove</goal>
-                        </goals>
-                        <phase>package</phase>
-                        <configuration>
-                            <fileset>
-                                <directory>target/dataverse-${project.version}.war/WEB-INF/lib/aws-java-sdk-bundle-${aws.version}.jar/META-INF</directory>
-                                <includes>
-                                    <include>javamail.providers</include>
-                                </includes>
-                            </fileset>
-                        </configuration>
-                    </execution>
-                </executions>
-            </plugin>

And something that needs to be changed (and complemented by some more stuff):

         <dependency>
             <groupId>com.amazonaws</groupId>
-            <artifactId>aws-java-sdk-bundle</artifactId>
+            <artifactId>aws-java-sdk-s3</artifactId>
             <version>${aws.version}</version>
         </dependency>

[matthew-a-dunlap]

Hey @poikilotherm , the reason we took this approach is due to dependency compatibility issues between the AWS SDK and Jackson/Glassfish. See around this post in 3921 for more info on that problem. The solution we went with was not the most elegant, but functioned well for our standard deployment strategy.

I am unaware at this point why we didn't go with pointing to the individual components, that seems like a good strategy to try. If you do try to fix this weird part of our build please make sure it still operates for our norm of keeping all the dependencies in the war. Thanks!

[poikilotherm]

Will have a look at the Jackson problems tomorrow... Thx for pointing that out.

Using the S3-only SDK seems to be alright, that's the only AWS stuff in use currently. This reduced the WAR size from 146MB to 128MB already, which seems good, right? :wink:

I checked all of the involved jar archives, as far as I can tell, there is no javamail related stuff in there, thus the mail problems should be avoided thoroughly.

[poikilotherm]

Hi @matthew-a-dunlap,

you wrote the lines following https://github.com/IQSS/dataverse/blob/c2dec152837174cf373c8299508c0482d3cb9c30/pom.xml#L73 . I'm not sure I fully understand what you meant with "an AWS SDK with unique references to needed libraries"... I guess this was related to the mentioned Jackson problems, right?

As far as I can see from the MVN AWS SDK, MVN AWS SDK Complete Bundle, MVN AWS SDK S3, there never have been explicit RUNTIME dependecies for the Jackson 2.6 stuff. EDIT: This is BS. Compile scope. My bad, sry. Anyway, there is no Jackson Databinding 2.6 on the classpath, see below.

There are compile scope deps against Jackson 2.6 in MVN AWS SDK Core. But since the release of .172, nowadays v.11.442, there have been only marginal changes in versioning and not for Jackson. So these "unique references" never were absent...?

What was the exact problem with Jackson you mentioned earlier?

The included Jackson Databinding in Glassfish 4.1 is v2.3.2 (lookup from the glassfish-4.1.zip JAR manifests). But this is overridden by jackson-databinding v2.5.3 within the WAR deps, which in turn is a dependency introduced by com.maxmind.geoip2. (Looked that up via mvn dependency:tree)

Included in the resulting dataverse WAR are the following Jackson JARs:

WEB-INF/lib/jackson-core-2.9.6.jar
WEB-INF/lib/jackson-dataformat-cbor-2.6.7.jar
WEB-INF/lib/jackson-annotations-2.9.6.jar
WEB-INF/lib/jackson-databind-2.5.3.jar
WEB-INF/lib/jackson-mapper-asl-1.9.7.jar
WEB-INF/lib/jackson-core-asl-1.9.7.jar

I'm puzzled...

[poikilotherm]

Maybe I have a hint about why you ran into problems and solved those when using the bundle: https://aws.amazon.com/de/blogs/developer/java-sdk-bundle/

To sum up that blog article: within the bundle, Jackson and other third party licenses are re-packaged under com.amazonaws.thirdparty and used inside. That would be a very good explanation why it started to "just work" when you switched to the complete bundle.

[poikilotherm]

Using the Maven Enforcer plugin and its dependency convergence rule leads to the following errors (among ALOT of others):

[WARNING] 
Dependency convergence error for com.fasterxml.jackson.core:jackson-databind:2.6.7.1 paths to dependency are:
+-edu.harvard.iq:dataverse:4.9.4
  +-com.amazonaws:aws-java-sdk-s3:1.11.172
    +-com.amazonaws:aws-java-sdk-core:1.11.172
      +-com.fasterxml.jackson.core:jackson-databind:2.6.7.1
and
+-edu.harvard.iq:dataverse:4.9.4
  +-com.amazonaws:aws-java-sdk-s3:1.11.172
    +-com.amazonaws:jmespath-java:1.11.172
      +-com.fasterxml.jackson.core:jackson-databind:2.6.7.1
and
+-edu.harvard.iq:dataverse:4.9.4
  +-com.maxmind.geoip2:geoip2:2.3.1
    +-com.maxmind.db:maxmind-db:1.0.0
      +-com.fasterxml.jackson.core:jackson-databind:2.4.2
and
+-edu.harvard.iq:dataverse:4.9.4
  +-com.maxmind.geoip2:geoip2:2.3.1
    +-com.fasterxml.jackson.core:jackson-databind:2.5.3
and
+-edu.harvard.iq:dataverse:4.9.4
  +-org.apache.tika:tika-parsers:1.19
    +-com.fasterxml.jackson.core:jackson-databind:2.9.6

And as we saw above, the v2.5.3 wins. This will take some work to sort out, especially if done for all defined deps (which is best practise anyway...).

[matthew-a-dunlap]

Hey @poikilotherm , sorry for the lack of info earlier, I had forgotten the details and some of them were lost to the sands of time and intern involvement. The movement of the 3rd party licenses to com.amazonaws.thirdparty was why we went with the big packaged library, as it was the most straightforward way of getting s3 operating. We talked about coming back to build our own specific copy of the s3 library but that's not been very high priority.

I'll bring it up during our team's tech hours today, there is definitely debt in cleaning up our dependencies in general and this one specifically.

[poikilotherm]

Most likely this is a matter of adding excludes to the dependency definitions or use dependencyManagement. See also http://timsteffens.blogspot.com/2014/05/solving-conflicts-with-transitive-maven.html?m=1 for managing these transitive deps.

[poikilotherm]

I just changed #5289 to point to this issue instead of #5288. That was mainly because @pdurbin suggested smaller chunks, and the work on my feature branch is mainly about this issue right now.

[pdurbin]

@dlmurphy heads up that pull request #5289 contains a fancy new way to create images from code/markup embedded in the rst files and I'm curious if you have the necessary software installed on Mac and Windows for this to work. Should work fine on Linux where we build the docs for public consumption.

[kcondon]

@poikilotherm @pdurbin I'm able to build and use somewhat the resulting war file but it throws a stack trace about a missing method when I load the homepage and an exception (not logged) when clicking on a dataset card to view the dataset page. Works fine with /develop.

The missing method stack trace has to do with Jackson:

[2018-11-16T14:09:07.433-0500] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.metrics.MetricsServiceBean] [tid: _ThreadID=54 _ThreadName=jk-connector(4)] [timeMillis: 1542395347433] [levelValue: 800] [[ query: EJBQueryImpl(DataReadQuery(sql="select cvv.strvalue, count(dataverse_id) from dataversesubjects join controlledvocabularyvalue cvv ON cvv.id = controlledvocabularyvalue_id group by cvv.strvalue order by count desc;"))]]

[2018-11-16T14:09:21.844-0500] [glassfish 4.1] [WARNING] [] [javax.enterprise.resource.webcontainer.jsf.lifecycle] [tid: _ThreadID=52 ThreadName=jk-connector(2)] [timeMillis: 1542395361844] [levelValue: 900] [[ /search-include-fragment.xhtml @653,93 actionListener="#{SearchIncludeFragment.setDisplayCardValues()}": java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper; javax.el.ELException: /search-include-fragment.xhtml @653,93 actionListener="#{SearchIncludeFragment.setDisplayCardValues()}": java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper; at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:111) at javax.faces.event.MethodExpressionActionListener.processAction(MethodExpressionActionListener.java:147) at javax.faces.event.ActionEvent.processListener(ActionEvent.java:88) at javax.faces.component.UIComponentBase.broadcast(UIComponentBase.java:813) at javax.faces.component.UICommand.broadcast(UICommand.java:300) at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:790) at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1282) at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:81) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101) at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:646) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:575) at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546) at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:378) at org.ocpsoft.rewrite.servlet.impl.HttpRewriteResultHandler.handleResult(HttpRewriteResultHandler.java:42) at org.ocpsoft.rewrite.servlet.RewriteFilter.rewrite(RewriteFilter.java:297) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:198) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper; at com.amazonaws.partitions.PartitionsLoader.(PartitionsLoader.java:54) at com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30) at com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:64) at com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:52) at com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:105) at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:369) at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:337) at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46) at edu.harvard.iq.dataverse.dataaccess.S3AccessIO.(S3AccessIO.java:88) at edu.harvard.iq.dataverse.dataaccess.DataAccess.getStorageIO(DataAccess.java:70) at edu.harvard.iq.dataverse.dataaccess.DataAccess.getStorageIO(DataAccess.java:52) at edu.harvard.iq.dataverse.ThumbnailServiceWrapper.getDatasetCardImageAsBase64Url(ThumbnailServiceWrapper.java:213) at edu.harvard.iq.dataverse.ThumbnailServiceWrapper.getDatasetCardImageAsBase64Url(ThumbnailServiceWrapper.java:185) at edu.harvard.iq.dataverse.ThumbnailServiceWrapper$Proxy$$$_WeldClientProxy.getDatasetCardImageAsBase64Url(Unknown Source) at edu.harvard.iq.dataverse.search.SearchIncludeFragment.setDisplayCardValues(SearchIncludeFragment.java:1180) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at javax.el.ELUtil.invokeMethod(ELUtil.java:332) at javax.el.BeanELResolver.invoke(BeanELResolver.java:537) at javax.el.CompositeELResolver.invoke(CompositeELResolver.java:256) at com.sun.el.parser.AstValue.invoke(AstValue.java:283) at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:304) at org.jboss.weld.util.el.ForwardingMethodExpression.invoke(ForwardingMethodExpression.java:40) at org.jboss.weld.el.WeldMethodExpression.invoke(WeldMethodExpression.java:50) at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105) ... 56 more ]]

[pdurbin]

@kcondon yikes. Thanks.

[kcondon]

@pdurbin Np. Do you mind emailing Ops what is needed to get new doc image working (it's not). Alternatively, if you can install it directly or help me with installing it directly that would be great.

[pdurbin]

@kcondon sure, I just built the guides at http://guides.dataverse.org/en/5288-dependency-housekeeping/developers/dependencies.html using https://build.hmdc.harvard.edu:8443/job/guides.dataverse.org-5288-dependency-housekeeping/ but the nice images don't appear (they look great on my Mac and on @dlmurphy 's Windows laptop). It looks like this instead:

I emailed Ops asking them if they can yum install graphviz for us in https://help.hmdc.harvard.edu/Ticket/Display.html?id=269424

@michbarsinai you might be interested in all of this as the author of "Draw More, Work Less": http://www.mbarsinai.com/blog/2014/01/12/draw-more-work-less/ . Are PlantUML's days numbered?!? 😄

[kcondon]

@pdurbin Thanks! I was also looking at downloading rpm and trying to install it myself: http://rpmfind.net/linux/centos/6.10/os/x86_64/Packages/graphviz-2.26.0-10.el6.x86_64.rpm

[pdurbin]

Emily from Ops installed graphviz for us (thanks!) and the images look great:

As before, that's from http://guides.dataverse.org/en/5288-dependency-housekeeping/developers/dependencies.html

[poikilotherm]

@kcondon: whoops. Thanks for catching this. Seems like the compatibility needs more checks than just using the JUnit tests. Oh dear... Do you guys have some testing protocol for the UI?

I am wondering how to avoid leftovers that bite back in the future at somebodys production instance because something was missing from the currently manual tests. :scream: :fearful:

(Should I mention Selenium? /me runs for cover)

[kcondon]

@poikilotherm I was testing the following:

1. basic s3 works (upload files, download, thumbnails)
2. basic email notification works (create new user, check inbox for welcome email)
3. basic smoke test (create user, create dataverse, create dataset, upload files (txt, img, dta) publish, download files, verify all appears in UI, do all this while tail -F server.log, watching for errors.

I caught this error after basic testing was nearly done, just went back to homepage, clicked on existing dataset card.

We have been discussing Selenium.

My generate testing approach: test specific changes, test affected/related functionality at basic level, basic smoke test.

[poikilotherm]

Alright, digging into this. Primary reason is the S3 stuff, as seen above:

Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper; at com.amazonaws.partitions.PartitionsLoader.(PartitionsLoader.java:54)

This is caused by the old Jackson API v2.3 bundled with Glassfish 4.1, that misses the JsonParser.Feature compatibility introduced in v2.4. Will look into how to get around this, as v2.9.3 is shipped with the WAR.

EDIT: Well... Kinda stuck with this, circle dependencies ahead. Trying to use Jackson v2.9 from the WAR via

<class-loader delegate="false"/>

in glassfish-web.xml. Breaks deployment, seems like the Jersey 2.23 from the WAR is not compatible with the rest of Glassfish 4.1 (hitting java.lang.ClassCastException: Cannot cast org.glassfish.jersey.gf.cdi.internal.CdiComponentProvider to org.glassfish.jersey.server.spi.ComponentProvider exceptions...)

[poikilotherm]

Payara 4.1.2.181 might be a possible workaround for this. It includes updated dependencies like Jackson 2.9.4.

Please be aware that Payara 4.x is not receiving updates within the Community Channel, only for paid subscriptions. As 4.x is dead upstream, does not include Java EE 8, etc. IMHO we should opt for going to 5.x. I can't decide about product strategy: changing an application server twice between releases in a rather short time interval could leave a negative impression for customers.

Going for a bigger change chunk inline with a 5.x upgrade might be a better option. And of course there are other application servers around. @pdurbin pretty much disliked that the community channel of 5.x will receive security updated quarterly most of the time.

See also: https://blog.payara.fish/payara-5-and-payara-4-development-changes

[pdurbin]

@matthew-a-dunlap and I just talked through the diagram from the https://blog.payara.fish/payara-5-and-payara-4-development-changes which I'll put below:

We also discussed https://docs.payara.fish/v/181/security/security.html especially the following quote:

"Reported security vulnerabilities by the community or Payara Services Limited’s costumers are patched and released either in specific patch releases (for paid support customers exclusively) or quarterly releases. In some cases, we might release special hotfixes to the community to patch serious vulnerabilities that cannot wait for a quarterly release to be fixed."

I'm glad to hear that hotfixes might be made available to the community for free to address serious security vulnerabilities. I'd rather see "will be" than "might be" though. 😄 PrimeFaces definitely holds back security fixes from their free community version (see "possible fee" at https://github.com/IQSS/dataverse/issues/3883#issuecomment-323120605 ). Solr does not.

[poikilotherm]

Hey @pdurbin could you please take this issue out of Code Review? This issue is no longer addressed by #5289 as long as we are not using Glassfish 5+ or similar.

[pdurbin]

@poikilotherm sure, I moved it to Community Dev and assigned it to you. If there's a better place, please let me know. Thanks!