`mvn integration-test site` fails enforce-ban-duplicate-classes

[cburroughs]

mvn integration-test site
...<snip>...
[INFO] ------------------------------------------------------------------------
[INFO] Building java-manta-client-kryo-serialization 3.1.2-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-bytecode-version) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-ban-circular-dependencies) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-ban-duplicate-classes) @ java-manta-client-kryo-serialization ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
Duplicate classes found:

  Found in:
    com.joyent.manta:java-manta-client:jar:3.1.2-SNAPSHOT:compile
    com.twmacinta:fast-md5:jar:2.7.1:compile
  Duplicate classes:
    com/twmacinta/util/MD5State.class
    com/twmacinta/util/MD5InputStream.class
    com/twmacinta/util/MD5.class

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] java-manta ......................................... SUCCESS [ 11.936 s]
[INFO] java-manta-client .................................. SUCCESS [ 40.718 s]
[INFO] java-manta-client-kryo-serialization ............... FAILURE [  0.254 s]
[INFO] java-manta-cli ..................................... SKIPPED
[INFO] java-manta-benchmark ............................... SKIPPED
[INFO] java-manta-it ...................................... SKIPPED
[INFO] java-manta-examples ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 53.091 s
[INFO] Finished at: 2017-06-23T17:23:41-04:00
[INFO] Final Memory: 81M/761M

I've tried to reduce this to a simpler set of goals to no avail.

[cburroughs]

(I should also point out that I'm unsure if this is a "real" problem in the final jars or a strange corner case.)

[cburroughs]

Possible related, I see the following warning when doing mvn -DskipTests=true package while shading the benchmark jar:

[WARNING] java-manta-client-3.1.3-SNAPSHOT.jar, fast-md5-2.7.1.jar define 3 overlapping classes: 
[WARNING]   - com.twmacinta.util.MD5State
[WARNING]   - com.twmacinta.util.MD5InputStream
[WARNING]   - com.twmacinta.util.MD5
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/

[cburroughs]

In general we shade and relocate dependencies for the primary java-manta-client jar. So com.foo.Foo becomes com.joyent.manta.com.foo.Foo. This eliminates conflicts between the library versions java-manta uses, and what downstream applications are using.

The case of com.twmacinta.fast-md5 is complicated. We are shipping a class in the com.twmacinta package space to access package private classes. We can't relocate the package as that would relocate our own class, nor can we shade everything in that package but our class since the class would then loose the private access it needs.

Within the multi-module project I don't have a clear model for when a dependency on java-manta-client means: (A) give me those client classes (B) give me those client classes and their transitive dependencies (C) give me those client classes and their shaded transitive dependencies (D) give me those client classes, their their transitive dependencies, & their shaded transitive dependencies. I am also unsure if the answer changes based on the goal.

Adding a bunch of

                <exclusion>
                    <groupId>com.twmacinta</groupId>
                    <artifactId>fast-md5</artifactId>
                </exclusion>

to the intra-model dependencies appears to resolve all issues. I don't think that is harmful, but it is also not entirely satisfying.

@dekobon Do you have an historical context or advice on how to proceed?

[dekobon]

I can't seem to reproduce this error. In fact, I can't seem to make the enforce plugin run at all now:

 ✘ elijah@Elijahs-MacBook-Pro  ~/Development/Joyent/java-manta   master ●  mvn enforcer:enforce
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] java-manta
[INFO] java-manta-client
[INFO] java-manta-client-kryo-serialization
[INFO] java-manta-cli
[INFO] java-manta-benchmark
[INFO] java-manta-it
[INFO] java-manta-examples
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building java-manta 3.1.3-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (default-cli) @ java-manta ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] java-manta ......................................... FAILURE [  0.519 s]
[INFO] java-manta-client .................................. SKIPPED
[INFO] java-manta-client-kryo-serialization ............... SKIPPED
[INFO] java-manta-cli ..................................... SKIPPED
[INFO] java-manta-benchmark ............................... SKIPPED
[INFO] java-manta-it ...................................... SKIPPED
[INFO] java-manta-examples ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.902 s
[INFO] Finished at: 2017-06-28T14:04:37-07:00
[INFO] Final Memory: 10M/245M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (default-cli) on project java-manta: The parameters 'rules' for goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce are missing or invalid -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginParameterException

[cburroughs]

So `mvn integration-test site does not induce an error on your workstation? Does the behavior change if you clean and rm -r ~/.m2/repository/com/joyent/?

My understanding is that we can't enforcer:enforce due to subtleties in how the pom is laid out: https://stackoverflow.com/a/6760365

[dekobon]

I just tried again after doing rm -r ~/.m2/repository/com/joyent/. It still succeeded for me on my Mac. Can you reproduce this on a cloud instance? If so, I can ssh in and take a look.

[cburroughs]

Cloud steps:

$ eval "$(triton env jpc-east-1)"
$ triton instance create ubuntu-16.04 g4-highcpu-1G -n mvndemo -w
Creating instance mvndemo (b521c6ac-c09d-6679-dfb3-ddd63b36e2bf, ubuntu-16.04@20170403)
Created instance mvndemo (b521c6ac-c09d-6679-dfb3-ddd63b36e2bf) in 18s
 $ triton ssh mvndemo
# apt-get update
# apt-get upgrade
# apt-get install openjdk-8-jre-headless openjdk-8-jdk-headless maven git
# git clone https://github.com/joyent/java-manta.git
# cd java-manta/
# mvn integration-test site
<MAVEN DOWNLOADS THE INTERNET>
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-bytecode-version) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-ban-circular-dependencies) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-ban-duplicate-classes) @ java-manta-client-kryo-serialization ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
Duplicate classes found:

  Found in:
    com.joyent.manta:java-manta-client:jar:3.1.4-SNAPSHOT:compile
    com.twmacinta:fast-md5:jar:2.7.1:compile
  Duplicate classes:
    com/twmacinta/util/MD5State.class
    com/twmacinta/util/MD5InputStream.class
    com/twmacinta/util/MD5.class

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] java-manta ......................................... SUCCESS [ 52.365 s]
[INFO] java-manta-client .................................. SUCCESS [02:03 min]
[INFO] java-manta-client-kryo-serialization ............... FAILURE [  2.100 s]
[INFO] java-manta-cli ..................................... SKIPPED
[INFO] java-manta-benchmark ............................... SKIPPED
[INFO] java-manta-it ...................................... SKIPPED
[INFO] java-manta-examples ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:58 min
[INFO] Finished at: 2017-06-30T15:27:01+00:00
[INFO] Final Memory: 62M/162M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (enforce-ban-duplicate-classes) on project java-manta-client-kryo-serialization: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :java-manta-client-kryo-serialization

Your keys listed on github should get you into 165.225.139.167

[dekobon]

Dumping the dependency tree for java-manta-client-kryo-serialization doesn't show a direct dependency on com.twmacinta:fast-md5:

$ mvn dependency:tree
[INFO] ------------------------------------------------------------------------
[INFO] Building java-manta-client-kryo-serialization 3.1.4-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.10:tree (default-cli) @ java-manta-client-kryo-serialization ---
[INFO] com.joyent.manta:java-manta-client-kryo-serialization:jar:3.1.4-SNAPSHOT
[INFO] +- com.joyent.manta:java-manta-client:jar:3.1.4-SNAPSHOT:compile
[INFO] |  +- org.bouncycastle:bcpkix-jdk15on:jar:1.56:compile
[INFO] |  +- org.bouncycastle:bcprov-jdk15on:jar:1.56:compile
[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.21:compile
[INFO] |  \- io.mikael:urlbuilder:jar:2.0.8:compile
[INFO] +- org.apache.commons:commons-lang3:jar:3.4:provided
[INFO] +- org.apache.commons:commons-collections4:jar:4.1:test
[INFO] +- commons-io:commons-io:jar:2.5:test
[INFO] +- com.esotericsoftware:kryo:jar:4.0.0:compile
[INFO] |  \- com.esotericsoftware:reflectasm:jar:1.11.3:compile
[INFO] |     \- org.ow2.asm:asm:jar:5.0.4:compile
[INFO] +- com.dorkbox:MinLog-SLF4J:jar:1.8:compile
[INFO] +- org.objenesis:objenesis:jar:2.2:compile
[INFO] +- ch.qos.logback:logback-classic:jar:1.1.7:test
[INFO] |  \- ch.qos.logback:logback-core:jar:1.1.7:test
[INFO] +- org.testng:testng:jar:6.9.10:test
[INFO] |  +- com.beust:jcommander:jar:1.48:test
[INFO] |  \- org.beanshell:bsh:jar:2.0b4:test
[INFO] +- org.slf4j:slf4j-api:jar:1.7.21:compile
[INFO] \- org.mockito:mockito-core:jar:2.0.2-beta:test
[INFO]    \- org.hamcrest:hamcrest-core:jar:1.1:test
...

I also inspected all of the pom.xml files and I can't find a direct reference to that dependency.

[dekobon]

I can keep the error from happening by doing:

<ignoreClasses>
    <ignoreClass>com.twmacinta.util.*</ignoreClass>
</ignoreClasses>

However, that doesn't address the root cause. It seems as though something is getting confused within the enforcer plugin. What is notable is that this is the only class that is being shaded but not relocated.

[dekobon]

I tried mvn integration-test site and it failed. I tried mvn integration-test and it worked. I tried mvn site and that worked. I also tried mvn install site and that failed.I suspect that the combination of doing integration-test / install and site somehow confuses maven. I've seen this with other plugins before.

Many times when I'm setting up CI for Java, I will run different Maven goals in different stages because it helps to avoid this kind of weirdness. Most typically, I only run mvn site after mvn install has completed because a lot of the report plugins are not well maintained and they don't know how to manage unrealized dependency state.

I see two fixes: 1. Suppress the check for the specific class in question. I've already verified that it isn't duplicated in any of the final JARs. 2. Leave it as is and make a note that site should be run after running install or integration-test.

[indianwhocodes]

Latest Update on this issue since I am tracking it:

[WARNING] Rule 0: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
Duplicate classes found:

  Found in:
    com.joyent.util:fast-md5:jar:2.7.1:compile
    com.joyent.manta:java-manta-client:jar:3.3.2-SNAPSHOT:compile
  Duplicate classes:
    com/twmacinta/io/NullOutputStream.class
    com/twmacinta/util/MD5InputStream.class
    com/twmacinta/util/MD5.class
    com/twmacinta/util/MD5State.class
    com/twmacinta/util/MD5OutputStream.class

  Found in:
    com.joyent.manta:java-manta-client:jar:3.3.2-SNAPSHOT:compile
    com.joyent.manta:java-manta-client-unshaded:jar:3.3.2-SNAPSHOT:compile

[indianwhocodes]

PR #285 will have to be merged safely due to merge-conflicts.