search

sbt / sbt-assembly

Deploy über-JARs. Restart processes. (port of codahale/assembly-sbt)
MIT License
1.95k stars 224 forks source link

Ignore module-info.class #391

Open daggerok opened 4 years ago

daggerok commented 4 years ago

I have akka scala sbt multi project.

After adding akka-serialization-jackson dependency, assembly causing error:

[error] 1 error was encountered during merge
[error] java.lang.RuntimeException: deduplicate: different file contents found in the following:
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.10.3/jackson-annotations-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.10.3/jackson-core-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.10.3/jackson-databind-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.10.3/jackson-dataformat-cbor-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/datatype/jackson-datatype-jdk8/2.10.3/jackson-datatype-jdk8-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.10.3/jackson-datatype-jsr310-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-module-parameter-names/2.10.3/jackson-module-parameter-names-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-module-paranamer/2.10.3/jackson-module-paranamer-2.10.3.jar:module-info.class
[error]         at sbtassembly.Assembly$.applyStrategies(Assembly.scala:143)
[error]         at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
[error]         at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
[error]         at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
[error]         at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
[error]         at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:68)
[error]         at sbtassembly.Assembly$.inputs$1(Assembly.scala:58)
[error]         at sbtassembly.Assembly$.apply(Assembly.scala:85)
[error]         at sbtassembly.Assembly$.$anonfun$assemblyTask$1(Assembly.scala:244)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error]         at sbt.std.Transform$$anon$4.work(Transform.scala:67)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:281)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:19)
[error]         at sbt.Execute.work(Execute.scala:290)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:281)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] (assembly) deduplicate: different file contents found in the following:
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.10.3/jackson-annotations-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.10.3/jackson-core-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.10.3/jackson-databind-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.10.3/jackson-dataformat-cbor-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/datatype/jackson-datatype-jdk8/2.10.3/jackson-datatype-jdk8-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.10.3/jackson-datatype-jsr310-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-module-parameter-names/2.10.3/jackson-module-parameter-names-2.10.3.jar:module-info.class
[error] /Users/mkostromin/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-module-paranamer/2.10.3/jackson-module-paranamer-2.10.3.jar:module-info.class
[error] Total time: 5 s, completed Mar 20, 2020 2:51:34 AM

Probably I'm too stupid to understand, but this guide doesn't helped me fix problem and build fat jar.

I have tried different combinations, like this: image ...but was not successful

steps to reproduce

# clone assembly branch:
git clone https://github.com/daggerok/akka-stream-playground -b assembly
# and run assembly task:
cd akka-stream-playground && ./sbtw "project typed-akka-production-ready-api-scala-example" assembly

Please, explain how to understand, investigate and finally solve problem. Thank you in advice

Regards, Max

eed3si9n commented 4 years ago

https://github.com/sbt/sbt-assembly/blob/master/CONTRIBUTING.md#where-to-file-a-bug-report

For questions and specific support issues use stackoverflow with tags sbt and sbt-assembly.

eed3si9n commented 4 years ago

Looking at this more carefully, maybe the problem is sbt-assembly not working well with Java Module's module-info.class file.

daggerok commented 4 years ago

Don't know if it can help, but here are warnings I can see for other project with similar sets of dependencies with maven-shade-plugin:

[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] akka-actor-typed_2.13-2.6.4.jar, akka-actor_2.13-2.6.4.jar, akka-serialization-jackson_2.13-2.6.4.jar define 1 overlapping resources: 
[WARNING]   - reference.conf
[WARNING] jackson-core-2.10.3.jar, jackson-databind-2.10.3.jar, jackson-module-paranamer-2.10.3.jar define 1 overlapping resources: 
[WARNING]   - META-INF/NOTICE
[WARNING] jackson-datatype-jdk8-2.10.3.jar, jackson-datatype-jsr310-2.10.3.jar, jackson-module-parameter-names-2.10.3.jar, jackson-module-paranamer-2.10.3.jar, jackson-module-scala_2.13-2.10.3.jar define 1 overlapping resources: 
[WARNING]   - META-INF/services/com.fasterxml.jackson.databind.Module
[WARNING] akka-actor-typed-java-maven-example-1.0-SNAPSHOT.jar, akka-actor-typed_2.13-2.6.4.jar, akka-actor_2.13-2.6.4.jar, akka-serialization-jackson_2.13-2.6.4.jar, akka-slf4j_2.13-2.6.4.jar, config-1.4.0.jar, jackson-annotations-2.10.3.jar, jackson-core-2.10.3.jar, jackson-databind-2.10.3.jar, jackson-dataformat-cbor-2.10.3.jar, jackson-datatype-jdk8-2.10.3.jar, jackson-datatype-jsr310-2.10.3.jar, jackson-module-parameter-names-2.10.3.jar, jackson-module-paranamer-2.10.3.jar, jackson-module-scala_2.13-2.10.3.jar, logback-classic-1.2.3.jar, logback-core-1.2.3.jar, paranamer-2.8.jar, scala-java8-compat_2.13-0.9.0.jar, scala-library-2.13.1.jar, slf4j-api-1.7.30.jar define 1 overlapping resources: 
[WARNING]   - META-INF/MANIFEST.MF
[WARNING] jackson-core-2.10.3.jar, jackson-dataformat-cbor-2.10.3.jar define 1 overlapping resources: 
[WARNING]   - META-INF/services/com.fasterxml.jackson.core.JsonFactory
[WARNING] jackson-annotations-2.10.3.jar, jackson-core-2.10.3.jar, jackson-databind-2.10.3.jar, jackson-datatype-jsr310-2.10.3.jar, jackson-module-paranamer-2.10.3.jar, jackson-module-scala_2.13-2.10.3.jar define 1 overlapping resources: 
[WARNING]   - META-INF/LICENSE
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
[INFO] Attaching shaded artifact.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.016 s
[INFO] Finished at: 2020-03-20T18:21:49+02:00

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  <modelVersion>4.0.0</modelVersion>
  <groupId>org.example</groupId>
  <artifactId>akka-actor-typed-java-maven-example</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>

  <properties>
    <java.version>1.8</java.version>
    <maven.compiler.source>${java.version}</maven.compiler.source>
    <maven.compiler.target>${java.version}</maven.compiler.target>

    <encoding>UTF-8</encoding>
    <project.build.sourceEncoding>${encoding}</project.build.sourceEncoding>
    <project.reporting.outputEncoding>${encoding}</project.reporting.outputEncoding>

    <akka.version>2.6.4</akka.version>
    <scala.version>2.13</scala.version>
    <logback.version>1.2.3</logback.version>
    <maven-shade-plugin.version>3.2.2</maven-shade-plugin.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>com.typesafe.akka</groupId>
      <artifactId>akka-actor-typed_${scala.version}</artifactId>
      <version>${akka.version}</version>
    </dependency>
    <dependency>
      <groupId>com.typesafe.akka</groupId>
      <artifactId>akka-serialization-jackson_${scala.version}</artifactId>
      <version>${akka.version}</version>
    </dependency>
    <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-classic</artifactId>
      <version>${logback.version}</version>
    </dependency>

    <dependency>
      <groupId>com.typesafe.akka</groupId>
      <artifactId>akka-actor-testkit-typed_${scala.version}</artifactId>
      <version>${akka.version}</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <build>
    <defaultGoal>clean package</defaultGoal>

    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>${maven-shade-plugin.version}</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals><goal>shade</goal></goals>
            <configuration>
              <shadedArtifactAttached>true</shadedArtifactAttached>
              <shadedClassifierName>all</shadedClassifierName>
              <artifactSet>
                <includes>
                  <include>*:*</include>
                </includes>
              </artifactSet>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                  <resource>reference.conf</resource>
                </transformer>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <manifestEntries>
                    <Main-Class>daggerok.Main</Main-Class>
                  </manifestEntries>
                </transformer>
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

</project>

Regards, Maksim

SethTisue commented 4 years ago

https://stackoverflow.com/a/60114988/86485 shows how to use MergeStrategy to work around this

anupash commented 4 years ago

I completely agree with @daggerok that none of the mentioned solutions work.

Here is my assembly strategy

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case "application.prod.conf" => MergeStrategy.concat
  // https://stackoverflow.com/questions/54834125/sbt-assembly-deduplicate-module-info-class
  case PathList("META-INF", "versions", "9", "module-info.class") => MergeStrategy.discard
  case "module-info.class" => MergeStrategy.discard
  case x =>
    val oldStrategy: String => MergeStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

and I have very similar error 

[error] java.lang.RuntimeException: deduplicate: different file contents found in the following:
[error] /home/aashish/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/classgraph/classgraph/4.8.65/classgraph-4.8.65.jar:META-INF/versions/9/module-info.class
[error] /home/aashish/.cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.11.2/log4j-api-2.11.2.jar:META-INF/versions/9/module-info.class
[error] deduplicate: different file contents found in the following:
[error] /home/aashish/.cache/coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.11.2/jackson-annotations-2.11.2.jar:module-info.class
[error] /home/aashish/.cache/coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.11.2/jackson-core-2.11.2.jar:module-info.class
[error] /home/aashish/.cache/coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.11.2/jackson-databind-2.11.2.jar:module-info.class
[error] /home/aashish/.cache/coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.11.1/jackson-dataformat-yaml-2.11.1.jar:module-info.class
[error] /home/aashish/.cache/coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.11.1/jackson-datatype-jsr310-2.11.1.jar:module-info.class
[error] /home/aashish/.cache/coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/jaxrs/jackson-jaxrs-base/2.11.1/jackson-jaxrs-base-2.11.1.jar:module-info.class
[error] /home/aashish/.cache/coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/jaxrs/jackson-jaxrs-json-provider/2.11.1/jackson-jaxrs-json-provider-2.11.1.jar:module-info.class
[error] /home/aashish/.cache/coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-module-jaxb-annotations/2.11.1/jackson-module-jaxb-annotations-2.11.1.jar:module-info.class
[error] /home/aashish/.cache/coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-module-paranamer/2.11.2/jackson-module-paranamer-2.11.2.jar:module-info.class
[error] /home/aashish/.cache/coursier/v1/https/repo1.maven.org/maven2/jakarta/xml/bind/jakarta.xml.bind-api/2.3.2/jakarta.xml.bind-api-2.3.2.jar:module-info.class
[error]         at sbtassembly.Assembly$.applyStrategies(Assembly.scala:143)
[error]         at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
[error]         at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
[error]         at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
[error]         at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
[error]         at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:68)
[error]         at sbtassembly.Assembly$.inputs$1(Assembly.scala:58)
[error]         at sbtassembly.Assembly$.apply(Assembly.scala:85)
[error]         at sbtassembly.Assembly$.$anonfun$assemblyTask$1(Assembly.scala:244)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error]         at sbt.std.Transform$$anon$4.work(Transform.scala:67)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:281)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:19)
[error]         at sbt.Execute.work(Execute.scala:290)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:281)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)
necosta commented 3 years ago

Also stuck on this one. Using sbt-assembly version 0.15.0. Dependency tree shows only one usage of library. Any help would be much appreciated

deekayman commented 3 years ago

This is pretty sad. The internet is filled with the same problem: sbt's assemblyMergeStrategy does NOT work with certain file names. I've tried literally every single permutation of assemblyMergeStrategy. Nothing works.

ry0nw commented 3 years ago

@deekayman Have you tried different sbt versions?

sbt.version = 1.3.1 causes the issue for me. sbt.version = 1.4.1 fixes it.

asadhat commented 3 years ago

@ry0nw I tried with 1.4.7 Getting the same issue as given by @anupash

SethTisue commented 3 years ago

There was a Reddit thread on this recently that might have some solutions/workarounds that might be helpful to some: https://www.reddit.com/r/scala/comments/m335hi/how_to_handle_sbt_deduplication_errors_on/

adamrabung commented 3 years ago

FWIW, I needed to add the assemblyMergeStrategy directly to the subproject I wanted to run assemble on. Adding assemblyMergeStrategy as a top level key did not work.

LazyBun commented 3 years ago

Thanks @adamrabung, I've been trying to resolve this issue for quite a while and I couldn't figure out why this was happening. Adding the merge strategy directly to java subproject fixes this issue.

What I do not understand is why that fixes this issue. Why does assemblyMergeStrategy as top level work for my scala subprojects but not for java subproject?

Change I've made:

val javaModule = project
  .in(file("modules/java-subproject"))
  .settings(
    name := "java-subproject",
    commonJavaSettings
  )

into

val javaModule = project
  .in(file("modules/java-subproject"))
  .settings(
    name := "java-subproject",
    commonJavaSettings,
    assemblyMergeStrategy in assembly := [..(my exceptions)..]
  )
mrt181 commented 2 years ago

I just faced this issue with jackson and this is what resolved it for me

    assembly / assemblyMergeStrategy := {
      case PathList("module-info.class")                                 => MergeStrategy.discard
      case PathList("META-INF", "versions", xs @ _, "module-info.class") => MergeStrategy.discard
ches commented 1 year ago

module-info can contain service provider declarations, so simply discarding might not be such an acceptable workaround. You might depend on a concrete implementation of a service API, and upon deployment face errors that the implementation cannot be found.

(This is in a world where your deployment target is a Java 9+ runtime, and library authors use Java 9+ features like service providers via modules. In 2023 the latter has finally begun to enter reality).

Ref: Deploying service providers as modules

LyndonArmitage commented 8 months ago

module-info can contain service provider declarations, so simply discarding might not be such an acceptable workaround. You might depend on a concrete implementation of a service API, and upon deployment face errors that the implementation cannot be found.

(This is in a world where your deployment target is a Java 9+ runtime, and library authors use Java 9+ features like service providers via modules. In 2023 the latter has finally begun to enter reality).

Ref: Deploying service providers as modules

The ServiceLoader mechanism is used in SLF4J versions 2+.

This means discarding these classes renders logging with modern SLF4J impossible.

In fact, many libraries use ServiceLoader as described in this blog post. From Jackson to JDBC.

For example, in my current project, I am building a fat-jar that is deployed into an AWS Lambda.
I can either not have any logging, by using this:

    case PathList("module-info.class")               => MergeStrategy.discard
    case path if path.endsWith("/module-info.class") => MergeStrategy.discard

or my build fails with:

[error] Deduplicate found different file contents in the following:
[error]   Jar name = logback-classic-1.4.14.jar, jar org = ch.qos.logback, entry target = module-info.class
[error]   Jar name = logback-core-1.4.14.jar, jar org = ch.qos.logback, entry target = module-info.class
[error]   Jar name = jackson-annotations-2.12.7.jar, jar org = com.fasterxml.jackson.core, entry target = module-info.class
[error]   Jar name = jackson-core-2.12.7.jar, jar org = com.fasterxml.jackson.core, entry target = module-info.class
[error]   Jar name = jackson-databind-2.12.7.1.jar, jar org = com.fasterxml.jackson.core, entry target = module-info.class
[error]   Jar name = jackson-dataformat-cbor-2.12.6.jar, jar org = com.fasterxml.jackson.dataformat, entry target = module-info.class
[error] Total time: 6 s, completed 6 Feb 2024, 17:25:33

In my example, I don't even explicitly include Jackson libraries. My other alternative is to downgrade the sl4j libraries to 1.7.x, which is a version that has not been updated since Feb 08, 2022. EDIT: In the instance of SLF4J there is also the slf4j.provider system property that can be set that will override the ServiceLoader system.

Is there not some kind of merge strategy or specific functionality that can be implemented specifically for module-info?

malliina commented 2 months ago

module-info can contain service provider declarations, so simply discarding might not be such an acceptable workaround. You might depend on a concrete implementation of a service API, and upon deployment face errors that the implementation cannot be found. (This is in a world where your deployment target is a Java 9+ runtime, and library authors use Java 9+ features like service providers via modules. In 2023 the latter has finally begun to enter reality). Ref: Deploying service providers as modules

The ServiceLoader mechanism is used in SLF4J versions 2+.

This means discarding these classes renders logging with modern SLF4J impossible.

In fact, many libraries use ServiceLoader as described in this blog post. From Jackson to JDBC.

For example, in my current project, I am building a fat-jar that is deployed into an AWS Lambda. I can either not have any logging, by using this:

    case PathList("module-info.class")               => MergeStrategy.discard
    case path if path.endsWith("/module-info.class") => MergeStrategy.discard

or my build fails with:

[error] Deduplicate found different file contents in the following:
[error]   Jar name = logback-classic-1.4.14.jar, jar org = ch.qos.logback, entry target = module-info.class
[error]   Jar name = logback-core-1.4.14.jar, jar org = ch.qos.logback, entry target = module-info.class
[error]   Jar name = jackson-annotations-2.12.7.jar, jar org = com.fasterxml.jackson.core, entry target = module-info.class
[error]   Jar name = jackson-core-2.12.7.jar, jar org = com.fasterxml.jackson.core, entry target = module-info.class
[error]   Jar name = jackson-databind-2.12.7.1.jar, jar org = com.fasterxml.jackson.core, entry target = module-info.class
[error]   Jar name = jackson-dataformat-cbor-2.12.6.jar, jar org = com.fasterxml.jackson.dataformat, entry target = module-info.class
[error] Total time: 6 s, completed 6 Feb 2024, 17:25:33

In my example, I don't even explicitly include Jackson libraries. My other alternative is to downgrade the sl4j libraries to 1.7.x, which is a version that has not been updated since Feb 08, 2022. EDIT: In the instance of SLF4J there is also the slf4j.provider system property that can be set that will override the ServiceLoader system.

Is there not some kind of merge strategy or specific functionality that can be implemented specifically for module-info?

I had similar slf4j-related problems with the following errors:

[error] 3 error(s) were encountered during the merge:
[error] stack trace is suppressed; run last assembly for the full output
[error] (assembly) 
[error] Deduplicate found different file contents in the following:
[error]   Jar name = logback-classic-1.5.6.jar, jar org = ch.qos.logback, entry target = module-info.class
[error]   Jar name = logback-core-1.5.6.jar, jar org = ch.qos.logback, entry target = module-info.class
[error] Deduplicate found different file contents in the following:
[error]   Jar name = okio-jvm-3.6.0.jar, jar org = com.squareup.okio, entry target = META-INF/okio.kotlin_module
[error]   Jar name = okio-3.6.0.jar, jar org = com.squareup.okio, entry target = META-INF/okio.kotlin_module
[error] Deduplicate found different file contents in the following:
[error]   Jar name = kotlin-stdlib-jdk7-1.9.10.jar, jar org = org.jetbrains.kotlin, entry target = META-INF/versions/9/module-info.class
[error]   Jar name = kotlin-stdlib-jdk8-1.9.10.jar, jar org = org.jetbrains.kotlin, entry target = META-INF/versions/9/module-info.class
[error]   Jar name = kotlin-stdlib-1.9.10.jar, jar org = org.jetbrains.kotlin, entry target = META-INF/versions/9/module-info.class
[error]   Jar name = slf4j-api-2.0.13.jar, jar org = org.slf4j, entry target = META-INF/versions/9/module-info.class
[error] Total time: 2 s, completed 30 Jul 2024, 22.26.34

Please observe the logback jar entries in the beginning of the error message, and the slf4j jar at the end. I managed to get logging to work as follows:

assembly / assemblyMergeStrategy := {
  case PathList("module-info.class") => MergeStrategy.first
  case PathList("META-INF", "versions", "9", "module-info.class") => MergeStrategy.last
  case PathList("META-INF", xs @ _*) => MergeStrategy.first
  case x =>
    val oldStrategy = (ThisBuild / assemblyMergeStrategy).value
    oldStrategy(x)
}

I could also get the build errors to go away with .discard, but that indeed then broke slf4j at runtime as it did not appear to find any logging provider. But with the above assemblyMergeStrategy I managed to get everything to work.