Bug - sbt-assembly produces entirely different JAR with and without a test dependency

[oripwk]

I have a Scala project with sbt-assembly plugin. There's a test dependency (com.google.cloud:google-cloud-nio:0.123.1) which causes the plugin to generate a JAR with different content.

sbt version: 1.3.13 sbt-assembly version: 0.14.10 Scala version: 2.12.11

We're seeing different files in both JARs, and also existing files have different contents.

This is the repository for reproducing the issue: https://github.com/oripwk/sbt-assembly-reproduce

Steps to reproduce:

1. Build with "com.google.cloud" % "google-cloud-nio" % "0.123.1" % Test
2. Build without "com.google.cloud" % "google-cloud-nio" % "0.123.1" % Test
3. Extract the JARs to separate directories
4. Generate a list of all files and their MD5s for each of the dirs: find . -type f -exec md5 {} \; > md5s
5. Now compare the md5s files for both versions with a comparison tool of your choice.

I used vim diff and attaching the screenshots.

This is causing our production application to crash on a dependency conflict

What's going on here?

name := "sbt-assembly-reproduce"
scalaVersion := "2.12.11"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-sql" % "2.4.7" % Provided,
  "org.asynchttpclient" % "async-http-client" % "2.12.1" % Compile,
  "com.google.cloud" % "google-cloud-datastore" % "1.105.3" % Compile,
  "org.apache.hadoop" % "hadoop-aws" % "2.7.3" % Provided,
  "com.github.scopt" %% "scopt" % "3.7.1" % Compile,
  "org.scala-lang.modules" %% "scala-java8-compat" % "0.9.1" % Compile,
  "org.glassfish.jersey.core" % "jersey-common" % "2.22.2" % Compile,
  "org.brotli" % "dec" % "0.1.2" % Compile,
  "commons-codec" % "commons-codec" % "1.10",
  "com.typesafe.scala-logging" %% "scala-logging" % "3.9.2" % Compile,
  "commons-io" % "commons-io" % "2.6" % Compile,
  "com.helger" % "ph-css" % "6.2.3" % Compile,
  // Comment this line out
  "com.google.cloud" % "google-cloud-nio" % "0.123.1" % Test,
  "org.scalatest" %% "scalatest" % "3.1.2" % Test,
  "org.mockito" %% "mockito-scala-scalatest" % "1.14.8" % Test,
  "com.github.tomakehurst" % "wiremock" % "2.27.2" % Test,
  "org.gaul" % "s3proxy" % "1.7.1" % Test
)

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", "services", _*) => MergeStrategy.concat
  case PathList("META-INF", "native", _*) => MergeStrategy.singleOrError
  case PathList("META-INF", _*) => MergeStrategy.discard
  case _ => MergeStrategy.first
}

[eed3si9n]

Could you try with the latest stable sbt 1.5.4 please? This was a known problem with Coursier that leaked Test configuration into the dependency graph of Compile (https://github.com/coursier/coursier/issues/1382), and it was fixed in sbt 1.4.0 - https://github.com/sbt/sbt/pull/5575

[eed3si9n]

Also instead of MergeStrategy.first generally we recommend using the default merge strategy as the fallback. https://github.com/sbt/sbt-assembly#merge-strategy

[oripwk]

Could you try with the latest stable sbt 1.5.4 please? This was a known problem with Coursier that leaked Test configuration into the dependency graph of Compile (coursier/coursier#1382), and it was fixed in sbt 1.4.0 - sbt/sbt#5575

It fixed the problem. Thanks.

[eed3si9n]

cool. I'm closing this issue then.