search

scalacenter / sbt-dependency-submission

A Github Action to submit the dependency graph of an sbt build to the Dependency Submission API
Apache License 2.0
61 stars 17 forks source link

sbt-dependency-submission resolving wrong versions? #109

Closed mdedetrich closed 1 year ago

mdedetrich commented 1 year ago

At Pekko we have added sbt-dependency-submission to our project (see https://github.com/apache/incubator-pekko/pull/366) however we are getting an interesting/odd bug where although the plugin is submitting the dependencies, the version of sbt dependencies appear to be off?

Incase you can't see the dependabot alerts, here is an example

image

The thing is, the project is currently using Jackson 2.14.3 (see https://github.com/apache/incubator-pekko/blob/main/project/Dependencies.scala#L37-L38) and if you checkout the project and run libraryDependencies in the sbt shell 

[IJ]pekko > libraryDependencies
[info] pki / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, com.hierynomus:asn-one:0.5.0, org.slf4j:slf4j-api:1.7.36, org.scalatest:scalatest:3.1.4:test)
[info] actor-testkit-typed / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, ch.qos.logback:logback-classic:1.2.11:optional;provided;test, junit:junit:4.13.2:optional;provided;test, org.scalatest:scalatest:3.1.4:optional;provided;test, org.scalatestplus:junit-4-13:3.1.4.0:test)
[info] coordination / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test)
[info] cluster-sharding-typed / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10)
[info] actor-typed / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.slf4j:slf4j-api:1.7.36)
[info] docs / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.apache.pekko:pekko-theme-paradox:0.0.0+38-68da3106-SNAPSHOT:paradox-theme, org.scalatest:scalatest:3.1.4:test, junit:junit:4.13.2:test, io.spray:spray-json:1.3.6:test, com.google.code.gson:gson:2.9.1:test, org.iq80.leveldb:leveldb:0.12:optional;provided)
[info] actor-typed-tests / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10)
[info] testkit / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test, io.dropwizard.metrics:metrics-core:4.2.10:test, io.dropwizard.metrics:metrics-jvm:4.2.10:test, org.latencyutils:LatencyUtils:2.0.3:test, org.hdrhistogram:HdrHistogram:2.1.12:test)
[info] stream / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.reactivestreams:reactive-streams:1.0.4, com.typesafe:ssl-config-core:0.4.3, org.scalatest:scalatest:3.1.4:test)
[info] multi-node-testkit / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, io.netty:netty:3.10.6.Final)
[info] persistence-query / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:test, junit:junit:4.13.2:test, commons-io:commons-io:2.11.0:test, org.iq80.leveldb:leveldb:0.12:optional;provided, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided)
[info] osgi / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.osgi:org.osgi.core:6.0.0, org.osgi:org.osgi.compendium:5.0.0, ch.qos.logback:logback-classic:1.2.11:test, commons-io:commons-io:2.11.0:test, com.googlecode.pojosr:de.kalpatec.pojosr.framework:0.2.1:test, org.ops4j.pax.tinybundles:tinybundles:3.0.0:test, org.scalatest:scalatest:3.1.4:test, junit:junit:4.13.2:test)
[info] persistence-typed-tests / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:test, ch.qos.logback:logback-classic:1.2.11:test)
[info] persistence-testkit / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:test, ch.qos.logback:logback-classic:1.2.11:test)
[info] persistence / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.iq80.leveldb:leveldb:0.12:optional;provided, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided, org.scalatest:scalatest:3.1.4:test, org.scalatestplus:junit-4-13:3.1.4.0:test, junit:junit:4.13.2:test, commons-io:commons-io:2.11.0:test, commons-codec:commons-codec:1.15:test)
[info] cluster-tools / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test)
[info] remote-tests / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test, io.netty:netty:3.10.6.Final, io.aeron:aeron-driver:1.38.1, io.aeron:aeron-client:1.38.1)
[info] protobuf-v3 / libraryDependencies
[info]  List(com.google.protobuf:protobuf-java:3.16.1:optional;provided)
[info] stream-tests-tck / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:test, org.scalatestplus:testng-6-7:3.1.4.0:test, org.scalatestplus:scalacheck-1-14:3.1.4.0:test, junit:junit:4.13.2:test, org.reactivestreams:reactive-streams-tck:1.0.4:test)
[info] protobuf / libraryDependencies
[info]  List()
[info] actor / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, com.typesafe:config:1.4.2)
[info] discovery / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test)
[info] bench-jmh / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.openjdk.jmh:jmh-core:1.32, org.openjdk.jmh:jmh-generator-bytecode:1.32, org.openjdk.jmh:jmh-generator-reflection:1.32, ch.qos.logback:logback-classic:1.2.11, org.iq80.leveldb:leveldb:0.12:optional;provided, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided, org.jctools:jctools-core:3.3.0)
[info] cluster-metrics / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, io.kamon:sigar-loader:1.6.6-rev002:optional;provided;test, org.slf4j:jul-to-slf4j:1.7.36:test, org.slf4j:log4j-over-slf4j:1.7.36:test, ch.qos.logback:logback-classic:1.2.11:test, org.scalatestplus:mockito-3-4:3.1.4.0:test)
[info] bill-of-materials / libraryDependencies
[info]  List()
[info] stream-tests / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:test, org.scalatestplus:scalacheck-1-14:3.1.4.0:test, junit:junit:4.13.2:test, commons-io:commons-io:2.11.0:test, com.google.jimfs:jimfs:1.1:test)
[info] stream-testkit / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:test, org.scalatestplus:scalacheck-1-14:3.1.4.0:test, junit:junit:4.13.2:test)
[info] remote / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.agrona:agrona:1.15.1, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test, com.google.jimfs:jimfs:1.1:test, com.google.protobuf:protobuf-java:3.16.1:test, io.netty:netty:3.10.6.Final:optional, io.aeron:aeron-driver:1.38.1:optional, io.aeron:aeron-client:1.38.1:optional)
[info] distributed-data / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.lmdbjava:lmdbjava:0.7.0, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test)
[info] cluster / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test)
[info] actor-tests / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test, org.scalatestplus:junit-4-13:3.1.4.0:test, org.scalatestplus:scalacheck-1-14:3.1.4.0:test, commons-codec:commons-codec:1.15:test, org.apache.commons:commons-math:2.2:test, com.google.jimfs:jimfs:1.1:test, com.spotify:docker-client:8.16.0:test, com.sun.activation:javax.activation:1.2.0:provided;test)
[info] slf4j / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.slf4j:slf4j-api:1.7.36, ch.qos.logback:logback-classic:1.2.11:test)
[info] cluster-typed / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10)
[info] serialization-jackson / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, com.fasterxml.jackson.core:jackson-core:2.14.3, com.fasterxml.jackson.core:jackson-annotations:2.14.3, com.fasterxml.jackson.core:jackson-databind:2.14.3, com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.14.3, com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.14.3, com.fasterxml.jackson.module:jackson-module-parameter-names:2.14.3, com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.14.3, com.fasterxml.jackson.module:jackson-module-scala:2.14.3, org.lz4:lz4-java:1.8.0, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test)
[info] persistence-tck / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.scalatest:scalatest:3.1.4:compile, junit:junit:4.13.2:compile, org.iq80.leveldb:leveldb:0.12:optional;provided, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided)
[info] stream-typed / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10)
[info] persistence-shared / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.iq80.leveldb:leveldb:0.12:optional;provided, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided, ch.qos.logback:logback-classic:1.2.11:test)
[info] persistence-typed / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.iq80.leveldb:leveldb:0.12:optional;provided, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided, ch.qos.logback:logback-classic:1.2.11:test)
[info] cluster-sharding / libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10, org.iq80.leveldb:leveldb:0.12:optional;provided;multi-jvm;test, org.fusesource.leveldbjni:leveldbjni-all:1.8:optional;provided, junit:junit:4.13.2:test, org.scalatest:scalatest:3.1.4:test, commons-io:commons-io:2.11.0:test, site.ycsb:core:0.17.0:test)
[info] libraryDependencies
[info]  List(org.scala-lang:scala-library:2.13.10)
[IJ]pekko >

You can see that all of the jackson versions are being resolved to 2.14.3

adpi2 commented 1 year ago

sbt-dependenyc-submission does not submit only the runtime dependencies but also the dependencies of your build (sbt plugins) and development configurations (Test, ScalaTools, ScalaDocTools). That's why your snapshot contains three dependencies to jackson-databind:

image

sbt:pekko> ++3.3.0
sbt:pekko> show update
...
[info]  scala-doc-tool:
...
[info]      com.fasterxml.jackson.core:jackson-databind:2.13.3:default
...

You should be able to fix the dependabot alert by adding libraryDependency += "com.fasterxml.jackson.core:jackson-databind:2.14.3" in your project/plugins.sbt.

adpi2 commented 1 year ago

As a side note, if you want to ignore the dependencies coming from some configurations you can use the configs-ignore input.

There is no way currently to ignore the dependencies of the build itself though.

mdedetrich commented 1 year ago

sbt-dependenyc-submission does not submit only the runtime dependencies but also the dependencies of your build (sbt plugins) and development configurations (Test, ScalaTools, ScalaDocTools). That's why your snapshot contains three dependencies to jackson-databind

Oh wow, I didn't realize it also grabbed dependencies from sbt plugin itself, I though it was just grabbing dependencies from runtime/compile. Thanks for letting me know, I presume to filter out sbt-plugin dependencies one would use scala-tool and scala-doc-tool?

adpi2 commented 1 year ago

Thanks for letting me know, I presume to filter out sbt-plugin dependencies one would use scala-tool and scala-doc-tool?

scala-tool contains the compiler and its dependencies scala-doc-tool contains scaladoc and its dependencies The sbt-plugin dependencies are the compile dependencies of the meta-build. There is no way currently to ignore dependencies for the meta-build.

adpi2 commented 1 year ago

Actually I am wrong: we don't include the meta-build dependencies in the snapshot.

The jackson-databind:2.9.8 comes from the Test dependencies of actor-tests, coordination and discovery:

pekko > show actor-tests/Test/dependencyTree
[info] org.apache.pekko:pekko-actor-tests_2.13:0.0.0+26699-495dc110+20230612-1344-SNA..
...
[info]   +-com.spotify:docker-client:8.16.0
...
[info]   | +-org.glassfish.jersey.media:jersey-media-json-jackson:2.22.2
...
[info]   | | +-com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:2.9.8
...
[info]   | | | +-com.fasterxml.jackson.core:jackson-databind:2.9.8
...
[info]   | | +-com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.9.8
...
[info]   | | |   +-com.fasterxml.jackson.core:jackson-databind:2.9.8
...

You should be able to override this dependency in your build. Or you can ignore some projects/configs with the modules-ignore or configs-ignore inputs.

mdedetrich commented 1 year ago

Thanks for the help, found the root culprit https://github.com/spotify/docker-client