search

VirtusLab / bazel-steward

A bot to keep Bazel dependencies up to date
https://virtuslab.github.io/bazel-steward/
Apache License 2.0
61 stars 5 forks source link

Suggestions for scala and java deps are mangled #364

Closed gannicottb closed 6 months ago

gannicottb commented 6 months ago

Given this maven_install:

 maven_install(
        name = "maven",
        artifacts = [
            "ai.x:diff_2.12:2.0.1",
            "ch.qos.logback:logback-classic:1.3.6",
            "co.fs2:fs2-core_2.12:3.9.4",
            "co.fs2:fs2-io_2.12:3.9.4",
            "com.alexdupre:bitcoincash-lib_2.12:0.9.20",
            "com.amazonaws:aws-java-sdk:1.11.771",
            "com.beachape:enumeratum_2.12:1.7.2",
            "com.beachape:enumeratum-circe_2.12:1.7.2",
            "com.beachape:enumeratum-doobie_2.12:1.7.2",
            "com.chuusai:shapeless_2.12:2.3.10",
            "com.eed3si9n:gigahorse-okhttp_2.12:0.5.0",
            "com.fasterxml.jackson.core:jackson-databind:2.14.2",
            "com.fasterxml.jackson.module:jackson-module-scala_2.12:2.14.2",
            "com.github.alexarchambault:scalacheck-shapeless_1.16_2.12:1.3.1",
            "com.github.ben-manes.caffeine:caffeine:3.1.1",
            "com.github.blemale:scaffeine_2.12:5.2.1",
            "com.github.cb372:cats-retry_2.12:3.1.0",
            "com.github.fd4s:fs2-kafka_2.12:3.1.0",
            "com.github.fs2-blobstore:core_2.12:0.9.12",
            "com.github.fs2-blobstore:gcs_2.12:0.9.12",
            "com.github.fs2-blobstore:s3_2.12:0.9.12",
            "com.github.ghostdogpr:caliban-client_2.12:2.0.2",
            "com.github.ghostdogpr:caliban-tools_2.12:2.0.2",
            "com.github.jasync-sql:jasync-common:2.1.23",
            "com.github.jasync-sql:jasync-postgresql:2.1.23",
            "com.github.multiformats:java-multibase:v1.1.1",
            "com.github.pureconfig:pureconfig_2.12:0.17.5",
            "com.github.scopt:scopt_2.12:3.7.0",
            "com.github.scredis:scredis_2.12:2.2.5",
            "com.google.api.grpc:proto-google-iam-v1:1.7.0",
            "com.google.cloud:google-cloud-bigquery:2.18.2",
            "com.google.cloud:google-cloud-pubsub:1.123.0",
            "com.google.cloud:google-cloud-storage:2.14.0",
            "com.google.guava:guava:33.0.0-jre",
            "com.jsuereth:scala-arm_2.12:2.0",
            "com.monovore:decline-effect_2.12:2.4.1",
            "com.olegpy:better-monadic-for_2.12:0.3.1",
            "com.sksamuel.avro4s:avro4s-core_2.12:4.1.1",
            "com.softwaremill.retry:retry_2.12:0.3.6",
            "com.softwaremill.sttp.client3:async-http-client-backend-cats_2.12:3.9.2",
            "com.typesafe.akka:akka-actor_2.12:2.8.5",
            "com.typesafe.akka:akka-slf4j_2.12:2.8.5",
            "com.typesafe.akka:akka-stream_2.12:2.8.5",
            "com.typesafe.akka:akka-http-spray-json_2.12:10.1.11",
            "com.typesafe.scala-logging:scala-logging_2.12:3.9.5",
            "com.yugabyte:java-driver-core:4.15.0-yb-1",
            "com.yugabyte:java-driver-query-builder:4.15.0-yb-1",
            "commons-codec:commons-codec:1.15",
            "commons-io:commons-io:2.6",
            "eu.timepit:fs2-cron-cron4s_2.12:0.9.0",
            "fr.acinq:bitcoin-lib_2.12:0.9.17",
            "io.7mind.izumi:logstage-core_2.12:1.2.5",
            "io.7mind.izumi:logstage-rendering-circe_2.12:1.2.5",
            "io.circe:circe-core_2.12:0.14.6",
            "io.circe:circe-generic_2.12:0.14.6",
            "io.circe:circe-literal_2.12:0.14.6",
            "io.circe:circe-parser_2.12:0.14.6",
            "io.circe:circe-generic-extras_2.12:0.14.3",
            "io.circe:circe-optics_2.12:0.14.1",
            "io.confluent:kafka-streams-avro-serde:7.5.3",
            "io.getunleash:unleash-client-java:6.0.1",
            "io.github.embeddedkafka:embedded-kafka-schema-registry_2.12:7.5.3",
            "io.grpc:grpc-alts:1.51.1",
            "io.grpc:grpc-api:1.51.1",
            "io.grpc:grpc-auth:1.51.1",
            "io.grpc:grpc-context:1.51.1",
            "io.grpc:grpc-googleapis:1.51.1",
            "io.grpc:grpc-grpclb:1.51.1",
            "io.grpc:grpc-netty:1.51.1",
            "io.grpc:grpc-netty-shaded:1.51.1",
            "io.grpc:grpc-protobuf:1.51.1",
            "io.grpc:grpc-protobuf-lite:1.51.1",
            "io.grpc:grpc-services:1.51.1",
            "io.grpc:grpc-stub:1.51.1",
            "io.grpc:grpc-xds:1.51.1",
            "io.higherkindness:droste-core_2.12:0.9.0",
            "io.netty:netty-buffer:4.1.72.Final",
            "io.netty:netty-codec:4.1.72.Final",
            "io.netty:netty-codec-http:4.1.72.Final",
            "io.netty:netty-codec-socks:4.1.72.Final",
            "io.netty:netty-common:4.1.72.Final",
            "io.netty:netty-handler:4.1.72.Final",
            "io.netty:netty-handler-proxy:4.1.72.Final",
            "io.netty:netty-transport:4.1.72.Final",
            "io.netty:netty-transport-classes-epoll:4.1.72.Final",
            "io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.72.Final",
            "io.netty:netty-transport-native-unix-common:4.1.72.Final",
            "io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.72.Final",
            "io.prometheus:simpleclient:0.16.0",
            "io.prometheus:simpleclient_hotspot:0.16.0",
            "io.prometheus:simpleclient_httpserver:0.16.0",
            "io.prometheus:simpleclient_pushgateway:0.16.0",
            "net.liftweb:lift-json_2.12:3.5.0",
            "org.apache.avro:avro:1.11.3",
            "org.apache.kafka:kafka-clients:7.5.3-ccs",
            "org.bouncycastle:bcprov-jdk18on:1.77",
            "org.http4s:http4s-circe_2.12:0.23.12",
            "org.http4s:http4s-dsl_2.12:0.23.12",
            "org.http4s:http4s-ember-client_2.12:0.23.12",
            "org.http4s:http4s-ember-server_2.12:0.23.12",
            "org.http4s:http4s-prometheus-metrics_2.12:0.23.12",
            "org.roaringbitmap:RoaringBitmap:1.0.1",
            "org.rocksdb:rocksdbjni:8.9.1",
            "org.scala-lang.modules:scala-collection-compat_2.12:2.11.0",
            "org.scala-lang.modules:scala-java8-compat_2.12:1.0.2",
            "org.scalacheck:scalacheck_2.12:1.17.0",
            "org.scalamacros:paradise_2.12.18:2.1.1",
            "org.scalamock:scalamock_2.12:5.2.0",
            "org.scalatest:scalatest_2.12:3.2.17",
            "org.scalatestplus:mockito-4-2_2.12:3.2.11.0",
            "org.scalatestplus:scalacheck-1-16_2.12:3.2.14.0",
            "org.scalikejdbc:scalikejdbc-async_2.12:0.14.0",
            "org.scalikejdbc:scalikejdbc-joda-time_2.12:3.5.0",
            "org.scodec:scodec-core_2.12:1.11.10",
            "org.tpolecat:doobie-core_2.12:1.0.0-RC2",
            "org.tpolecat:doobie-hikari_2.12:1.0.0-RC2",
            "org.tpolecat:doobie-postgres_2.12:1.0.0-RC2",
            "org.tpolecat:doobie-scalatest_2.12:1.0.0-RC2",
            "org.typelevel:cats-effect_2.12:3.5.3",
            "org.typelevel:cats-effect-testkit_2.12:3.5.3",
            "org.typelevel:cats-effect-testing-scalatest_2.12:1.5.0",
            "org.typelevel:fs2-grpc-codegen_2.12:2.7.11",
            "org.typelevel:fs2-grpc-runtime_2.12:2.7.11",
            "org.typelevel:kittens_2.12:3.2.0",
            "org.typelevel:log4cats-slf4j_2.12:2.6.0",
            "org.web3j:abi:5.0.0",
            "org.yaml:snakeyaml:2.2",
            "software.amazon.awssdk:sts:2.23.14",
            maven.artifact(
                group = "io.confluent",
                artifact = "kafka-avro-serializer",
                version = "7.5.3",
                exclusions = [
                    "javax.ws.rs:javax.ws.ris-api",
                ],
            ),
            # needs to be separate to have the exclusion, as it causes a circular dependency without it
            # see https://github.com/grpc/grpc-java/issues/10576#issuecomment-1741257443
            maven.artifact(
                artifact = "grpc-core",
                exclusions = [
                    "io.grpc:grpc-util",
                ],
                group = "io.grpc",
                version = "1.51.1",
            ),
            maven.artifact(
                group = "org.apache.kafka",
                artifact = "kafka_2.12", 
                version = "7.5.3-ccs",
                exclusions = [
                    "com.sun.jdmk:jmxtools",
                    "com.sun.jmx:jmxri",
                    "javax.jms:jms",
                ],
            ),
            # Excludes the part that would need native libraries
            maven.artifact(
                group = "org.xerial.larray",
                artifact = "larray_2.12",
                version = "0.4.1",
                exclusions = [
                    "org.xerial.larray:larray-mmap",
                ],
            ),
        ],
        fail_if_repin_required = True,
        fail_on_missing_checksum = True,
        fetch_javadoc = True,
        fetch_sources = True,
        override_targets = {
            # same as bazel-deps "replacements"
            "io.netty:netty-transport-native-epoll": "@maven//:io_netty_netty_transport_native_epoll_linux_x86_64",
            "org.scala-lang:scala-compiler": "@io_bazel_rules_scala_scala_compiler",
            "org.scala-lang:scala-library": "@io_bazel_rules_scala_scala_library",
            "org.scala-lang:scala-reflect": "@io_bazel_rules_scala_scala_reflect",
            "org.scala-lang.modules:scala-parser-combinators": "@io_bazel_rules_scala_scala_parser_combinators//:io_bazel_rules_scala_scala_parser_combinators",
            "org.scala-lang.modules:scala-xml": "@io_bazel_rules_scala_scala_xml//:io_bazel_rules_scala_scala_xml",
        },
        repositories = [
            "https://oss.sonatype.org/content/repositories/public/",
            "https://repo1.maven.org/maven2",
            "https://packages.confluent.io/maven/",
            "https://jitpack.io",
        ],
        excluded_artifacts = [
            "org.slf4j:slf4j-log4j12",
        ],
        version_conflict_policy = "pinned",
    )

Running bazel-steward like this results in strange results. Every one of the branches it creates contain "fixes" like

             "io.github.embeddedkafka:embedded-kafka-schema-registry_2.12:7.5.3",
             "io.grpc:grpc-alts:1.51.1",
             "io.grpc:grpc-api:1.51.1",
-            "io.grpc:grpc-auth:1.51.1",
+           1.62.2rpc:grpc-auth:1.51.1",
             "io.grpc:grpc-context:1.51.1",
             "io.grpc:grpc-googleapis:1.51.1",
             "io.grpc:grpc-grpclb:1.51.1",

(and for a scala dep)

             "org.typelevel:cats-effect_2.12:3.5.3",
-            "org.typelevel:cats-effect-testkit_2.12:3.5.3",
+            3.5.4typelevel:cats-effect-testkit_2.12:3.5.3",
             "org.typelevel:cats-effect-testing-scalatest_2.12:1.5.0",

Have you ever seen anything like this? I don't even know where to start debugging this, the heuristics seem to be thrown off by something, I'm just not sure what. I can try to get a minimal reproducer going (because our real use case is a bit more complicated than what I'm sharing here), but figured I'd check to see if this is some kind of known gotcha.

gannicottb commented 6 months ago

Ok, I've got a reproducer - minimal repo here On main, I had Bazel-steward create branches via coursier launch org.virtuslab:bazel-steward:1.3.0 --main org.virtuslab.bazelsteward.app.Main -- --no-remote

commit ff72ed3ef8072203a02b7981ad791791b8258ca4 (bazel-steward/ch.qos.logback/logback-classic/1.5.3)
Author: Brandon Gannicott <brandon.gannicott@gmail.com>
Date:   Wed Mar 27 17:22:10 2024 -0500

    Updated ch.qos.logback:logback-classic to 1.5.3

diff --git a/WORKSPACE b/WORKSPACE
index b1439c2..254210b 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -97,7 +97,7 @@ load("@rules_jvm_external//:defs.bzl", "maven_install")
 maven_install(
     name = "data_onchain_3rdparty",
     artifacts = [
-        "ch.qos.logback:logback-classic:1.4.14",
+        "ch.qos.logback:1.5.3k-classic:1.4.14",
         "com.typesafe.scala-logging:scala-logging_2.12:3.9.5",
         "org.slf4j:slf4j-api:2.0.7",
     ],
gannicottb commented 6 months ago

Hm, I'm getting better results with version 1.4.0 in my reproducer repo. Hard to imagine what could have changed in one minor version to fix what I'm seeing, but maybe...

gannicottb commented 6 months ago

Trouble is, the original example is using the latest code from master, which leaves me with an apparent issue on master, but a reproducer for the behavior only for v1.3.0

gannicottb commented 6 months ago

After digging into this a bit more, I'm no longer seeing the garbled suggestions, so I think there was something subtly wrong with my setup. Closing unless it happens again.