scala_proto_library adds conflicting fastparse dependency to classpath

andrewhamon commented 3 years ago

I am trying to use Caliban, a graphql library, in my scala bazel project.

Caliban depends on fastparse version 2.3.0. If I depend on the generated scala code from scala_proto_library, that pulls in a dependency on fastparse 2.1.3 (com.lihaoyi:fastparse_2.12:2.1.2). These conflicting versions then cause runtime errors (specifically a NoSuchMethodError, which I assume is from trying to call a method that exists in 2.3 but not in 2.1.

If I put the scala proto dependencies after the caliban dependency, then fastparse 2.3.0 comes first on the classpath and I don't get runtime errors (at least not the specific runtime error I was encountering before, who knows what else is lurking).

This leads me to the following questions:

is there a way to use a different version of fastparse when generating protos?
is fastparse needed at all other than to codegen scala? If not, it seems like it would be ideal if it could be omitted from the classpath entirely.
if fastparse is required at runtime by the generated scala protobuf code, am I creating a ticking time bomb by reversing the dependency order to ensure 2.3.0 comes first on the classpath?

Another observation -- even if I put the scala toolchain into direct dependency mode, a dependency on a scala proto library still transitively pulls in fastparse. This is not the behavior I would expect.

andrewhamon commented 3 years ago

If I only depend on caliban like so:

scala_binary(
    name = "caliban",
    srcs = ["src/main/scala/com/flexport/workitems/graphql/GraphqlExperiment.scala"],
    deps = [
      "@maven//:com_github_ghostdogpr_caliban_2_12",
    ],
    main_class = "example.Hello"
)

Then this is the classpath specified to scalac:

bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/com/github/ghostdogpr/caliban_2.12/0.9.2/stamped_caliban_2.12-0.9.2.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/com/lihaoyi/geny_2.12/0.6.0/stamped_geny_2.12-0.6.0.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/com/lihaoyi/sourcecode_2.12/0.2.1/stamped_sourcecode_2.12-0.2.1.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/com/lihaoyi/fastparse_2.12/2.3.0/stamped_fastparse_2.12-2.3.0.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/dev/zio/zio-streams_2.12/1.0.1/stamped_zio-streams_2.12-1.0.1.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/dev/zio/zio-stacktracer_2.12/1.0.1/stamped_zio-stacktracer_2.12-1.0.1.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/org/scala-lang/scala-library/2.12.12/stamped_scala-library-2.12.12.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/dev/zio/izumi-reflect-thirdparty-boopickle-shaded_2.12/1.0.0-M5/stamped_izumi-reflect-thirdparty-boopickle-shaded_2.12-1.0.0-M5.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/dev/zio/izumi-reflect_2.12/1.0.0-M5/stamped_izumi-reflect_2.12-1.0.0-M5.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/dev/zio/zio_2.12/1.0.1/stamped_zio_2.12-1.0.1.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/org/scala-lang/scala-reflect/2.12.8/stamped_scala-reflect-2.12.8.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/dev/zio/zio-query_2.12/0.2.5/stamped_zio-query_2.12-0.2.5.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/org/scala-lang/scala-compiler/2.12.8/stamped_scala-compiler-2.12.8.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/org/scala-lang/modules/scala-xml_2.12/1.0.6/stamped_scala-xml_2.12-1.0.6.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/com/propensive/magnolia_2.12/0.17.0/stamped_magnolia_2.12-0.17.0.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/com/propensive/mercator_2.12/0.2.1/stamped_mercator_2.12-0.2.1.jar
external/io_bazel_rules_scala_scala_library/scala-library-2.12.12.jar
external/io_bazel_rules_scala_scala_reflect/scala-reflect-2.12.12.jar

If I add a dependency on a scala proto, like so:

scala_binary(
    name = "caliban",
    srcs = ["src/main/scala/com/flexport/workitems/graphql/GraphqlExperiment.scala"],
    deps = [
      "@maven//:com_github_ghostdogpr_caliban_2_12",
      "//protobuf/grpc/health/v1:health_scala_grpc",
    ],
    main_class = "example.Hello"
)

then this is the resulting classpath:

bazel-out/darwin-fastbuild/bin/protobuf/grpc/health/v1/health_proto_scalapb.jar
bazel-out/darwin-fastbuild/bin/protobuf/google/api/annotations_proto_scalapb.jar
bazel-out/darwin-fastbuild/bin/protobuf/google/api/http_proto_scalapb.jar
external/scala_proto_rules_scalapb_runtime/scalapb-runtime_2.12-0.9.7.jar
bazel-out/darwin-fastbuild/bin/external/com_google_protobuf/libprotobuf_java-hjar.jar
external/scala_proto_rules_scalapb_lenses/lenses_2.12-0.9.7.jar
external/scala_proto_rules_scalapb_fastparse/fastparse_2.12-2.1.3.jar
external/io_bazel_rules_scala_scala_library/scala-library-2.12.12.jar
external/io_bazel_rules_scala_scala_reflect/scala-reflect-2.12.12.jar
external/scala_proto_rules_grpc_api/grpc-api-1.24.0.jar
external/scala_proto_rules_perfmark_api/perfmark-api-0.17.0.jar
external/scala_proto_rules_scalapb_runtime_grpc/scalapb-runtime-grpc_2.12-0.9.7.jar
external/scala_proto_rules_grpc_core/grpc-core-1.24.0.jar
external/scala_proto_rules_grpc_stub/grpc-stub-1.24.0.jar
external/scala_proto_rules_grpc_protobuf/grpc-protobuf-1.24.0.jar
external/scala_proto_rules_grpc_netty/grpc-netty-1.24.0.jar
external/scala_proto_rules_grpc_context/grpc-context-1.24.0.jar
external/scala_proto_rules_guava/guava-26.0-android.jar
external/scala_proto_rules_opencensus_api/opencensus-api-0.22.1.jar
external/scala_proto_rules_opencensus_impl/opencensus-impl-0.22.1.jar
external/scala_proto_rules_disruptor/disruptor-3.4.2.jar
external/scala_proto_rules_opencensus_impl_core/opencensus-impl-core-0.22.1.jar
external/scala_proto_rules_opencensus_contrib_grpc_metrics/opencensus-contrib-grpc-metrics-0.22.1.jar
external/scala_proto_rules_google_instrumentation/instrumentation-api-0.3.0.jar
external/scala_proto_rules_netty_codec/netty-codec-4.1.32.Final.jar
external/scala_proto_rules_netty_codec_http/netty-codec-http-4.1.32.Final.jar
external/scala_proto_rules_netty_codec_http2/netty-codec-http2-4.1.32.Final.jar
external/scala_proto_rules_netty_codec_socks/netty-codec-socks-4.1.32.Final.jar
external/scala_proto_rules_netty_handler/netty-handler-4.1.32.Final.jar
external/scala_proto_rules_netty_buffer/netty-buffer-4.1.32.Final.jar
external/scala_proto_rules_netty_transport/netty-transport-4.1.32.Final.jar
external/scala_proto_rules_netty_resolver/netty-resolver-4.1.32.Final.jar
external/scala_proto_rules_netty_common/netty-common-4.1.32.Final.jar
external/scala_proto_rules_netty_handler_proxy/netty-handler-proxy-4.1.32.Final.jar
bazel-out/darwin-fastbuild/bin/external/com_google_protobuf/descriptor_proto_scalapb.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/com/github/ghostdogpr/caliban_2.12/0.9.2/stamped_caliban_2.12-0.9.2.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/com/lihaoyi/geny_2.12/0.6.0/stamped_geny_2.12-0.6.0.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/com/lihaoyi/sourcecode_2.12/0.2.1/stamped_sourcecode_2.12-0.2.1.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/com/lihaoyi/fastparse_2.12/2.3.0/stamped_fastparse_2.12-2.3.0.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/dev/zio/zio-streams_2.12/1.0.1/stamped_zio-streams_2.12-1.0.1.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/dev/zio/zio-stacktracer_2.12/1.0.1/stamped_zio-stacktracer_2.12-1.0.1.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/org/scala-lang/scala-library/2.12.12/stamped_scala-library-2.12.12.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/dev/zio/izumi-reflect-thirdparty-boopickle-shaded_2.12/1.0.0-M5/stamped_izumi-reflect-thirdparty-boopickle-shaded_2.12-1.0.0-M5.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/dev/zio/izumi-reflect_2.12/1.0.0-M5/stamped_izumi-reflect_2.12-1.0.0-M5.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/dev/zio/zio_2.12/1.0.1/stamped_zio_2.12-1.0.1.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/org/scala-lang/scala-reflect/2.12.8/stamped_scala-reflect-2.12.8.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/dev/zio/zio-query_2.12/0.2.5/stamped_zio-query_2.12-0.2.5.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/org/scala-lang/scala-compiler/2.12.8/stamped_scala-compiler-2.12.8.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/org/scala-lang/modules/scala-xml_2.12/1.0.6/stamped_scala-xml_2.12-1.0.6.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/com/propensive/magnolia_2.12/0.17.0/stamped_magnolia_2.12-0.17.0.jar
bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/com/propensive/mercator_2.12/0.2.1/stamped_mercator_2.12-0.2.1.jar

liucijus commented 3 years ago

You can try defining your own proto deps toolchain (https://github.com/bazelbuild/rules_scala/blob/master/scala_proto/BUILD#L45). scala_proto_deps_toolchain can take a list of dep providers, which you can use to pass your own deps (https://github.com/bazelbuild/rules_scala/blob/master/scala_proto/scala_proto_toolchain.bzl#L58)

olafurpg commented 3 years ago

You may be able to avoid this problem by upgrading to the latest ScalaPB, which doesn't bring in a transitive dependency on fastparse

❯ cs resolve 'com.thesamet.scalapb:scalapb-runtime_2.13:0.11.0-M3'
com.google.protobuf:protobuf-java:3.12.2:default
com.thesamet.scalapb:lenses_2.13:0.11.0-M3:default
com.thesamet.scalapb:scalapb-runtime_2.13:0.11.0-M3:default
org.scala-lang:scala-library:2.13.3:default
org.scala-lang.modules:scala-collection-compat_2.13:2.2.0:default

❯ cs resolve 'com.thesamet.scalapb:scalapb-runtime_2.13:0.9.7'    
com.google.protobuf:protobuf-java:3.8.0:default
com.lihaoyi:fastparse_2.13:2.1.3:default # <------ transitive Fastparse
com.lihaoyi:sourcecode_2.13:0.1.7:default
com.thesamet.scalapb:lenses_2.13:0.9.7:default
com.thesamet.scalapb:scalapb-runtime_2.13:0.9.7:default
org.scala-lang:scala-library:2.13.1:default

andrewhamon commented 3 years ago

@liucijus I will look into that, but I'm still a bit lost on where the versions actually get set?

liucijus commented 3 years ago

I'm still a bit lost on where the versions actually get set?

In rules_scala default artifact versions are defined in https://github.com/bazelbuild/rules_scala/blob/master/third_party/repositories/scala_2_12.bzl. Those versions are used if you set up proto rules with helper scala_proto_repositories from https://github.com/bazelbuild/rules_scala/blob/master/scala_proto/scala_proto.bzl#L35

rfan-debug commented 3 years ago

Hey, @liucijus . I ran into a similar NoSuchMethodError:

 java.lang.NoSuchMethodError: 'scalapb.GeneratedMessage scalapb.LiteParser$.readMessage(com.google.protobuf.CodedInputStream, scalapb.GeneratedMessageCompanion)'

It turned out to be that the scalaPB's runtime version does not match. In the default scala_proto_library, it depends on scalaPB **0.9.0, but we have an external dependency that generates scala code based on scalaPB 0.10.10.

I tried to add my own deps into scala_proto_deps_toolchain in a BUILD.bazel file like this way: https://gist.github.com/rfan-debug/89cf894663c923c6162400b49057bf44. (All the dependencies here are manually resolved for scalaPB 0.10.10)

and replaced the default scala_proto_repositories()

load("@io_bazel_rules_scala//scala_proto:scala_proto.bzl", "scala_proto_repositories")
scala_proto_repositories()

with my toolchain registeration:

register_toolchains("//tools/scalapb:scala_proto_deps_toolchain")

However, it threw out the a different runtime error indicating that the scalapb cannot be found:

no such package '@scala_proto_rules_scalapb_runtime//': The repository '@scala_proto_rules_scalapb_runtime', The repository '@scala_proto_rules_scalapb_runtime' could not be resolved and referenced by '//protobuf/core:core_scala_proto'

liucijus commented 3 years ago

@rfan-debug could you make a standalone git repository, which demonstrates the problem? I think you still load something that depends on DEFAULT_SCALAPB_COMPILE_DEPS

hshamji commented 3 years ago

hi @liucijus , I've run into a similar issue. I think I've successfully created a scala_proto toolchain upgrading the scalaPB deps, however, when I build I don't see a difference (ie. protos still compiled using the default scalaPB version).

I ran bazel build with the toolchain_resolution_debug flag and couldn't see the scala_proto toolchain being used/resolved. The closest toolchains were a) @io_bazel_rules_scala//scala:toolchain_type and b) @rules_proto_grpc//protobuf:toolchain_type.

A little bit stuck here. Any suggestions? TY!

liucijus commented 3 years ago

@hshamji please make a small project on github to reproduce the problem

hshamji commented 3 years ago

Hi @liucijus, here is a small (though not minimal) project with the same issue. The project compiles 2 tensorflow proto files, Example and Feature.

If you clone the repo and run bazel build --verbose_failures --toolchain_resolution_debug //protobuf/tensorflow/core/example:example_scala_proto I see the toolchains I mentioned above being resolved but not the scala_proto toolchain.

Thank you,

simuons commented 3 years ago

@hshamji could you try removing these lines from your project. I think this causes problem for you as it registers default_deps_toolchain and overrides yours

liucijus commented 3 years ago

@hshamji for some reason your repro works just fine on my computer:

bazel_proto_toolchain_error$ bazel build //protobuf/tensorflow/core/example:example_scala_proto
Starting local Bazel server and connecting to it...
(15:40:08) INFO: Writing tracer profile to '/home/vaidas/.cache/bazel/_bazel_vaidas/1d2453404588031599bfcccb3e9dd4fc/command.profile.gz'
(15:40:08) INFO: Invocation ID: f7be814e-1c1b-4146-8f7d-d894af0999fe
(15:40:08) INFO: Current date is 2021-07-09
(15:40:12) INFO: SHA256 (https://repo.maven.apache.org/maven2/com/thesamet/scalapb/lenses_2.12/0.9.7/lenses_2.12-0.9.7-sources.jar) = 42c0038d07bc8c7e239744f8858d82cdbd38906ad1c317b3354db753a093ae99
(15:40:14) INFO: SHA256 (https://repo.maven.apache.org/maven2/com/thesamet/scalapb/scalapb-runtime_2.12/0.9.7/scalapb-runtime_2.12-0.9.7-sources.jar) = 920dcd19966aaa2fe49b9b1d8104d6d88b7461619e2cd217f84620eaee75a7b5
(15:40:14) INFO: SHA256 (https://repo.maven.apache.org/maven2/com/lihaoyi/fastparse_2.12/2.1.3/fastparse_2.12-2.1.3-sources.jar) = d7d33157d9dc83d3dd2cec6b4835bede4ce7b7d5e0bef6a08ff6c9e0efe78dd9
(15:40:17) INFO: Analyzed target //protobuf/tensorflow/core/example:example_scala_proto (58 packages loaded, 1308 targets configured).
(15:40:17) INFO: Found 1 target...
(15:45:46) INFO: From Building external/io_bazel_rules_scala/src/java/io/bazel/rulesscala/scalac/scalac.jar (5 source files):
warning: [path] bad path element "/home/vaidas/.cache/bazel/_bazel_vaidas/1d2453404588031599bfcccb3e9dd4fc/bazel-workers/multiplex-worker-4-Javac/__main__/external/io_bazel_rules_scala_scala_compiler/scala-reflect.jar": no such file or directory
warning: [path] bad path element "/home/vaidas/.cache/bazel/_bazel_vaidas/1d2453404588031599bfcccb3e9dd4fc/bazel-workers/multiplex-worker-4-Javac/__main__/external/io_bazel_rules_scala_scala_compiler/scala-library.jar": no such file or directory
Target //protobuf/tensorflow/core/example:example_scala_proto up-to-date:
  bazel-bin/protobuf/tensorflow/core/example/example_scala_proto.jar
  bazel-bin/protobuf/tensorflow/core/example/example_scala_proto_java.jar
(15:46:02) INFO: Elapsed time: 355.590s, Critical Path: 103.04s
(15:46:02) INFO: 451 processes: 40 internal, 397 linux-sandbox, 14 worker.
(15:46:02) INFO: Build completed successfully, 451 total actions

I see you use some @rules_proto_grpc indirections (I was not aware of this project until now), I think they may have some changes which put hard dependency on internal dep lists in rules_scala.

hshamji commented 3 years ago

Thank you for looking at this!

@simuons , oh, yes, good catch. So, when I comment out those lines I get the error that it can't find targets that the custom toolchain is trying to provide.

(09:37:08) ERROR: /Users/hshamji/Desktop/repos/to_delete/caro/protobuf/tensorflow/core/example/BUILD.bazel:13:20: no such target '//external:io_bazel_rules_scala/dependency/proto/scalapb_runtime': target 'io_bazel_rules_scala/dependency/proto/scalapb_runtime' not declared in package 'external' defined by /Users/hshamji/Desktop/repos/to_delete/caro/WORKSPACE and referenced by '//protobuf/tensorflow/core/example:example_scala_proto'

@liucijus , the build command completes on my machine, but it uses the default version of the scalaPB, not the one in my toolchain. In the output you posted you can see it is using scalapb-runtime_2.12/0.9.7, for example, whereas the toolchain refers to the jar in @3rdparty_jvm//com/thesamet/scalapb:scalapb_runtime which is version 0.10.10.

Let me try not using @rules_proto_grpc and see if that makes a difference. Thank you

hshamji commented 3 years ago

@simuons @liucijus , I experimented a bit further and have observed some interesting behavior.

I'm not using master, but the version mentioned in the README (rules_scala_version = "5df8033f752be64fbe2cedfd1bdbad56e2033b15").

I replaced the scala_proto_repositories() function with its underlying function calls (here) to trace the steps. The interesting part is that when I comment out the registering of the toolchain (which is the final step) everything still works and builds (bazel build --verbose_failures --toolchain_resolution_debug //protobuf/tensorflow/core/example:example_scala_proto). This suggests the toolchain is not being used, and instead the bind calls are operative.

Further, I tried replacing the bind calls to use the scalaPB 0.10.10 version packages instead (branch here). When I build again (same command) this looks like it is picking up the dependencies I want, but it causes another error that suggests that I have not replaced all of the necessary dependencies and there is some mismatch.

Am I interpreting this incorrectly (that the scala_proto toolchain is not being used in this version of rules_scala)? Thank you

liucijus commented 3 years ago

@hshamji maybe this example will help you: https://github.com/liucijus/rules-scala-proto-toolchain It uses custom deps via proto deps toolchain. In general you should not use binds with deps toolchain - deps toolchains were created to remove the need to rebind dependecies.

hshamji commented 3 years ago

Hi @liucijus,

Thanks for this example. I tried modifying the versions of the scalaPB package in the WORKSPACE file in your sample repo but it still shows it is using the old/default version I changed lines 112-117 in WORKSPACE to:

        "com.thesamet.scalapb:protoc-bridge_2.12:0.7.14",
        "com.thesamet.scalapb:scalapbc_2.12:0.10.10",
        "com.thesamet.scalapb:scalapb-runtime_2.12:0.10.10",
        "com.thesamet.scalapb:scalapb-runtime-grpc_2.12:0.10.10",
        "com.thesamet.scalapb:lenses_2.12:0.10.10",

but the build command still shows version 0.9.7

INFO: Found 1 target...
Target //scala/proto_example:proto_scala_lib up-to-date:
  bazel-bin/scala/proto_example/proto_lib_scalapb.jar
  bazel-bin/external/maven/v1/https/jcenter.bintray.com/com/lihaoyi/fastparse_2.12/2.1.3/fastparse_2.12-2.1.3.jar
  bazel-bin/external/maven/v1/https/jcenter.bintray.com/com/thesamet/scalapb/lenses_2.12/0.9.7/lenses_2.12-0.9.7.jar
  bazel-bin/external/maven/v1/https/jcenter.bintray.com/com/thesamet/scalapb/scalapb-runtime_2.12/0.9.7/scalapb-runtime_2.12-0.9.7.jar
...

And to clarify on my earlier message. All those bind statements were from the rules_scala source for that version. I just copied the contents of the scala_proto_repositories() function into the WORKSPACE so I evaluate each step.

andrewhamon commented 2 years ago

@hshamji did you make sure to re-pin the maven dependencies by running bazel run @unpinned_maven//:pin?

Anyways, @liucijus I finally got around to setting up custom toolchains for everything (and prior to that I found some workaround to this specific issue, probably upgrading my rules_scala version or something). Now I don't have nightmares about runtime dependency mismatches :)

bazelbuild / rules_scala

scala_proto_library adds conflicting fastparse dependency to classpath #1126