search

pubref / rules_maven

Transitive maven dependencies with Bazel.
Other
33 stars 7 forks source link

Scala Support #12

Closed rynbrd closed 6 years ago

rynbrd commented 6 years ago

Is there currently a way to make this work with Scala deps? If so, can you provide an example? If not, do you anticipate adding it?

pcj commented 6 years ago

Not personally using scala, but it was my impression that scala rules can depend on vanilla maven_jar, java_import etc. In that case there should be no issue. Interested to learn more.

rynbrd commented 6 years ago

Yes, that seems to be working.

Is there a way to globally exclude dependencies? That would allow me to avoid doing an exclude for scala-library for every Scala dependency.

pcj commented 6 years ago

How do you mean... Do you want to exclude them at the level of maven_repository (see the omit attribute) or within a BUILD file, to filter out various jars when a @maven//:compile rule is depended on?

rynbrd commented 6 years ago

At the level of maven_repository would be sufficient. It would be more convenient if a list of artifacts to exclude could be provided to maven_repositories, however.

I want to exclude the Scala standard library jars as Maven dependencies. The jars included in local toolchain should be used. I'm having quite a bit of trouble accomplishing this currently.

I have this maven_repository defined:

scala_version = "2.11"
kstreams_version = "0.10.2.1"

maven_repository(
    name = "kafka",
    deps = [
        "org.apache.kafka:kafka_%s:%s" % (scala_version, kstreams_version),
        "org.apache.kafka:kafka-streams:%s" % kstreams_version,
    ],
    force = [
        "log4j:log4j:1.2.17",
        "org.apache.zookeeper:zookeeper:3.4.9",
        "org.slf4j:slf4j-api:1.7.21",
        "org.slf4j:slf4j-log4j12:1.7.21",
    ],
)
load("@kafka//:rules.bzl", "kafka_compile")
kafka_compile()

Which results in this output:

ERROR: error loading package '': Encountered error while reading extension file 'rules.bzl': no such package '@kafka//': Traceback (most recent call last):
    File "/home/sadpengu/.cache/bazel/_bazel_sadpengu/10b7eb9316ccbb3e9bb38047841831d0/external/org_pubref_rules_maven/maven/internal/maven_repository.bzl", line 507
        _execute(rtx, [java, "-jar", launcher_jar, ..."])
    File "/home/sadpengu/.cache/bazel/_bazel_sadpengu/10b7eb9316ccbb3e9bb38047841831d0/external/org_pubref_rules_maven/maven/internal/maven_repository.bzl", line 385, in _execute
        fail((" ".join(cmds) + ("failed: %s" ...)))
/usr/bin/java -jar /home/sadpengu/.cache/bazel/_bazel_sadpengu/10b7eb9316ccbb3e9bb38047841831d0/external/gradle_distribution/lib/gradle-launcher-4.1.jar dependenciesfailed: 
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':dependencies'.
> Could not resolve all dependencies for configuration ':compile'.
   > A conflict was found between the following modules:
      - org.scala-lang:scala-library:2.11.8
      - org.scala-lang:scala-library:2.11.6

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

* Get more help at https://help.gradle.org

BUILD FAILED in 0s
.
INFO: Elapsed time: 0.712s

I've tried adding omit options to maven_repository but it results in the above output. I've tried the following:

    omit = [
        "org.scala-lang:scala-library:2.11.8",
        "org.scala-lang:scala-library:2.11.6",
    ],
    omit = [
        "org.scala-lang:scala-library",
    ],

The exclude option seems to work, however:

    exclude = {
        "org.apache.kafka:kafka_%s" % scala_version: [
            "org.scala-lang:scala-library",
        ],
    },

This results in the expected output:

WARNING: /home/sadpengu/.cache/bazel/_bazel_sadpengu/10b7eb9316ccbb3e9bb38047841831d0/external/org_pubref_rules_maven/maven/internal/maven_repository.bzl:528:9: 
# CLOSED-FORM RULE:
# You can copy this to your WORKSPACE To suppress this message. 
maven_repository(
    name = 'kafka',
    deps = [
        'org.apache.kafka:kafka_2.11:0.10.2.1',
        'org.apache.kafka:kafka-streams:0.10.2.1',
    ],
    exclude = {
        'org.apache.kafka:kafka_2.11': [
            'org.scala-lang:scala-library',
        ],
    },
    force = [
        'log4j:log4j:1.2.17',
        'org.apache.zookeeper:zookeeper:3.4.9',
        'org.slf4j:slf4j-api:1.7.21',
        'org.slf4j:slf4j-log4j12:1.7.21',
    ],
    transitive_deps = [
        'c54d4b5a5e89af75a80b6d5857400165ce5188d0:com.101tec:zkclient:0.10',
        '45b426f7796b741035581a176744d91090e2e6fb:com.fasterxml.jackson.core:jackson-annotations:2.8.0',
        '60d059f5d2930ccd1ef03535b713fd9f933d1ba7:com.fasterxml.jackson.core:jackson-core:2.8.5',
        'b3035f37e674c04dafe36a660c3815cc59f764e2:com.fasterxml.jackson.core:jackson-databind:2.8.5',
        'f82c035cfa786d3cbec362c38c22a5f5b1bc8724:com.yammer.metrics:metrics-core:2.2.0',
        '5af35056b4d257e4b64b9e8069c0746e8b08629f:log4j:log4j:1.2.17',
        'c708bb2590c0652a642236ef45d9f99ff842a2ce:net.jpountz.lz4:lz4:1.3.0',
        'cdd846cfc4e0f7eefafc02c0f5dce32b9303aa2a:net.sf.jopt-simple:jopt-simple:5.0.3',
        'b22c04d7b8ff77c2697ed69bf349c66c560d0770:org.apache.kafka:connect-api:0.10.2.1',
        '2aa75f3f6e5309262d7a96416bc6b8c95c4480c6:org.apache.kafka:connect-json:0.10.2.1',
        'a5a1d215fb8b7ae051b42f4c383ef6ad45af4833:org.apache.kafka:kafka_2.11:0.10.2.1',
        '3dd2aa4c9f87ac54175d017bcb63b4bb5dca63dd:org.apache.kafka:kafka-clients:0.10.2.1',
        'd1267f08afae8cb3a947752151023e0a7044b4ac:org.apache.kafka:kafka-streams:0.10.2.1',
        'cbd9135689d6622a411e1e9b0ce880b502be5277:org.apache.zookeeper:zookeeper:3.4.9',
        '7c553b9afd312fd1d444350301a12d3fb9c223aa:org.rocksdb:rocksdbjni:5.0.1',
        '7369d653bcfa95d321994660477a4d7e81d7f490:org.scala-lang.modules:scala-parser-combinators_2.11:1.0.4',
        '139535a69a4239db087de9bab0bee568bf8e0b70:org.slf4j:slf4j-api:1.7.21',
        '7238b064d1aba20da2ac03217d700d91e02460fa:org.slf4j:slf4j-log4j12:1.7.21',
        '48d92871ca286a47f230feb375f0bbffa83b85f6:org.xerial.snappy:snappy-java:1.1.2.6',
    ],
)
.
INFO: Found 1 target...
Target //:hello up-to-date:
  bazel-bin/hello.jar
  bazel-bin/hello
INFO: Elapsed time: 2.961s, Critical Path: 0.01s

The problem with exclude is that I must specify for which dependency the exclusion is for. This is very difficult for our use case as I must exclude scala-library for every Scala dependency (of which there are many).

A further problem is that exclude does not seem to work for transitive dependencies. For instance, I have this maven_repository:

scala_version = "2.11"

maven_repository(
    name = "genson",
    deps = [
        "com.owlike:genson-scala_%s:1.4" % scala_version,
    ],
)
load("@genson//:rules.bzl", "genson_compile")
genson_compile()

This pulls in scala-library and scala-reflect from Maven which are part of the standard library:

WARNING: /home/sadpengu/.cache/bazel/_bazel_sadpengu/10b7eb9316ccbb3e9bb38047841831d0/external/org_pubref_rules_maven/maven/internal/maven_repository.bzl:528:9: 
# CLOSED-FORM RULE:
# You can copy this to your WORKSPACE To suppress this message. 
maven_repository(
    name = 'genson',
    deps = [
        'com.owlike:genson-scala_2.11:1.4',
    ],
    transitive_deps = [
        '8d6f7ee301653b6a221483663cf1c72a52967c8a:com.owlike:genson:1.4',
        '6309a5ca43d54eb9cbfe1776909f40c1d4c23503:com.owlike:genson-scala_2.11:1.4',
        'aa8f7553253318c468f90ce58a85f94bd1a224eb:org.scala-lang:scala-library:2.11.0',
        '90be1e4ead42b3e9ac53044bfe1b032cd9d78229:org.scala-lang:scala-reflect:2.11.0',
    ],
)
.
INFO: Found 1 target...
Target //:hello up-to-date:
  bazel-bin/hello.jar
  bazel-bin/hello
INFO: Elapsed time: 2.218s, Critical Path: 0.01s

The following exclude has no effect and results in the same output:

    exclude = {
        "com.owlike:genson-scala_%s:1.4" % scala_version: [
            "org.scala-lang:scala-library",
            "org.scala-lang:scala-reflect",
        ],
    },

At this point I'm not sure how to solve this problem. It doesn't seem like rules_maven is capable of correctly pulling in Scala dependencies.

pcj commented 6 years ago

Thanks for the additional detail. I'll take a look at this tonight. Seems like the omit option should work, and if it not that's a bug. You might try constructing a build.gradle file that does work for your needs and comparing that to what rules_maven is emitting.

pcj commented 6 years ago

@BlueDragonX can you try #13:

git_repository(
  name = "org_pubref_rules_maven",
  remote = "https://github.com/pubref/rules_maven",
  commit = "525757e5807de1a5f167a9dd0a5cf4270a1ca63d"
)
rynbrd commented 6 years ago

No luck. With this:

maven_repository(
    name = "kafka",
    deps = [
        "org.apache.kafka:kafka_%s:%s" % (scala_version, kstreams_version),
        "org.apache.kafka:kafka-streams:%s" % kstreams_version,
    ],
    omit = [
        "org.scala-lang:scala-library",
    ],
    force = [
        "log4j:log4j:1.2.17",
        "org.apache.zookeeper:zookeeper:3.4.9",
        "org.slf4j:slf4j-api:1.7.21",
        "org.slf4j:slf4j-log4j12:1.7.21",
    ],
)

I still get this:

ERROR: error loading package '': Encountered error while reading extension file 'rules.bzl': no such package '@kafka//': Traceback (most recent call last):
    File "/home/sadpengu/.cache/bazel/_bazel_sadpengu/8b6c27415a88bf788d0e4d934f8f6a07/external/org_pubref_rules_maven/maven/internal/maven_repository.bzl", line 536
        _execute(rtx, [java, "-jar", launcher_jar, ..."])
    File "/home/sadpengu/.cache/bazel/_bazel_sadpengu/8b6c27415a88bf788d0e4d934f8f6a07/external/org_pubref_rules_maven/maven/internal/maven_repository.bzl", line 414, in _execute
        fail((" ".join(cmds) + ("failed: %s" ...)))
/usr/bin/java -jar /home/sadpengu/.cache/bazel/_bazel_sadpengu/8b6c27415a88bf788d0e4d934f8f6a07/external/gradle_distribution/lib/gradle-launcher-4.1.jar dependenciesfailed: 
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':dependencies'.
> Could not resolve all dependencies for configuration ':compile'.
   > A conflict was found between the following modules:
      - org.scala-lang:scala-library:2.11.8
      - org.scala-lang:scala-library:2.11.6

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

* Get more help at https://help.gradle.org

BUILD FAILED in 0s
.
INFO: Elapsed time: 0.728s
pcj commented 6 years ago

The way the omit is currently implemented, it elides entries in the generated BUILD and rules.bzl files, not at the level of gradle (perhaps it should). What if you try force [ "org.scala-lang:scala-library:2.11.8" ] in addition to your existing forced artifacts?

rynbrd commented 6 years ago

That works to solve the conflict but the conflict isn't the root problem. I'm trying not to require scala-library as a maven dependenc since it is already provided.

pcj commented 6 years ago

In this case, here's what happening:

  1. A build.gradle file is generated that lists your root artifacts.
  2. Gradle computes the set of transitive dependencies and prints the dependency tree.
  3. rules_maven parses the dependency tree and builds it's own dependency graph from that. At this point the scala-library artifacts are still in that graph.
  4. When rules_maven emits the BUILD file that lists the java_library.export dependencies (for a particular configuration), it skips over those entries named in the omit attribute. Even though they are printed in transitive_deps attribute, those scala-library deps are never part of a java_library rule (like @genson//:compile. So, your scala rules never see them, and they should not exist in the classpath other than what is already provided by scala.

You can review the generated files via cd $(bazel info output_base)/external/genson && cat BUILD && cat rules.bzl.

Did you get a compile or runtime error? Or am I misunderstanding the problem?