Error running parse from a different repository

[lriuui0x0]

I got the following snippet in my WORKSPACE file to download bazel-deps as an external dependency named bazel_deps:

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
    name = "bazel_deps",
    branch = "master",
    remote = "https://github.com/johnynek/bazel-deps",
)

When I run bazel run @bazel_deps//:parse -- generate, it returns an error:

ERROR: Skipping '@bazel_deps//:parse': error loading package '@bazel_deps//': Unable to find package for @io_bazel_rules_scala//scala:scala_toolchain.bzl: The repository '@io_bazel_rules_scala' could not be resolved.

However, if I cloned bazel-deps manually instead of through bazel's git_repository rule, and run parsing with bazel run //:parse -- generate, it seems to be running without problem.

[ianoc-stripe]

Bazel deps isn't really designed to be included as an external dependency in bazel. Nothing particularly blocking it but there isn't all of the handy bzl files to set it up.

But what this error is saying is that bazel_deps requires having the rules_scala setup in your work space for it to operate. So if you add/setup the rules scala and its associated toolchains based on the instructions in https://github.com/bazelbuild/rules_scala

you can likely get further -- you will also need to have any 3rdparty dependencies used by bazel deps available in your workspace too then.

(FWIW we normally use it as an external repo to run the tool that writes out files into our repos we check in, so its checked out into a cached location and built.)

[ianoc-stripe]

Maybe what your getting at is there is no transitive including of workspaces/dependencies in bazel today, there was some specs i think at some point but i don't believe it ever saw the light of day in the code base

[lriuui0x0]

Maybe what your getting at is there is no transitive including of workspaces/dependencies in bazel today, there was some specs i think at some point but i don't believe it ever saw the light of day in the code base

@ianoc-stripe OK, that's really...surprising. Thanks for the info!!

[lriuui0x0]

@ianoc-stripe So if that's the case, how do people manage transitive workspace dependencies? Surely the dependency chain is going to bloat quickly? Do people basically generate binaries, so people down the chain only need to depend on binaries?

I'm quite new to bazel. I hope you don't mind me asking beginner questions!

[ianoc-stripe]

Generally speaking most folks are working with monorepos or a few large repos can keep the chain depth shallow is really the answer there i think.

Once you start getting into deep chains you have to worry a lot about conflicting dependency requests in the various chains and how that might interact. So unclear how easily that could really be solved.

For the most part in a monorepo the work relative to the size of the repo in my experience isn't too bad keeping it all in shape, some repos will provide a script/thing you can include that will try setup their dependencies. But if 2 repos then collide on what they expect you'd have to figure that out for yourself. (See https://github.com/grpc/grpc-java/blob/master/repositories.bzl )

For bazel-deps though we just use a simple bash script that users invoke to use it which clones/updates/builds the repo as necessary and use it like a binary. Shipping a binary internally for your downstreams would also be fine there too. (It doesn't come into the compile/build time chains of repos that use it, so it side steps these challenges)

[lriuui0x0]

@ianoc-stripe OK, thanks for the info. I want to download and use bazel-deps as part of full build. From the readme description, it sounds like you have to invoke bazel-deps parsing manually. Is there a way to automate this process? I'm thinking about a custom rule or something that runs bazel-deps //:parse automatically?

[ianoc-stripe]

@lriuui0x0 our goal with bazel deps is to run it offline and commit in the result, its not really designed to be used as part of the build. You ~could do this, but it would require changes i think to bazel-deps to support nicely.

Bazel deps produces something akin to a lock file from say npm in java script /bundle in ruby, it emits a file that contains the sha + resolved versions and information as of that time. That information is then checked in -- and it can be reviewed at that time. Reviewing can be important here to try notice if adding a dependency has transitively changed the version of another dependency and you think it could be problematic.

[lriuui0x0]

@ianoc-stripe Thank you for the information! Is it worth considering release a binary version in the future that can be used directly by a bazel command? We can still run the command manually.

[ianoc-stripe]

@lriuui0x0 its not a bad idea to release linux/mac binaries i think for this I don't think. Probably could look to see what would be involved/needed for travis to produce these

[johnynek]

Would be great if we could use native-image to make Linux and Mac binaries and have Travis CI build them and upload them on every push to master.

[ianoc-stripe]

Yeah that would simplify the usage nicely too. Not entirely sure one would want to use bazel run irregardless since if you wind up messing up your workspace.bzl in the operation/fiddling with it then bazel is unlikely to just startup/want to run the app itself i think? Would make it a much faster/nicer experience to startup and run tho and let that be an option if people wanted

[ianoc]

@lriuui0x0 we've added to the release tooling included binaries for linux + Mac that you should be able to use in a repository rule or via some bash scripts. It will also include a generated bash script to provide a getting started point once https://github.com/johnynek/bazel-deps/pull/283 lands

[johnynek]

Please reopen if you feel the issue is still a problem.