Support bzlmod / bazel central registry

[malt3]

I'm in the process of planning a migration of my teams monorepo from bazel WORKSPACE to bzlmod. Bazeldnf is one dependency that does not yet support bzlmod.

If you are interested in supporting bzlmod (and publishing releases to bcr), I'd like to take a stab at this in the upcoming months.

Here are the guidelines to publish a ruleset to BCR. Publishing new releases can be automated with a github app.

Design spec for module extension

We need to create a module extension for rpm rules (rpm rules are currently repo rules that create new repositories. This works differently under bzlmod).

The dominant pattern for bzlmod is to read a domain specific lockfile and generate repositories from that. A good example is how rules_go does it:

https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md

Basically, they parse a go.mod and go.sum file and generate repositories from them.

For bazeldnf, I think this could be great as well. Instead of having rpm rules that are bazeldnf specific, we could try to standardize on a lockfile format for rpms. One example of such a format is implemented here: https://github.com/microsoft/rpmoci

This means bazeldnf could share the same lockfile as rpmoci and there is no potential for rpm rules to be out of sync with the lockfile. The actual usage could look similar to this:

rpm_deps = use_extension("@bazeldnf//:extensions.bzl", "rpm_deps")
rpm_deps.from_file(rpmlock = "//:rpmoci.lock")

# All *direct* rpm dependencies of the module have to be listed explicitly.
use_repo(
    rpm_deps,
    "curl-minimal",
    "systemd",
)

This would then result in rpm rules during module resolution. Note that we do not specify all packages here, but only direct dependencies. The lockfile pins the transitive closure of all required rpms and bazeldnf would generate the real set of rpm rules (including transitive deps) during module resolution.

Another advantage of this is improved support for vendoring / mirroring rpms. Since the urls of the rpm rules are not commited in the source code, they could be easily influenced to point to:

• a base url of my own yum mirror
• a content addressable store (base url + hash of rpm)
• a fallback list of public mirrors
• a local vendor directory

This would make the actual dependency management of rpms a lot simpler.

[rmohr]

I would really love to see bzlmod support!

[malt3]

One idea I had is implementing this first: https://github.com/bazelbuild/rules_go/issues/3646. It would allow bazeldnf to be build from sources (instead of shipping prebuilt binaries) but still have an isolated set of dependencies. This would mean you get the binary "for free", by just adding a module rule. The remainder of the work would just be making sure the starlark part of bazeldnf works correctly with bzlmod.

[rmohr]

One idea I had is implementing this first: bazelbuild/rules_go#3646. It would allow bazeldnf to be build from sources (instead of shipping prebuilt binaries) but still have an isolated set of dependencies.

Yes, that would work now. I think I would still prefer keeping the prebuilt binaries, but now we could make the source-code compile make a proper fallback in case that no binaries are present.

[rmohr]

I think that for instnace a lot of the grpc stuff works like this since a long time: prebuilt binaries to speed up, but if you happen to be on an env where no prebuilds are there, build from source (which can take quite a bit of time).

[malt3]

I created #61 which is a step towards supporting bzlmod.

[manuelnaranjo]

Jumping a bit late to the party, but better late than never. If we make the rpm_deps.from_file to get an extra argument called name, then we could use that name to create a jump repo (like rules_jvm_external does with maven) and then we would only depend on that one.

I like that rpmoci compatible approach. But an earlier approach should just be a wrapper for the current rpm we use from WORKSPACE

How can I help? I see that @malt3 MR was for bazel 6, now we have 7 available.

Shall we create a bazeldnf channel in slack and try to coordinate through it?

[manuelnaranjo]

FYI I created https://github.com/rmohr/bazeldnf/pull/67 which bumps to bazel 7 and adds support for bzlmod, I haven't tested it yet with all the supported bazel versions, but I think this is a smaller changeset than the one proposed in #61 , which makes it easier to review

[manuelnaranjo]

@malt3 @rmohr I had some fun during the holidays, and now I have a branch with support for bzlmod with lockfiles and a I think a nice API.

this is what I came up as a lock file

Once you have a lock file you add it to your MODULE.bazel as here (BTW this is all managed by the bazeldnf rpmtree plugin)

And then you can either use the rpmtree available as @<repo-name>//:rpms or you can import RPMS from @<repo-name>//:rpms.bzl and do whatever you need with it.

I can clean-up the branch to make the history cleaner, but before I do that I wanted to show you what was accomplished.

I tried to keep track of the rpm dependencies, but turns out that there are cyclic dependencies, and those are painful to deal with

[rmohr]

@malt3 @rmohr I had some fun during the holidays, and now I have a branch with support for bzlmod with lockfiles and a I think a nice API.

one question on that: I am used to having .bzl files as well for managing dependencies. What exactly would be the benefit of the JSON file over those? I am mostly concerned about being discoverable/compatible with tools like buildozer/buildifier which can probably not make much out of these files.

[manuelnaranjo]

The bzl approach is still possible, you can do it either in MODULE.bazel or you could have a bzl file that exposes a module extension.

The problem with bzl is that use_repo expects one entry per Bazel repository available to the workspace, making it a duplicated effort to maintain.

The json contains all your dependencies and the bzlmod module creates a single repository for the entire rpmtree, making maintenance of the bazeldnf user a bit easier IMO.

If you look into bzlmod usage of golang vs the one of rules_jvm_external you will notice similar setups, where golang contains a gigantic list similar to the one from go.mod non transitive dependencies, while rules_jvm_external moves all this complexity down the json file and the generated jump repository

El mié, 3 ene 2024, 22:26, Roman Mohr @.***> escribió:

@malt3 https://github.com/malt3 @rmohr https://github.com/rmohr I had some fun during the holidays, and now I have a branch with support for bzlmod with lockfiles and a I think a nice API.

one question on that: I am used to having .bzl files as well for managing dependencies. What exactly would be the benefit of the JSON file over those?

— Reply to this email directly, view it on GitHub https://github.com/rmohr/bazeldnf/issues/60#issuecomment-1875993161, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEFIW5UE6UJXEQY3S5HUXTYMXEH5AVCNFSM6AAAAAAZM3UPWGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVHE4TGMJWGE . You are receiving this because you commented.Message ID: @.***>

[rmohr]

Thanks @manuelnaranjo I like the direction. I did not look too much into bzlmod yet. So I am not sure if it is possible to bring those features in separately. If not, that is fine as well. But I would need some time to review and understand the final cleaned up PR. :)

[manuelnaranjo]

FYI https://github.com/rmohr/bazeldnf/pull/77 got merged, this adds bzlmod, now I'll add the lock file support that we have in our branch, we have some issues with dependency resolution being flaky with the resolver