Bazel Central Repository versions of protobuf do not reflect C++ version compatibility

[gonzojive]

What version of protobuf and what language are you using? v21.7 and v27.3 as published on Bazel Central Repository

Language: C++

What operating system (Linux, Windows, ...) and version? Linux, Ubuntu LTS

What runtime / compiler are you using (e.g., python version or gcc version) unsure

What did you do?

Defined a bazel module that uses C++ APIs in v21.7 that were broken in v26 (the syntax() method of FileDescriptor):

module(
    name = "protobuf_javascript_gonzojive",
    version = "3.21.5",
)

bazel_dep(name = "protobuf", version = "21.7", repo_name = "com_google_protobuf")
bazel_dep(name = "abseil-cpp", version = "20211102.0")

When I use this module as follows, bazel's minimum version selection algorithm will select v27.3, which results in a compilation error:

module(
    name = "test_1",
    version = "0.0.0",
)

bazel_dep(name = "protobuf_javascript_gonzojive", version = "3.21.5")
#bazel_dep(name = "protobuf", version = "21.7")
bazel_dep(name = "protobuf", version = "27.3")

The following results in no compilation error because the version of the repository chosen by bazel is compatible with protobuf_javascript_gonzojive:

module(
    name = "test_1",
    version = "0.0.0",
)

bazel_dep(name = "protobuf_javascript_gonzojive", version = "3.21.5")
bazel_dep(name = "protobuf", version = "21.7") # Notice 21.7 instead of 27.3

What did you expect to see

I would expect some way to specify the bazel module dependencies of my protobuf_javascript_gonzojive bazel module to ensure it will compile when used elsewhere.

The versioning policy for protobuf is quite intentional. I'm not sure if the way protobuf is currently being packaged and released to the Bazel Central Registry is as intentional or not. All of the releases so far have the same compatibility level, so bazel's minimum version selection algorithm will select the max version used by any dependency, even if there are different C++ major versions within those different bazel module versions.

[zhangskz]

We previously received guidance from Bazel that "If a project has to ship incompatible changes frequently, it’s actually best to not increase the compatibility level. Otherwise, it will cause the dependency resolution to fail frequently and force the end users to use overrides."

I believe this is because Bzlmod does not allow different compatibility levels in the resolved dependency graph, so this would cause dependency resolution to fail frequently.

So it appears for upgrades from v21.x -> v27.x, the best practice is actually not to raise the compatibility level since only some users would hit incompatible changes (e.g. removal of syntax() method) requiring overrides and we should reserve compatibility level for more widespread compatibility breaks.

[reddaly]

Did you consider releasing different bzlmod packages, one for the C++ libraries and another for protoc?

[gonzojive]

There's some discussion of this on bazel chat.

There is a max_compatibility_level attribute in bazel_dep that dependents could use to express "I am compatible with protoc 27.1 through 29.*"

max_compatibility_level: The maximum compatibility_level supported for the module to be added as a direct dependency. The version of the module implies the minimum compatibility_level supported, as well as the maximum if this attribute is not specified.

Could this help address the problem highlighted above?

So it appears for upgrades from v21.x -> v27.x, the best practice is actually not to raise the compatibility level since only some users would hit incompatible changes.

A dependent module would specify bazel_dep(name = "protobuf", version = "27.1", max_compatibility_level = 28) when they depend on protobuf. When protobuf v28 is needed by some other module, bazel would resolve protobuf to v28 without needing to update the bazel_dep above.

One issue that might arise is you don't know what the max_compatibility_level is for a module because you don't know, at time of publishing, if your module is compatible with future major releases. Updating the max_compatibility_level of previous releases... would require updating those old releases . I'm not sure if that's a big problem.