Protobuf/Grpc version policy

[JulienFS]

As this lib is made to be used in many components there's a high chance that it is to be embedded in a project already relying on protobuf.

The version pinning is quite strict at this time (~4.21) and no options were left for people with a protobuf stack pre v21 but to use older release of the lib.

For the context I'm working on implementing a fine grained authz system thanks to SpiceDB in a quite vast project leveraging tensorflow and having a lib with such strict version pinning and fast pace makes me uncomfortable.

As gRPC guarantees forward compatibility up to the next major included (for non-experimental/deprecated features) and protobuf offers forward compatibility up to the next major excluded, I would expect wider ranges.

Can you give us the details of your policy regarding this matter ?

Wild thought : as generated pb_py files are tied to a protobuf version, making a separated lib holding these files with its own protobuf version requirement would allow this lib to use a wider range of protobuf while still having version-specific files already compiled.

[vroldanbet]

@JulienFS we don't have any policy defined at the moment, this would have been unintentionally bumped to the latest version as per this contribution https://github.com/authzed/authzed-py/pull/68

Could you specify which is the gRPC version you'll like to see supported?

[jzelinskie]

I think it makes sense to make some of our non-dev dependencies ranges, so long as we can reasonably test them.

Before doing so, we need to answer a few questions: 1) What's the lowest version of grpcio, protobuf, protoc-gen-validate, and google_api? 2) What's the support window we'd like to maintain on these versions going forward?

It also looks like we can drop async_generator entirely as it's not directly imported and is specifically to backport Python 3.6 functionality into Python 3.5.

[JulienFS]

We have close to 20 components, with 10 different grpc versions ranging from 1.18.0 to 1.51.3. Among these components we have more than 10 different protobuf versions ranging from 3.5.1 to 4.21.12.

I don't know how familiar you are with protobuf version management with such number of components. As the plan for us is to use authzed in most components I'm planning to move the pb_py files in separate repos to allows our components to evolve quite independantly from protobuf version requirement, only providing extra pb_py versions if a new external dependency forces us into a new range. It would greatly reduces the burden of managing versions when integrating new external libs plus we can leverage new features for our libs and components while still maintaining compatibility with older protobuf.

Obviously if you have any ressource on how to manage such a mess I would greatly appreciate it.

EDIT: woopsy, bad manners, thank to you two for answering so fast :hugs:

[tstirrat15]

We've attempted to relax the version constraints to the degree that we can. Our desire is to be up-to-date with the currently-supported version of protobuf, so a range from that to the next major is what we'll be targeting. This is the current range: https://github.com/authzed/authzed-py/blob/main/pyproject.toml#L18

(side note: protobuf being kind-of-semver makes this look weird)

My sense is that for a system that uses a sufficient number of protobuf-generated libraries, especially if those libraries don't agree on major versions, it'd be best (although a not-insignificant lift) to pull the generation of those clients into your own codebase and use a single version of protobuf for all of the generation. This is my understanding of how the protobuf devs intend codegen to be used.

I'm going to close this issue in the meantime. If you feel there's more conversation to be had, please bug me and I'll reopen and we can continue the conversation.