Support for latest protobuf (>5)

[matejsp]

Is your feature request related to a problem? Please describe. We already are using latest protobuf (protobuf==5.27.2) and would like to use it with grpc_requests. We are using robot framework calling grpc calls.

Describe the solution you'd like Support for latest protobuf.

Describe alternatives you've considered Downgrading protobuf to 4.25.3.

Issues with pinning:

Collecting grpcio>=1.60.1 (from grpc-requests==0.1.19)
  Downloading ***********/packages/grpcio/1.65.1/grpcio-1.65.1-cp311-cp311-manylinux_2_34_aarch64.whl (108.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 108.5/108.5 MB 20.9 MB/s eta 0:00:00
Collecting grpcio-reflection>=1.60.1 (from grpc-requests==0.1.19)
  Downloading ***********/packages/grpcio-reflection/1.65.1/grpcio_reflection-1.65.1-py3-none-any.whl (22 kB)
Collecting protobuf<5.0.0dev,>=3.20.3 (from grpc-requests==0.1.19)
  Downloading ***********/packages/protobuf/4.25.3/protobuf-4.25.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_34_aarch64.whl (966 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 966.4/966.4 kB 16.1 MB/s eta 0:00:00
Collecting google-api-core>=2.11.1 (from grpc-requests==0.1.19)
  Downloading ***********/packages/google-api-core/2.19.1/google_api_core-2.19.1-py3-none-any.whl (139 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.4/139.4 kB 233.6 MB/s eta 0:00:00
Collecting cryptography>=41.0.7 (from grpc-requests==0.1.19)
  Downloading ***********/packages/cryptography/42.0.8/cryptography-42.0.8-cp311-cp311-manylinux_2_34_aarch64.whl (1.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 23.0 MB/s eta 0:00:00
Collecting cffi>=1.12 (from cryptography>=41.0.7->grpc-requests==0.1.19)
  Downloading ***********/packages/cffi/1.16.0/cffi-1.16.0-cp311-cp311-linux_aarch64.whl (422 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 423.0/423.0 kB 129.7 MB/s eta 0:00:00
Collecting googleapis-common-protos<2.0.dev0,>=1.56.2 (from google-api-core>=2.11.1->grpc-requests==0.1.19)
  Downloading ***********/packages/googleapis-common-protos/1.63.2/googleapis_common_protos-1.63.2-py2.py3-none-any.whl (220 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 220.0/220.0 kB 100.6 MB/s eta 0:00:00
Collecting proto-plus<2.0.0dev,>=1.22.3 (from google-api-core>=2.11.1->grpc-requests==0.1.19)
  Downloading ***********/packages/proto-plus/1.24.0/proto_plus-1.24.0-py3-none-any.whl (50 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.1/50.1 kB 188.1 MB/s eta 0:00:00
Collecting google-auth<3.0.dev0,>=2.14.1 (from google-api-core>=2.11.1->grpc-requests==0.1.19)
  Downloading ***********/packages/google-auth/2.32.0/google_auth-2.32.0-py2.py3-none-any.whl (195 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 195.5/195.5 kB 173.0 MB/s eta 0:00:00
Collecting requests<3.0.0.dev0,>=2.18.0 (from google-api-core>=2.11.1->grpc-requests==0.1.19)
  Downloading ***********/packages/requests/2.32.3/requests-2.32.3-py3-none-any.whl (64 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.9/64.9 kB 209.2 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of grpcio-reflection to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core>=2.11.1 (from grpc-requests==0.1.19)
  Downloading ***********/packages/google-api-core/2.19.0/google_api_core-2.19.0-py3-none-any.whl (139 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.0/139.0 kB 223.5 MB/s eta 0:00:00
ERROR: Cannot install grpc-requests and grpc-requests==0.1.19 because these package versions have conflicting dependencies.

The conflict is caused by:
    grpc-requests 0.1.19 depends on protobuf<5.0.0dev and >=3.20.3
    google-api-core 2.19.0 depends on protobuf!=3.20.0, !=3.20.1, !=4.21.0, !=4.21.1, !=4.21.2, !=4.21.3, !=4.21.4, !=4.21.5, <5.0.0.dev0 and >=3.19.5
    grpcio-reflection 1.65.1 depends on protobuf<6.0dev and >=5.26.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

Basically latest grpcio-reflection 1.65.1 needs newer protobuf than grcp_requests ...

[artificial-aidan]

Changes we need to consider with 5.x.x

https://protobuf.dev/news/v26/

[ViridianForge]

Thanks for calling that out @artificial-aidan. @matejsp - I'll pick this up when I get a chance, I believe we'll have to write some shims to handle the breaking method changes that were made in 5.0.

Tangentially - we should look at adjusting the CI tests to run with different versions of protobuf.

[wesky93]

I think it's time to give up on single versioning. In the future, why not specify the protobuf and grpc version support coverage per major version in grpc_requests?

[wesky93]

like this

[artificial-aidan]

I think I disagree with that. Because the dependencies are pretty simple in grpc_requests and managing the different protobuf versions with helper functions is pretty simple.

Managing multiple version publishing, and adding back porting fixes/features becomes painful.

[wesky93]

I understand. So, in this case if we include support for protobuf>=5.27.2 or later versions, and then add the dependency restriction after including support for protobuf>=27.2 and later versions?

[artificial-aidan]

Starting in I think 4.21.x protobuf changed how they version their packages. The minor/patch versions are reserved for protobuf versioning, and the major version is language specific, in case language packages need to make breaking changes. So as far as grpc_requests goes, we just need to be restricting to major versions (because minor bumps won't be breaking changes). So after adding support for protobuf 5.x.x we just need to set the dependency restriction to <=5. And when 6 is released we can evaluate if the breaking changes effect the code or not.

[ViridianForge]

Also - we do have precedent for adding a shim to do specific setup based on the version of protobuf version available to the environment running the code. For Protobuf 5, we could (hopefully) just add to that shim to use the changed methods to add the requested support.

In terms of messaging what we support, I think we could just make it clear that grpc_requests will follow support timelines for its dependencies. (i.e. when we removed 3.7 compatibility because that was end of life) Maybe even make that clear for important ones like protobuf with a wee badgey the way we do with Python.

[artificial-aidan]

Yeah that would be good. Protobuf has clear support timelines too.

https://protobuf.dev/support/version-support/#python

[ViridianForge]

Support and documentation on this ticket has just been merged into develop. I'll get a new release out this week.

Used this as a good excuse to leverage nox to run tests in more combinations locally and expand the test matrices in the PR pipeline to cover supported versions.