[sdk] Protobuf 4 Support

[alanhdu]

Environment

The latest kfp version pins protobuf < 4.0:

https://github.com/kubeflow/pipelines/blob/71c8e77d8a583c629e6f83dcd629fee8133f6d2a/sdk/python/requirements.in#L27

What would it take for kfp to support protobuf==4.23? It looks like Python's protobuf package went from 3.20 -> 4.21 and other packages are now starting to require protobuf > 4.

The list of breaking changes is available at https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates.

[zijianjoy]

/assign @connor-mccarthy

[stepanhruda]

Any progress on this? New packages are starting to depend on protobuf 4, making dependency resolution impossible.

[connor-mccarthy]

We plan to release at least one version of kfp==2.x.x that depends on protobuf 3 before bumping to depend on protobuf 4.

[lrodriguezesc]

Any ETA here?

[anuvedantham]

bumping this up, any ETA?

[connor-mccarthy]

After more investigation, I'm not sure that we can bump major versions within KFP SDK v2. Doing so would require removing support for protobuf 3 per the protobuf cross-version runtime guarantees. This would prevent some users from upgrading KFP SDK versions. For a other smaller dependencies with greater compatibility across major versions, this might be acceptable. This is not the case for protobuf, unfortunately.

Please feel free to provide additional information if there is something I'm missing. I will close for now.

[mynewestgitaccount]

Please feel free to provide additional information if there is something I'm missing.

@connor-mccarthy I feel like there must be something being missed here.

A number of packages have successfully managed to allow simultaneous support of protobuf 3 and 4 (e.g. take a look at google-cloud-pubsub's requirements), so I don't follow why it's an impossibility for kfp.

This would prevent some users from upgrading KFP SDK versions.

Keep in mind that it's equally true that some users are prevented from using the KFP SDK at all due to the lack of protobuf 4 support. By not allowing protobuf 4 at all, kfp introduces an impossible-to-resolve dependency conflict between itself and packages that depend on protobuf 4. More and more projects are beginning to require protobuf 4 (often transitively), so this conflict is becoming more and more problematic.

[connor-mccarthy]

I will reopen this for additional investigation and keep in our backlog. If feasible, I suspect we will need to update the protobuf version in quite a few libraries to implement this change, including kfp, kfp-kubernetes, kfp-pipeline-spec, and kfp_server_api, with careful attention to the version range that kfp specifies for each of these dependencies.

[rolweber]

Note that the major version change to 4.21.x was only for the Python support of protobuf. If you're installing from conda-forge, the underlying libprotobuf stayed at major version 3 back then. My understanding is that the serialization format remained compatible with older versions, but Python clients have to handle the switch to a different underlying package, upb (micro protocol buffers). For Python clients that use the public Python API, that change should be transparent.

https://github.com/protocolbuffers/protobuf/releases/tag/v21.0

from conda-forge...

protobuf 4.21.1 py310hd8f1fbe_0
-------------------------------
...
dependencies: 
  - libgcc-ng >=12
  - libprotobuf 3.21.1.*

There was a major version change of libprotobuf with release 22.0. Afaict, the breaking changes there affect neither Python clients nor the serialization format. https://github.com/protocolbuffers/protobuf/releases/tag/v22.0

[dslove]

Any good news so far? This issue is blocking us moving our products to kubeflow (and Vertex AI). Thanks.

[mmozerwayfair]

Just as a heads up: Support for the protobuf 3.x python packages ended on July 1st, 2023: https://protobuf.dev/support/version-support/

[connor-mccarthy]

We'll be releasing a version of the KFP SDK that is compatible with Protobuf 4 in Q1 2024. This is currently a WIP.

[mehul1811]

The 2.6.0 update was released a couple of hours ago, that solves the issue! Thanks, KFP team!