[Task]: Update the minor version of cloudpickle library prior to Beam release.

tvalentyn commented 2 years ago

What needs to happen?

If a Beam dependency has a flexible upper bound, users will download the most recent compatible version of a dependency at sdk installation time. Overtime, the version used at job submission may become newer than the version installed in a released Beam container. Given that forwards-compatiblity of a pickle library is not guaranteed, the pipeline may fail to unpickle at runtime.

To mitigate, pickle libraries should be specified in install_requires with tight version bounds limiting to a particular minor version. This can cause inconvenience to Beam user, if we depend on an old version of a library. Therefore, we should periodically update the version we use, at least once per release cycle.

We can consider to close this issue when either condition is met:

Beam vendors cloudpickle.
Beam stages used version of cloudpickle at pipeline submission time, and uses it at runtime.
Beam communicates to the users a list of vetted versions of each Beam dependency that users must install to use Beam in a supported configuration. Then, we can open up a version range.

Until then, don't close this issue, instead, move it to the next release milestone after updating the version in https://github.com/apache/beam/blob/master/sdks/python/setup.py

Issue Priority

Priority: 3

Issue Component

Component: sdk-py-core

kennknowles commented 1 year ago

Looks like we are on the latest

johnjcasey commented 1 year ago

updated for 2.45: https://github.com/apache/beam/pull/25143

damccorm commented 1 year ago

Moving to 2.47 since there is not a new release

damccorm commented 1 year ago

Still on latest, so I'll move to 2.48

riteshghorse commented 1 year ago

Still on latest, moving to 2.49.0

Abacn commented 1 year ago

2.2.1 is still the latest, moving to 2.50.0

lostluck commented 1 year ago

2.50 release manager here. This issue is currently tagged for the 2.50.0 release, which cuts in a week on August 9th.

Please complete work and get it into the main branch in that time, or move this issue to the 2.51 Milestone: https://github.com/apache/beam/milestone/15

riteshghorse commented 1 year ago

No updates to cloudpickle version. Moving to 2.51.0

damccorm commented 1 year ago

It looks like there's been a major version upgrade (and no minor version upgrade) - https://pypi.org/project/cloudpickle/#history

@tvalentyn I don't think this should be a release blocker and probably represents non-trivial work to investigate/upgrade.

I'm going to move the release blocker and we should think about a good way to fund this going forward

jrmccluskey commented 11 months ago

Similar status as before for cloudpickle, major increment to 3.0.0 was before the 2.52.0 branch cut. @tvalentyn any objections to rolling this up to 2.54?

tvalentyn commented 11 months ago

no objections.

lostluck commented 9 months ago

There's one week until the 2.54.0 cut and this issue is tagged for that release, if possible/necessary please complete the necessary work before then, or move this to the 2.55.0 Release Milestone.

Abacn commented 8 months ago

2.2.1 is still the latest of 2.x as of 2.55.0 cut

damccorm commented 6 months ago

No update here

Abacn commented 1 month ago

We should move to the next milestone each time (not just remove it)

Abacn commented 1 month ago

per #32617 cloudpickle 2.2.1 looks fine for Python 3.12, however we should revisit cloudpickle 3.0 later, move to the next milestone.

apache / beam