[Feature request] Consider installing the same artifact in Beam Python RC Docker containers as is being published to PyPi

tvalentyn commented 10 months ago

What happened?

Pipeline I ran failed with an error:

_Pipeline construction environment and pipeline runtime environment are not compatible. If you use a custom container image, check that the Python interpreter minor version and the Apache Beam version in your image match the versions used at pipeline construction time. Submission environment: beam:version:sdk_base:apache/beam_python3.11_sdk:2.50.0rc1. Runtime environment: beam:version:sdk_base:apache/beam_python3.11sdk:2.50.0. Worker ID: beamapp-valentyn-08220117-08211817-m76c-harness-v38w

The rootcause is: starting from 2.50.0, we no longer stage Beam SDK. Starting from several releases back we also check that submission and runtime versions match. However Python Docker containers we build for RCs don't install the SDK RC version of Beam SDK tarball.

docker run --rm -it --entrypoint=/bin/bash apache/beam_python3.10_sdk:2.50.0.rc1

cat /usr/local/lib/python3.10/site-packages/apache_beam/version.py
...
"""Apache Beam SDK version information and utilities."""

__version__ = '2.50.0'

This issue blocks further validation of RC1 for Python Dataflow pipelines.

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

[X] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

tvalentyn commented 10 months ago

cc: @AnandInguva

tvalentyn commented 10 months ago

Avenues to address:

The right thing to do would be to install in RC Docker containers the same package content that is being published as RC. however this might be unplanned work for 2.50.0 release and less straightforward given how we currently produce an RC artifact (there is a github actions workflow that makes an edit to version.py and that is separate from workflows that publish Docker containers). cc: @damccorm as FYI re release improvements.
strip RC suffixes during validation in https://github.com/apache/beam/blob/448184e80e60c3774bcfc3aac138d64ccf7444a7/sdks/python/apache_beam/runners/worker/bundle_processor.py#L839 . We could make a targeted fix for 2.51.0 branch for that.
revert https://github.com/apache/beam/pull/27436 for now and follow up for 2.51.0.

chamikaramj commented 10 months ago

The rootcause is: starting from 2.50.0, we no longer stage Beam SDK

To clarify, what do you mean by this ?

I could run a Python x-lang pipeline on the RC by installing the SDK via the zip file and providing the same file to the "--sdk_location" flag.

https://dist.apache.org/repos/dist/dev/beam/2.50.0/python/apache-beam-2.50.0.zip

AnandInguva commented 10 months ago

Starting from several releases back we also check that submission and runtime versions match.

Can we skip this condition check when the sdk is an RC?

AnandInguva commented 10 months ago

The rootcause is: starting from 2.50.0, we no longer stage Beam SDK

To clarify, what do you mean by this ?

I could run a Python x-lang pipeline on the RC by installing the SDK via the zip file and providing the same file to the "--sdk_location" flag.

https://dist.apache.org/repos/dist/dev/beam/2.50.0/python/apache-beam-2.50.0.zip

Beam python downloads the sdk sdist or wheel from pypi and stages it to the staging environment. Then the boot.go looks for this staged file and installs it on Dataflow. We do this for Runner v1 since the default container for runner v1 doesn't contain Beam but from 2.50.0, runner v1 is deprecated, so we stopped staging SDK since Runner v2 containers have Beam installed in them and there is no need to stage Beam SDK.

If a tarball is passed to the --sdk_location, we stage it and install it on the Dataflow but by default we don't stage anything from 2.50.0

damccorm commented 10 months ago

I'm basically always +1 on reverting in situations like this; I think it is almost always the fastest and safest thing to do, especially when the feature is helping us but not providing customer value. Reverting a revert is easy, and then we can pin an extra commit on that PR to get the fix forward in the next release.

I put up https://github.com/apache/beam/pull/28094 to do this

strip RC suffixes during validation

I'm +1 on this as the long term fix. Its simple and still allows us to exercise all the functionality we need to do validation.

damccorm commented 10 months ago

Discussed offline, we're going to try rolling forward and only revert if we run into issues. @tvalentyn is going to take this forward

tvalentyn commented 10 months ago

fix merged to master and CP created for release branch. Repurposing the issue for potential follow up work:

The right thing to do would be to install in RC Docker containers the same package content that is being published as RC.

lostluck commented 10 months ago

CP Merged in. I'm going to start the RC2 process soon. Please validate fix once RC2 is available.

apache / beam