search

PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.78k stars 1.54k forks source link

Build one `aws`, `azure`, and `gcp`-flavored image per release #12367

Open kevingrismore opened 5 months ago

kevingrismore commented 5 months ago

First check

Prefect Version

2.x

Describe the current behavior

Users must currently either pip install a prefect-* integration package at worker startup time, or build their own worker images, for the following worker types:

AWS Elastic Container Service: prefect-aws Azure Container Instances: prefect-azure Google Cloud Run: prefect-gcp Google Cloud Run V2: prefect-gcp Google Vertex AI: prefect-gcp

Installing packages at startup time can lead to a few undesirable outcomes, depending on the startup command they choose for their worker.

  1. The integration package requires a newer version of prefect than is on the chosen image and the install is attempted by prefect itself, leading to errors.
  2. The integration package requires a newer version of prefect than is on the chosen image and is installed prior to worker startup, making the running prefect version no longer in sync with the chosen image

The alternative of asking users to build their own images to maintain prefect and integration package compatibility feels like a bumpy experience for maintaining workers.

Describe the proposed behavior

Build a version.number-python3.x-aws, version.number-python3.x-azure, and version.number-python3.x-gcp image per release, where 3.x is strictly one python version. Whether it's 3.11 or 3.12 I'll leave to the experts, but our Helm chart points to 2-python3.11-kubernetes.

This should significantly reduce potential for worker startup related issues, and make onboarding workers significantly easier, especially as we approach agent deprecation. It has the added benefit of pairing Prefect versions with compatible integration packages, as our Kubernetes images already do.

If this is taken up, we should also update any documentation or guides for hosting workers that reference startup commands and images.

Example Use

image: 2-python3.11-gcp command: prefect worker start -p my-cloud-run-v2-pool

Additional context

No response

zzstoatzz commented 5 months ago

just to surface point made in internal thread: I agree this makes a lot of sense, especially because for the worker container the python minor version can be independent of the runtime image version (latter of which will almost inevitably require custom deps) so that we don't sprawl the matrix - common sense enhancement imo!