PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.35k stars 1.5k forks source link

Image override in `job_variables` not applied for custom flow runs for GCP Cloud Run Push work pool #14442

Open MartijnvanElferen opened 3 days ago

MartijnvanElferen commented 3 days ago

First check

Bug summary

When creating a custom flow run for a deployment using a GCP Cloud Run Push work pool, the 'image' value specified in job_variables is not being applied to override the default image. While Prefect correctly acknowledges the custom image in the flow run creation response and UI, the actual execution on Google Cloud Run still uses the default image from the deployment, ignoring the override attempt.

Reproduction

I Created a GCP Cloud Run Push V2 work pool with the default job template, and left the image variable empty. Then, I deployed a flow using the .deploy method.

if __name__ == "__main__":
    process_pricing_xx_files.deploy(
        parameters={"host": 'sysmgr.xxx.com', "username": 'FTP-Agent'},
        work_pool_name="cloud-run-push-pricing",
        image="eu.gcr.io/xxx/prefect/prefect-pricing:production",
        build=False
    )

I attempted to run the deployment with a custom image (note the image tag!) using the CLI:

prefect deployment run \
--id "de27fd5c-eab1-4f40-8bbc-7a6684e1ceb2" \
--job-variable image="eu.gcr.io/xxx/prefect/prefect-pricing:development"

I observed that Prefect correctly acknowledged the custom image in the flow run creation response:

Created flow run 'dramatic-swan'.
└── UUID: 8eb04784-b910-43b9-8e94-48c91fc93ac1
└── Parameters: {'host': 'sysmgr.xx.com', 'username': 'FTP-Agent'}
└── Job Variables: {'image': 'eu.gcr.io/xxx/prefect/prefect-pricing:development'}
└── Scheduled start time: ...
└── URL: ...

I also verified in the Prefect UI the flow run page showing the correct overriden image under Job Variables. Then I checked Google Cloud Run and found that the job was still using the production image instead of the development image specified in the custom run.

This reproduction demonstrates that while Prefect acknowledges the custom image in its responses and UI, the actual execution on Google Cloud Run does not use the overridden image value.

Versions (prefect version output)

Version:             2.19.7
API version:         0.8.4
Python version:      3.12.3
Git commit:          60f05122
Built:               Fri, Jun 28, 2024 11:27 AM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         cloud

Additional context

Thread in Slack regarding this issue can be found here.

zzstoatzz commented 3 days ago

hi @MartijnvanElferen - thanks for the issue!

we've identified that this is specific to push pools and are working on a fix now - update to come!