PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
16.02k stars 1.57k forks source link

Workers should identify image sha if applicable #15432

Open EmilRex opened 2 weeks ago

EmilRex commented 2 weeks ago

Describe the current behavior

Currently you can see which image was used to execute a deployment-triggered flow run by looking at the flow run job variables, the deployment itself, and the work pool configuration. If the image used is not uniquely enough identified (i.e. prefecthq/prefect:3-latest) it is not possible to determine exactly which image was used.

Describe the proposed behavior

It would be exceptionally nifty if the image sha used to run a deployment was logged or otherwise associated with a flow run. Additionally it would be great to also see the image name used as it can possibly live in the three places described above. The image sha can be particularly helpful when an image may or may not be cached on a node, such as in k8s.

Example Use

No response

Additional context

No response

zzstoatzz commented 2 weeks ago

good idea @EmilRex! what do you think about this?

i.e. as a related resource on a prefect.worker.executed-flow-run event image

but this would also show in prefect.worker.submitted-flow-run events

setup make a deployment ```python from prefect import flow if __name__ == "__main__": flow.from_source( source="https://gist.github.com/3eb9333625d465cb74381c5116be7aee.git", entrypoint="tour_of_artifacts.py:main", ).deploy(name="test-flow", work_pool_name="docker-work", build=False, push=False) ``` start the worker ```bash prefect worker start --pool docker-work ``` trigger a run ```bash prefect deployment run 'main/test-flow' --param 'data={"value": "2"}' -jv image="prefecthq/prefect:3-latest" ```
EmilRex commented 2 weeks ago

@zzstoatzz I think that would be great, especially since we're showing the events so well with the new run tracing. I love it!