search

PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.95k stars 1.57k forks source link

Update artifacts mid flow run #10955

Open ChrisPaul33 opened 11 months ago

ChrisPaul33 commented 11 months ago

First check

Prefect Version

2.x

Describe the current behavior

Today we can only create a new artifact through the python SDK but we would like to update an existing artifact mid flow run (because the run is very long).

Describe the proposed behavior

Make the rest API update option available in the SDK. Maybe add a way to create a "special" artifact that keeps getting updated till the end of the flow run,

Example Use

No response

Additional context

No response

serinamarie commented 11 months ago

Hi @ChrisPaul33, thanks for the issue! Can you provide an example where having a keyed artifact (versions of the same artifact) wouldn't fit your use case? Thanks!

ChrisPaul33 commented 11 months ago

I have a lot of websites that I ping and the whole operation takes a few hours. I don't want it to have multiple artifacts and I would like to see the whole data in the same place. Sometimes we like to take the data mid-run to start analyze the results we got.

serinamarie commented 11 months ago

Gotcha, and you mentioned in community that this is in relation to updating a single artifact (same UUID) which is an important distinction.

  1. I think that data should probably be made a required field for updating an artifact, using the corresponding Pydantic model: https://github.com/PrefectHQ/prefect/blob/70f2d40d7b3999a62d83171e4caa6843502938ea/src/prefect/client/schemas/actions.py#L580-L585

  2. We can add a method for updating artifacts to the sdk here: https://github.com/PrefectHQ/prefect/blob/70f2d40d7b3999a62d83171e4caa6843502938ea/src/prefect/artifacts.py#L1-L3

  3. Expanding our docs to include how/when to update (especially compared to generating a new artifact with the same key) would be ideal but could be done in a separate PR.

Thanks again for raising the issue, @ChrisPaul33! We always welcome contributions as well :)

EnriqueG23 commented 11 months ago

Hi @serinamarie ,

I am highly interested in contributing in PrefectHQ with a "good first issue". Could you please indicate me which one can I contribute in --perhaps this one? And also, be assigned to it?

I am looking for something related to coding not documentation issues.

Looking forward to your response and contributing with the project. Thanks.