PrefectHQ / prefect

Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
https://prefect.io
Apache License 2.0
15.29k stars 1.5k forks source link

Turn on task persistence by default #14102

Closed desertaxle closed 1 week ago

desertaxle commented 1 week ago

To ensure we get idempotency for tasks out of the box, we need to ensure that tasks persist their results by default. With this PR, persist_result will default to True for all tasks. If persist_result is set to false, then the cache_policy will be ignored.

To facilitate writing results by default, we will attempt to load a default storage block matching the default storage block configured with PREFECT_DEFAULT_RESULT_STORAGE_BLOCK. If that block doesn't exist (defaults to {hostname}-storage), we will create and save that block for the user. That block will be reused for subsequent runs.

Closes https://github.com/PrefectHQ/prefect/issues/14087

Example

Checklist

desertaxle commented 1 week ago

Related to https://github.com/PrefectHQ/prefect/issues/13800