PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.95k stars 1.57k forks source link

Flow run hangs when persisting to s3 without S3Bucket.load #15339

Closed DGolubets closed 2 weeks ago

DGolubets commented 2 weeks ago

Bug summary

The following code hangs:

from prefect import flow

from prefect_aws.s3 import S3Bucket

# hangs with the following line commented
# s3_bucket_block = S3Bucket.load("s3")

@flow(
    persist_result=True,
    result_storage="s3-bucket/s3",
)
def flow1():
    return "result"

if __name__ == "__main__":
    flow1()

If I uncomment s3_bucket_block = S3Bucket.load("s3") then it runs fine.

Version info (prefect version output)

Version:             3.0.1
API version:         0.8.4
Python version:      3.10.12
Git commit:          c6b2ffe1
Built:               Fri, Sep 6, 2024 10:05 AM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         server
Pydantic version:    2.9.1
Integrations:
  prefect-kubernetes: 0.5.0
  prefect-aws:       0.5.0
  prefect-dask:      0.3.0
  prefect-ray:       0.4.0

Additional context

No response

zzstoatzz commented 2 weeks ago

hi @DGolubets - thank you for the issue!

hmm as of now I am not able to reproduce this with 3.0.1 (or on main).

are you able to load any blocks in any context?

DGolubets commented 2 weeks ago

@zzstoatzz I've tried in a clean environment and it indeed works fine. It must be something with my existing project then.