PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.96k stars 1.57k forks source link

Make credentials optional for S3Bucket block #13023

Open rmnvncnt opened 8 months ago

rmnvncnt commented 8 months ago

Description

S3Bucket blocks implicitely require a set of credentials to work, even if a the block can be declared without one. For instance, it is possible to register this block without issue :

from prefect_aws import S3Bucket

block = S3Bucket(
    bucket_name="foo",
    bucket_folder="bar",
)

block.save("foo")

Yet, using the block (as a result_storage option, for instance) will trigger :

Failed("Task run encountered an exception AttributeError: 'dict' object has no attribute 'get_params_override'")

Proposal

zzstoatzz commented 6 months ago

hi @rmnvncnt - can you show your flow code that resulted in this error?

Failed("Task run encountered an exception AttributeError: 'dict' object has no attribute 'get_params_override'")

something seems a little odd about that error

NodeJSmith commented 5 months ago

FWIW, I have an S3 Bucket block with no credentials, as we have prefect running on an EC2 that it can get the creds from, and it works fine, no errors