search

PrefectHQ / prefect-aws

Prefect integrations with AWS.
https://PrefectHQ.github.io/prefect-aws/
Apache License 2.0
85 stars 40 forks source link

Bonkers issue with S3Bucket / MinIOCredentials #251

Closed chrisgoddard closed 1 year ago

chrisgoddard commented 1 year ago

Ok - maybe I'm taking crazy pills but I'm testing the following code:

minio_credentials_block = MinIOCredentials.load("local-minio-credentials")

s3_bucket_block = S3Bucket(
    bucket_name="flow-storage",
    credentials=minio_credentials_block,
)

print(s3_bucket_block.credentials)

I can confirm that the mini_credentials_block is completely valid and I can get a s3 client back from it that works.

BUT - print(s3_bucket_block.credentials) evaluates as an EMPTY AwsCredentials object - 

AwsCredentials(aws_access_key_id=None, aws_secret_access_key=None, aws_session_token=None, profile_name=None, region_name=None, aws_client_parameters=AwsClientParameters(api_version=None, use_ssl=False, verify=False, verify_cert_path=None, endpoint_url='http://localhost:9000', config=None))

I guess the AwsClientParameters are correct - but why is it an instance of AwsCredentials rather than MinIOCredentials? I get the same behavior whether I load for Prefect Cloud like the above example or I define the block locally.

I've literally dropped print statements into the prefect-aws source code and the arguments are coming through as expected - but as soon as the object is initialized, the value of .credentials is what you see above.

Any help would be much appreciated - at the very least seeing whether anyone else can replicate the issue.

zanieb commented 1 year ago

This is probably due to some wild default behavior in Pydantic

The field has two models with a Union so it's probably matching the first one and being coerced.

https://github.com/PrefectHQ/prefect-aws/blob/87d4ddfc11a948dd1edee6b2c1fd00539500b84b/prefect_aws/s3.py#L255

Perhaps https://github.com/pydantic/pydantic/issues/5515

chrisgoddard commented 1 year ago

Would it make sense to create an abstract base class for Credentials which both AwsCredentials and MinIOCredentials inherit from?