Open uuazed opened 1 year ago
I am getting the same error. When using AWS Credentials Block, credentials in the push_project_to_s3 function is set with the following values.
{'aws_access_key_id': 'xxxXXXxxx', 'aws_secret_access_key': 'xxxXXXXxxxXXXxxx', 'aws_session_token': None, 'profile_name': None, 'region_name': 'ap-northeast-2', 'aws_client_parameters': { 'api_version': None, 'use_ssl': True, 'verify': True, 'verify_cert_path': None, 'endpoint_url': None, 'config': None}}
And when I executed the code below client = boto3.client( "s3", credentials, client_parameters, config=Config(**advanced_config) )
I get the following error because of the arguments profile_name, aws_client_parameters, which are not required by boto3.client. TypeError: client() got an unexpected keyword argument 'profile_name'
If you temporarily remove the unnecessary fields as shown below, the deployment will be successful. del credentials['profile_name'] del credentials['aws_client_parameters']
But when I run the flow, I get the same error in pull_project_from_s3 :)
Additionally, it would be nice to have a way to authenticate via s3 block.
Also encountering this. Temporary workaround is to create Secret
blocks for each of the required values and build the dictionaries manually:
Secret(value=...).save('deploy-s3-access-key', overwrite=True)
Secret(value=...).save('deploy-s3-secret-key', overwrite=True)
Secret(value=...).save('deploy-s3-endpoint', overwrite=True)
# prefect.yaml
push: # same thing for pull
- prefect_aws.projects.steps.push_project_to_s3:
# ...
credentials:
aws_access_key_id: "{{ prefect.blocks.secret.deploy-s3-access-key }}"
aws_secret_access_key: "{{ prefect.blocks.secret.deploy-s3-secret-key }}"
client_parameters:
endpoint_url: "{{ prefect.blocks.secret.deploy-s3-endpoint }}"
This is potentially fixed by PrefectHQ/prefect-aws#322
I've created a new project using the docker-s3 template. The credentials of my AWS service account are stored in a prefect block. According to the prefect project tutorial, this is the recommended setup. This is the push section in my deployment.yaml :
When initiating a deploy via
prefect deploy --name ...
, the step fails withTypeError: Session.client() got an unexpected keyword argument 'profile_name'
Expectation / Proposal
Using credentials stored in a prefect blocks works. Even better, I can use a prefect s3 block directly.
Traceback / Example