search

nestauk / discovery_utils

Data analysis utils for Discovery Hub projects
MIT License
0 stars 0 forks source link

[20] Fix s3 #21

Closed Jack-0-0 closed 6 months ago

Jack-0-0 commented 6 months ago

Closes #20.

aws_access_key_id and aws_secret_access_key in boto3.client have been updated to be keyword arguments for function utils.s3.s3_client

utils.s3._download_obj updated so kwargs_boto and kwards_reading are empty dicts if none are given in the function inputs

Removed unused import from _fileobj_to_dict

beingkk commented 6 months ago

Happy for you to merge @Jack-0-0

FYI @sqr00t - don't think this has any impact on the airflow pipeline, you're probably not using the s3 util from here?

sqr00t commented 6 months ago

Just adding a comment post-merge (no issues). I'm not using the s3 util, Metaflow and Airflow has it's own S3 integration. This part of utils was an pytesting exercise for Tom.