search

nestauk / daps_utils

Tools for setting up and running pipelines in a Data Analytics and Production System (DAPS).
0 stars 1 forks source link

[70] Dynamic retrieval and setting of aws env vars #71

Closed jaklinger closed 3 years ago

jaklinger commented 3 years ago

Closes #70

docker_utils now dynamically retrieves the AWS credentials and populates them into the MetaflowTask via ARG, i.e. this optionally allows them to be retrieved by:

... # e.g. ojd_daps/config/metaflowtask/Dockerfile
ARG AWS_DEFAULT_REGION
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
...
ENV AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
ENV AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
ENV AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
...