Support default AWS Credentials chain

[davehowell]

This PR addresses this issue to provide functionality to support the default AWS Credentials chain https://github.com/snowplow/dataflow-runner/issues/36

• Tests were added to increase coverage of the new functionality

• I have successfully tested this in my AWS account, with the snowflake-transformer / snowflake-loader as detailed in these docs https://github.com/snowplow-incubator/snowplow-snowflake-loader/wiki/Setup-Guide

• Both the cluster.json and the playbook.json config files now support using the value "default" e.g.

     "credentials":{
        "accessKeyId":"default",
        "secretAccessKey":"default"
     }

• The previous functionality is still supported, i.e. providing one of "env", "iam" or static credentials (although I personally think this is redundant at best, and not great security practices at worst). Compatibility is important

---

As part of setting up the dev / build environment I also had to address these issues with the set up https://github.com/snowplow/dataflow-runner/issues/62

So those fixes are also included in this PR or submitted as PRs to the ansible-playbooks repo:

• https://github.com/snowplow/ansible-playbooks/pull/135
• https://github.com/snowplow/ansible-playbooks/pull/136

@chuwy I would really appreciate review and feedback

[coveralls]

Coverage increased (+0.08%) to 94.053% when pulling 32a2dc3d637122383dfa7e59fab411901576e5c9 on davehowell:feature/support-default-creds-chain into cab2958fc926025fbcf2af4ea16941a0cd99dfdd on snowplow:master.

[oguzhanunlu]

hey @davehowell , thanks for the PR! I had a quick look and it seems good but I'll need to test it before merging and I can do this in the upcoming days.

In the meantime, could you please create issue for each of your commits and update your commit messages so that you have 1:1 relationship for issues and commits, have a look at the commit history in master.

[oguzhanunlu]

Local vagrant setup failed with the following:

default: failed: [127.0.0.1] => (item=apt-get install -y winehq-stable) => {"changed": true, "cmd": ["apt-get", "install", "-y", "winehq-stable"], "delta": "0:00:01.161148", "end": "2020-08-13 17:07:06.049241", "item": "apt-get install -y winehq-stable", "rc": 100, "start": "2020-08-13 17:07:04.888093"}
default: stderr: E: There are problems and -y was used without --force-yes

[davehowell]

@oguzhanunlu I had a lot of trouble getting the dev environment set up, see the following for all the fixes https://github.com/snowplow/dataflow-runner/issues/62

I've squashed it to one commit because it's all about closing #36

[oguzhanunlu]

Thanks @davehowell for the swift response! We'll handle the playbook PR first and see if it is fixed

[oguzhanunlu]

hey @davehowell , fyi, we'll switch from ansible+vagrant setup to docker for this repository. I'll prepare a PR to do the transformation and we'll rebase your work on top of that

[oguzhanunlu]

Hey @davehowell , I just cherry-picked your commit into ongoing release branch at https://github.com/snowplow/dataflow-runner/pull/64 and will close this one. Many thanks for the PR!