feat: manage aws auth through profiles

[roleyfoley]

Intent of Change

• New feature (non-breaking change which adds functionality)

Description

• Updates setCredentials when using aws to provide authentication through named profiles instead of using the access key and secret key that is passed or generated on each script call
• Extend auth support to handle different processes that can be used across any authenticated call performed by hamlet
• Consolidates the setCredentials Script into one central point. Includes a wrapper function for compatibility between how the automation scripts determine the account to use in setCredentials and the generation side
• Handle default cases and existing implementations
• Refactor manageDocker to support the new credentials process and reformat the script to align with our current practices

Our existing providers now support environment variable based configuration that in turns configures how authentication is managed to access a cloud provider

The configuration is provided trough environment variables name-spaced under HAMLET_<PROVIDER>_AUTH

Each of these variables can be qualified with the Hamlet Account id to allow for configuring different auth config for different accounts. The values mentioned above are the global values that will be used if a qualified value can't be found

the format for the qualification is HAMLET_<ACCOUNT_NAME>_<CONFIG_VAR> So to override the auth source for an account called ACCT1 the variable would be HAMLET_ACCT1_AWS_AUTH_SOURCE

AWS

• HAMLET_AWS_AUTH_SOURCE defines how the base access key and secret key are found

  • ENV the standard AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables. In order to support using these with the AWS_PROFILE variable setCredentials will copy the value of these variables into a new profile and then use that profile as a default profile to access accounts
  • USER allows you to define prefixed AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables such as USER1_AWS_ACCESS_KEY_ID. These values will be copied into a new profile and used as a default profile to then access other accounts
  • INSTANCE sets the credentials_source for all profiles to the EC2 or ECS instance metadata sources. The appropriate source is selected automatically and can be explicitly set with INSTANCE:ECS and INSTANCE:EC2 as the auth source
  • CONFIG use an existing aws profile configuration file ( ~/.aws/config or AWS_CONFIG_FILE). The process will look for a profile named based on the AWS account id or hamlet account name ( preference is given to the AWS Account Id)

• HAMLET_AWS_ACCOUNT_ID - Maps an aws account Id to a hamlet account id. This is used during automation scripts where the CMDB isn't used to find these details. Otherwise the ProviderId value from the Account is used

• HAMLET_AWS_AUTH_ROLE - Sets the name or ARN of the role that is used by the auth source to assume into to the aws account you want to access. Role name is recommended so that it can be used across different accounts

• HAMLET_AWS_AUTH_MFA_SERIAL - Sets the serial number of the MFA token that will be required to assume role into each AWS account

The default method is the ENV based approach which aligns with how the Cli normally works. If an AWS config file can be found then the mode will switch to CONFIG unless overridden

For all sources except for the CONFIG source hamlet maintains its own config and credentials file under ${HAMLET_HOME_DIR}/.aws/. This ensures that we don't override user configuration and allows us to name and manage the configuration as we need

After setCredentials has been run there will be 3 variables exported by the script

• AWS_PROFILE - sets the profile to use to access the requested account
• AWS_CONFIG_FILE - sets the config file to the hamlet config file that we have generated
• AWS_CREDENTIALS_FILE - sets the shared credentials file to the hamlet credentials file

The AWS cli supports using these values and will use it for all cli calls that are made after they have been set

The script will also validate that the chosen profile can access the requested AWS_ACCOUNT_ID as part of the script. So on each call to setCredentials there will be call to authenticate with the selected profile.

Azure

• HAMLET_AZ_AUTH_METHOD - Defines the authentication method to use

  • SERVICE - Uses a managed service principal username and password
  • MANAGED - Uses the authentication of the system running the call ( similar to instance role in AWS )
  • INTERACTIVE - Prompts the user to login via a web browser
  • NONE - Skips authentication to azure completely

• HAMLET_AZ_ACCOUNT_ID - Defines the subscription Id of the Azure account to log into

• HAMLET_AZ_TENANT_ID - Defines the tenant Id of the azure account to log into ( only required if working with multiple tenants )

• HAMLET_AZ_USERNAME - HAMLET_AZ_PASS - used when authenticating with the SERVICE* method

Motivation and Context

Closes #245 fixes #199 This standardises how we authenticate with cloud providers and defines our approach to authentication. For AWS moving to profile based configuration allows the aws cli to manage credential refresh and removes the need to share and store access keys and secret keys across different scripts.

How Has This Been Tested?

Tested locally across automation and generation scripts using the ENV, USER and CONFIG methods

Related Changes

Prerequisite PRs:

• None

Dependent PRs:

• None

Consumer Actions:

• None

[roleyfoley]

One thing found while testing

If you are generating deployments in a product that you currently don't have access to such as isolated environments and provider access is required the access check script will fail

Added a new HAMLET_AWS_AUTH_SOURCE called NONE . Which similar to the Azure setup will skip the auth process but still allow you to generate a template.

This is different from the current behaviour, but I don't mind it as a feature.

[ml019]

Added #253 to capture cleanup of credentials post hamlet command runs. Added #254 to capture cleanup of the integrator tree

[ml019]

@roleyfoley agree adding NONE is a good feature when doing local template generation testing that involves scripts like pregeneration that run provider cli commands.