Some Vault engines require a unique call for multiple fields

[julien-carsique-sonarsource]

Some engines like AWS have use cases that require a unique call for getting multiple fields. Else the returned values are not consistent and not working.

In this case, AWS STS is delivering temporary credentials made of three values: access_key, secret_key, and security_token. https://developer.hashicorp.com/vault/docs/secrets/aws#sts-assumerole

This will not work:

env:
  AWS_ACCESS_KEY_ID:  VAULT[aws/sts/<PRESET> access_key]
  AWS_SECRET_ACCESS_KEY:  VAULT[aws/sts/<PRESET> secret_key]
  AWS_SESSION_TOKEN:  VAULT[aws/sts/<PRESET> security_token]

The current workaround is to call the vault manually:

task:
  set_aws_credentials_script: |
    export VAULT_TOKEN=$(curl -s -X PUT -H "X-Vault-Request: true" -d "{\"jwt\":\"${CIRRUS_OIDC_TOKEN}\",\"role\":\"${CIRRUS_VAULT_ROLE}\"}" "${CIRRUS_VAULT_URL}/v1/auth/${CIRRUS_VAULT_AUTH_PATH}/login" | jq -r .auth.client_token)
    export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \
      $(curl -s -H "X-Vault-Request: true" -H "X-Vault-Token: $VAULT_TOKEN" "$CIRRUS_VAULT_URL/v1/aws/sts/<PRESET>" \
        | jq -rc '.data|(.access_key,.secret_key,.security_token)'))
    {
      echo "AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}"
      echo "AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}"
      echo "AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN}"
    } >> $CIRRUS_ENV

This issue will happen for all engines returning temporary tokens with multiple values. Even for other use cases, and static values, it sounds like an improvement to always make a unique call to the vault.

What do you think of using a cache to make a unique call and then assign the values to the requested fields?

[edigaryev]

I think that due to the dynamic nature of Secret Engines you've mentioned it's also possible for some cases to require a new value each time, for example, caching a VAULT[generators/random password] may result in the lack of entropy.

What do you think about a slightly more explicit approach that will cover both your and the aforementioned cases:

1. VAULT[key ...] will continue to always retrieve the new value (current behavior), but it will also cache the value under key internally (we currently support only one Vault per task run, so no need for namespacing).

2. VAULT_CACHED[key ...] can be used to retrieve the key from the cache. If no key exists in the cache — it will act like VAULT[key ...].

This way you can completely ignore the cache for some keys by always using VAULT[key ...], and for the keys that need to be cached use VAULT_CACHED[key ...]:

Your example would look like:

env:
  AWS_ACCESS_KEY_ID: VAULT_CACHED[aws/sts/<PRESET> access_key]
  AWS_SECRET_ACCESS_KEY: VAULT_CACHED[aws/sts/<PRESET> secret_key]
  AWS_SESSION_TOKEN: VAULT_CACHED[aws/sts/<PRESET> security_token]