AWS Credentials aren't cached for sinks that do manual request signing

lattwood commented 4 weeks ago

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When using the credential_process feature of the AWS SDK, I would expect it to cache the credentials until it gets closer to the expiration time.

This does not appear to be the case, and it is running the process every time it does a remote write to AWS, creating a lot of INFO level logs as you can see in this issue.

We need to use this functionality because there's no support for passing a session token in the vector config, and we're running it on all our Nomad agents. We use Nomad & Vault to provide access to IAM roles as required via the attached shell scripts (once they've been templated through Nomad).

Configuration

sinks:
  amp:
    type: prometheus_remote_write
    inputs:
    - prometheus_sanitization
    endpoint: "${var.sink_prometheus_remote_write_url}"
    auth:
      strategy: aws
      region: ${var.aws_region}
    aws:
      region: ${var.aws_region}
    healthcheck:
      enabled: false

Version

0.42.0

Debug Output

2024-10-25T18:25:19.204457Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=126}: aws_config::profile::credentials: constructed abstract provider from config file chain=ProfileChain { base: CredentialProcess("/bin/bash ** arguments redacted **"), chain: [] }
2024-10-25T18:25:19.204532Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=126}: aws_config::profile::credentials::exec: first credentials will be loaded from CredentialProcess("/bin/bash ** arguments redacted **") base=CredentialProcess("/bin/bash ** arguments redacted **")
2024-10-25T18:25:19.211338Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=126}: aws_config::profile::credentials: loaded base credentials creds=Credentials { provider_name: "CredentialProcess", access_key_id: "REDACTED_ACCESS_KEY", secret_access_key: "** redacted **", expires_after: "2024-10-25T19:13:34Z" }
2024-10-25T18:25:21.514860Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=127}: aws_config::profile::credentials: constructed abstract provider from config file chain=ProfileChain { base: CredentialProcess("/bin/bash ** arguments redacted **"), chain: [] }
2024-10-25T18:25:21.514886Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=127}: aws_config::profile::credentials::exec: first credentials will be loaded from CredentialProcess("/bin/bash ** arguments redacted **") base=CredentialProcess("/bin/bash ** arguments redacted **")
2024-10-25T18:25:21.521285Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=127}: aws_config::profile::credentials: loaded base credentials creds=Credentials { provider_name: "CredentialProcess", access_key_id: "REDACTED_ACCESS_KEY", secret_access_key: "** redacted **", expires_after: "2024-10-25T19:13:34Z" }
2024-10-25T18:25:22.600666Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=128}: aws_config::profile::credentials: constructed abstract provider from config file chain=ProfileChain { base: CredentialProcess("/bin/bash ** arguments redacted **"), chain: [] }
2024-10-25T18:25:22.600692Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=128}: aws_config::profile::credentials::exec: first credentials will be loaded from CredentialProcess("/bin/bash ** arguments redacted **") base=CredentialProcess("/bin/bash ** arguments redacted **")
2024-10-25T18:25:22.608006Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=128}: aws_config::profile::credentials: loaded base credentials creds=Credentials { provider_name: "CredentialProcess", access_key_id: "REDACTED_ACCESS_KEY", secret_access_key: "** redacted **", expires_after: "2024-10-25T19:13:34Z" }
2024-10-25T18:25:32.600937Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=129}: aws_config::profile::credentials: constructed abstract provider from config file chain=ProfileChain { base: CredentialProcess("/bin/bash ** arguments redacted **"), chain: [] }
2024-10-25T18:25:32.600962Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=129}: aws_config::profile::credentials::exec: first credentials will be loaded from CredentialProcess("/bin/bash ** arguments redacted **") base=CredentialProcess("/bin/bash ** arguments redacted **")
2024-10-25T18:25:32.607491Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=129}: aws_config::profile::credentials: loaded base credentials creds=Credentials { provider_name: "CredentialProcess", access_key_id: "REDACTED_ACCESS_KEY", secret_access_key: "** redacted **", expires_after: "2024-10-25T19:13:34Z" }
2024-10-25T18:25:36.517717Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=130}: aws_config::profile::credentials: constructed abstract provider from config file chain=ProfileChain { base: CredentialProcess("/bin/bash ** arguments redacted **"), chain: [] }
2024-10-25T18:25:36.517742Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=130}: aws_config::profile::credentials::exec: first credentials will be loaded from CredentialProcess("/bin/bash ** arguments redacted **") base=CredentialProcess("/bin/bash ** arguments redacted **")
2024-10-25T18:25:36.523200Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=130}: aws_config::profile::credentials: loaded base credentials creds=Credentials { provider_name: "CredentialProcess", access_key_id: "REDACTED_ACCESS_KEY", secret_access_key: "** redacted **", expires_after: "2024-10-25T19:13:34Z" }
2024-10-25T18:25:42.601059Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=131}: aws_config::profile::credentials: constructed abstract provider from config file chain=ProfileChain { base: CredentialProcess("/bin/bash ** arguments redacted **"), chain: [] }
2024-10-25T18:25:42.601087Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=131}: aws_config::profile::credentials::exec: first credentials will be loaded from CredentialProcess("/bin/bash ** arguments redacted **") base=CredentialProcess("/bin/bash ** arguments redacted **")
2024-10-25T18:25:42.607754Z  INFO sink{component_kind="sink" component_id=amp component_type=prometheus_remote_write}:request{request_id=131}: aws_config::profile::credentials: loaded base credentials creds=Credentials { provider_name: "CredentialProcess", access_key_id: "REDACTED_ACCESS_KEY", secret_access_key: "** redacted **", expires_after: "2024-10-25T19:13:34Z" }

Example Data

This is the contents of the file located at the value of the AWS_CONFIG environment variable, or "~/.aws/config"

[default]
region = ${var.aws_region}
credential_process = /bin/bash {{ env "NOMAD_SECRETS_DIR" }}/aws/credential_process

This is the contents of the file referred to by {{ env "NOMAD_SECRETS_DIR" }}/aws/credential_process

#!/bin/bash
set -euo pipefail
{{ with secret "${var.vault_aws_secret_backend_role_path}" }}
echo "{
    'Version': 1,
    'AccessKeyId': '{{ .Data.access_key }}',
    'SecretAccessKey': '{{ .Data.secret_key }}',
    'SessionToken': '{{ .Data.security_token }}',
    'Expiration': '`/usr/bin/date --iso-8601=seconds --date="@$(( $( /bin/stat -c '%Y' {{ env "NOMAD_SECRETS_DIR" }}/aws/credential_process ) + {{ .LeaseDuration }}))"`'
}" | sed s/\'/\"/g
{{ end }}

Additional Context

No response

References

No response

jszwedko commented 3 weeks ago

Hi @lattwood ,

Thanks for this report! We just delegate to the AWS Rust SDK for credentials loading, so it seems like there might be an issue upstream. I'm not seeing any reports there though 🤔 We are also behind on aws-config (ref: https://github.com/vectordotdev/vector/pull/20663) so it could be a bug that they've fixed.

jszwedko commented 3 weeks ago

Oh, actually, I missed that this was the Prometheus Remote Write sink which does somewhat manual request signing. It does seem like we might not be using the cached credential provider there and instead, as you observed, loading them on each request.

lattwood commented 4 days ago

@jszwedko any thoughts on when this might be fixed? I'm trying to dig into some issues w/ remote shipping/prom writing & having stderr gummed up with a ton of INFO logs is a bit of a problem.

lattwood commented 4 days ago

screenshot attached.

pretty bad 😅

jszwedko commented 4 days ago

It is pretty bad. Unfortunately I'm not sure exactly when it'll be addressed, but I can say that bugs are prioritized based on impact and severity. Of course, PRs always welcome if someone feels motivated.

vectordotdev / vector