search

garden-io / garden

Automation for Kubernetes development and testing. Spin up production-like environments for development, testing, and CI on demand. Use the same configuration and workflows at every step of the process. Speed up your builds and test runs via shared result caching
https://garden.io
Mozilla Public License 2.0
3.32k stars 268 forks source link

Mutagen daemon captures ephemeral AWS creds #5304

Open liath opened 10 months ago

liath commented 10 months ago

Bug

Current Behavior

When you start up the garden console it spawns a daemon for mutagen that captures the current environment. Unfortunately my team uses aws-vault to generate short lived AWS creds which are cached in mutagen's environment. Every time mutagen attempts to talk to our EKS it is running the awscli command EKS sets in our ~/.kube/config, which will have the cached aws creds from the envvars. This all works until the creds expire, then the garden console shows a ton of mutagen errors with an ambiguous message:

Credentials were refreshed, but the refreshed credentials are still expired.
Unable to connect to the server: getting credentials: exec: executable aws failed with exit code 255)

Expected behavior

mutagen would ideally not even really be a name we know from garden lol, it should just silently be able to Do The Thing™.

Reproducible example

Workaround

Killing mutagen whenever you see the error is an acceptable workaround, pkill mutagen is easy enough. I think it might also be reasonable to install a wrapper in the ~/.kube/config command to fetch the token that checks for expired creds and re-ups them from aws-vault if needed.

Suggested solution(s)

I don't know that this is garden's problem to be honest. I think the right solution is at worst a note in the docs for EKS to look out for this specific edge case. Maybe you could check for the presence of the AWS_SESSION_TOKEN envvar and restart the daemon every time the garden console is started again?

Additional context

This only affects users with ephemeral AWS creds, user's with static (AKIA) keys and SSO users (which are ephemeral but refreshes happen automagically) won't ever see this. The AWS_SESSION_TOKEN envvar is a strong indicator for this, but that can also be set in the aws config files so it's not 100%.

Your environment

twelvemo commented 10 months ago

Thanks for flagging this @liath . We'll take a closer look soon, but i think this might be related to #3475