Automation for Kubernetes development and testing. Spin up production-like environments for development, testing, and CI on demand. Use the same configuration and workflows at every step of the process. Speed up your builds and test runs via shared result caching
When you start up the garden console it spawns a daemon for mutagen that captures the current environment. Unfortunately my team uses aws-vault to generate short lived AWS creds which are cached in mutagen's environment. Every time mutagen attempts to talk to our EKS it is running the awscli command EKS sets in our ~/.kube/config, which will have the cached aws creds from the envvars. This all works until the creds expire, then the garden console shows a ton of mutagen errors with an ambiguous message:
Credentials were refreshed, but the refreshed credentials are still expired.
Unable to connect to the server: getting credentials: exec: executable aws failed with exit code 255)
Expected behavior
mutagen would ideally not even really be a name we know from garden lol, it should just silently be able to Do The Thing™.
Reproducible example
create AWS account with an EKS cluster and IAM user that uses STS tokens to authenticate
run any garden config targetting that EKS cluster
wait for creds to expire
re-launch garden with fresh creds
observe mutagen errors as daemon has no way of knowing new creds exist
Workaround
Killing mutagen whenever you see the error is an acceptable workaround, pkill mutagen is easy enough. I think it might also be reasonable to install a wrapper in the ~/.kube/config command to fetch the token that checks for expired creds and re-ups them from aws-vault if needed.
Suggested solution(s)
I don't know that this is garden's problem to be honest. I think the right solution is at worst a note in the docs for EKS to look out for this specific edge case. Maybe you could check for the presence of the AWS_SESSION_TOKEN envvar and restart the daemon every time the garden console is started again?
Additional context
This only affects users with ephemeral AWS creds, user's with static (AKIA) keys and SSO users (which are ephemeral but refreshes happen automagically) won't ever see this. The AWS_SESSION_TOKEN envvar is a strong indicator for this, but that can also be set in the aws config files so it's not 100%.
Bug
Current Behavior
When you start up the garden console it spawns a daemon for mutagen that captures the current environment. Unfortunately my team uses aws-vault to generate short lived AWS creds which are cached in mutagen's environment. Every time mutagen attempts to talk to our EKS it is running the awscli command EKS sets in our
~/.kube/config
, which will have the cached aws creds from the envvars. This all works until the creds expire, then the garden console shows a ton of mutagen errors with an ambiguous message:Expected behavior
mutagen would ideally not even really be a name we know from garden lol, it should just silently be able to Do The Thing™.
Reproducible example
Workaround
Killing mutagen whenever you see the error is an acceptable workaround,
pkill mutagen
is easy enough. I think it might also be reasonable to install a wrapper in the~/.kube/config
command to fetch the token that checks for expired creds and re-ups them from aws-vault if needed.Suggested solution(s)
I don't know that this is garden's problem to be honest. I think the right solution is at worst a note in the docs for EKS to look out for this specific edge case. Maybe you could check for the presence of the AWS_SESSION_TOKEN envvar and restart the daemon every time the garden console is started again?
Additional context
This only affects users with ephemeral AWS creds, user's with static (
AKIA
) keys and SSO users (which are ephemeral but refreshes happen automagically) won't ever see this. The AWS_SESSION_TOKEN envvar is a strong indicator for this, but that can also be set in the aws config files so it's not 100%.Your environment