search

airbnb / synapse

A transparent service discovery framework for connecting an SOA
MIT License
2.07k stars 251 forks source link

ec2tag watcher fails if metadata service is unavailable during a credential refresh #233

Open ianbamforth opened 7 years ago

ianbamforth commented 7 years ago

The DefaultProvider takes aws keys from environment variables / config if available, otherwise falling back to ec2 metadata IAM role. If you are using keys from the latter, they refresh automatically when they get near to expiry. If the metadata service is down at this point, Synapse enters a broken state, where it has no credentials and cannot recover. We have seen this in our production environment.

I'm currently looking at a patch whereby I specifically select the EC2Provider if no keys are provided by the environment / config, inserting the following into ec2tag.rb before the call to AWS::EC2.new:

unless ((@discovery['aws_access_key_id'] || ENV['aws_access_key_id']) \ && (@discovery['aws_secret_access_key'] || ENV['aws_secret_access_key'] )) AWS.config(:credential_provider => AWS::Core::CredentialProviders::EC2Provider.new(:retries => 0)) end

Does this seem like a reasonable approach? Happy to submit a PR.

jolynch commented 7 years ago

@ianbamforth that seems reasonable to me. If it's been working in production for you I'm happy to merge it :-)