search

WhoopInc / vagrant-s3auth

Vagrant plugin for private, versioned boxes on Amazon S3.
MIT License
108 stars 23 forks source link

AWS STS credentials not supported #33

Open jcshort opened 7 years ago

jcshort commented 7 years ago

Seeing some issues with bucket region detection using a cross account role. The AWS_REGION env var seems to be ignored when using assumed roles, and it doesn't appear that assumed roles are allowed to set region in ~/.aws/config either.

some strings redacted with caps.

aws config:

[profile VAGRANT_BOXES]
role_arn = arn:aws:iam::ACCOUNT_NUMBER:role/VAGRANT_ROLE
source_profile = OTHER_ACCOUNT
Bringing machine 'VAGRANT' up with 'virtualbox' provider...
==> VAGRANT: Box 'VAGRANT' could not be found. Attempting to find and install...
    VAGRANT: Box Provider: virtualbox
    VAGRANT: Box Version: >= 0
==> VAGRANT: Loading metadata for box 's3://BUCKET/VAGRANT'
    VAGRANT: URL: s3://BUCKET/VAGRANT
==> VAGRANT: Adding box 'VAGRANT' (v#.#.##) for provider: virtualbox
    VAGRANT: Downloading: s3://BUCKET/VAGRANT.box.##
HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/aws-sdk-core/plugins/regional_endpoint.rb:34:in `after_initialize': missing region; use :region option or export region name to ENV['AWS_REGION'] (Aws::Errors::MissingRegionError)
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/seahorse/client/base.rb:84:in `block in after_initialize'
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/seahorse/client/base.rb:83:in `each'
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/seahorse/client/base.rb:83:in `after_initialize'
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/seahorse/client/base.rb:21:in `initialize'
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/seahorse/client/base.rb:105:in `new'
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/aws-sdk-core/assume_role_credentials.rb:39:in `initialize'
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/aws-sdk-core/shared_config.rb:148:in `new'
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/aws-sdk-core/shared_config.rb:148:in `assume_role_from_profile'
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/aws-sdk-core/shared_config.rb:111:in `assume_role_credentials_from_config'
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/aws-sdk-core/credential_provider_chain.rb:94:in `assume_role_with_profile'
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/aws-sdk-core/credential_provider_chain.rb:77:in `assume_role_credentials'
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/aws-sdk-core/credential_provider_chain.rb:12:in `block in resolve'
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/aws-sdk-core/credential_provider_chain.rb:11:in `each'
    from HOME/.vagrant.d/gems/2.3.4/gems/aws-sdk-core-2.6.50/lib/aws-sdk-core/credential_provider_chain.rb:11:in `resolve'
    from HOME/.vagrant.d/gems/2.3.4/gems/vagrant-s3auth-1.3.2/lib/vagrant-s3auth/util.rb:79:in `s3_credential_provider'
    from HOME/.vagrant.d/gems/2.3.4/gems/vagrant-s3auth-1.3.2/lib/vagrant-s3auth/extension/downloader.rb:12:in `s3auth_credential_source'
    from HOME/.vagrant.d/gems/2.3.4/gems/vagrant-s3auth-1.3.2/lib/vagrant-s3auth/extension/downloader.rb:46:in `s3auth_download'
    from HOME/.vagrant.d/gems/2.3.4/gems/vagrant-s3auth-1.3.2/lib/vagrant-s3auth/extension/downloader.rb:77:in `rescue in execute_curl_with_s3auth'
    from HOME/.vagrant.d/gems/2.3.4/gems/vagrant-s3auth-1.3.2/lib/vagrant-s3auth/extension/downloader.rb:72:in `execute_curl_with_s3auth'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/util/downloader.rb:147:in `download!'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builtin/box_add.rb:459:in `download'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builtin/box_add.rb:334:in `block in box_add'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builtin/box_add.rb:326:in `each'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builtin/box_add.rb:326:in `box_add'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builtin/box_add.rb:279:in `add_from_metadata'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builtin/box_add.rb:114:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builder.rb:116:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/runner.rb:66:in `block in run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/util/busy.rb:19:in `busy'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/runner.rb:66:in `run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builtin/handle_box.rb:82:in `handle_box'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builtin/handle_box.rb:42:in `block in call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builtin/handle_box.rb:36:in `synchronize'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builtin/handle_box.rb:36:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builder.rb:116:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/runner.rb:66:in `block in run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/util/busy.rb:19:in `busy'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/runner.rb:66:in `run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builtin/call.rb:53:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/plugins/providers/virtualbox/action/check_virtualbox.rb:17:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/builder.rb:116:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/runner.rb:66:in `block in run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/util/busy.rb:19:in `busy'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/action/runner.rb:66:in `run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/machine.rb:227:in `action_raw'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/machine.rb:202:in `block in action'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/environment.rb:567:in `lock'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/machine.rb:188:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/machine.rb:188:in `action'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.9.6/lib/vagrant/batch_action.rb:82:in `block (2 levels) in run'
benesch commented 7 years ago

Ugh, the way the AWS SDK can find credentials gets more complicated with every release. Sorry about this! I haven't tested vagrant-s3auth at all with assumed roles.

Based on the stack trace, this is actually failing while constructing a dummy AWS credentials provider that's only used to print where your credentials are coming from to stdout—i.e., whether they came from an env var or a profile. I suspect you'd have a slightly different stack trace if you set AWS_REGION to some placeholder, like AWS_REGION=us-east-1. Would you mind to give that a shot? That is, run

AWS_REGION=us-east-1 vagrant up

and see if the stack trace looks any different? If it does, post it here so I can take a look!

Otherwise, I'll dig into this eventually, but it might be a while until I find the time to replicate your IAM setup.

xakraz commented 6 years ago

I think it is the STS way that is not supported.

According to the current doc, and as many other softwares do, the plugin is only capable to use credentials coming from ~/.aws/credentials formatted as a pair of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY .... and not as STS with source_profiles ....

xakraz commented 6 years ago

And I confirm that by many tries in my team

@jcshort, I would suggest you to rename your issue, "AWS STS credentials not supported" :)

xakraz commented 6 years ago

We may be able to do something with https://github.com/a2ikm/aws_config ?

lib/vagrant-s3auth/util.rb
---------------------------------
require 'aws_config'
....
      def self.s3_client(region = DEFAULT_REGION)
        # AWS STS support
        # https://github.com/aws/aws-sdk-ruby/issues/1256
        credentials_provider = self.s3_credential_provider
        if credentials_provider == ::Aws::SharedCredentials and !credential_provider.profile_name.nil? then
          role_arn = AWSConfig[credential_provider.profile_name].role_arn  || nil
          unless role_arn.nil? do
             credentials = Aws::AssumeRoleCredentials.new(
                client: Aws::STS::Client.new(region: region, credentials: credentials),
                duration_seconds: "1800",
                role_arn: role_arn,
                role_session_name: "vagrant"
             )
            ::Aws::S3::Client.new(
              region: region,
              credentials: credentials
            )
          end
        end

        # Otherwise, return the simple client
        ::Aws::S3::Client.new(region: region)
     end
benesch commented 6 years ago

Hmm, looks promising. Might you be interested in submitting a PR, @xakraz?

xakraz commented 6 years ago

Yes, I will try to test it this week and submit a PR :D

benesch commented 6 years ago

Awesome! Let me know if I can be of any help.

On Mon, Feb 5, 2018 at 6:46 PM, Xavier Krantz notifications@github.com wrote:

Yes, I will try to test it this week and submit a PR :D

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/WhoopInc/vagrant-s3auth/issues/33#issuecomment-363260387, or mute the thread https://github.com/notifications/unsubscribe-auth/AA15IDMMh6dMtazvIW3yROK1T-PksGa7ks5tR5LpgaJpZM4OOwvm .

jcshort commented 6 years ago

confirmed working in vagrant 2.1.1, with two caveats:

jcshort commented 6 years ago

Just kidding, I misled myself with a stale version of the box downloaded with a local IAM user.