search

iconara / bigshift

A tool for moving tables from Redshift to BigQuery
BSD 3-Clause "New" or "Revised" License
65 stars 10 forks source link

undefined method `instance_profile_credentials_retries' for #<Aws::Credentials access_key_id="<some-acces-key-id"> (NoMethodError) #14

Closed irvifa closed 5 years ago

irvifa commented 7 years ago

Hey, I got an error when using aws-sdk-core-3.6.0

${SOME_PATH}/.rvm/gems/ruby-2.4.1/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/credential_provider_chain.rb:27:in `providers': undefined method `instance_profile_credentials_retries' for #<Aws::Credentials access_key_id="<some-access-key>"> (NoMethodError)
Did you mean?  instance_variable_defined?
    from ${SOME_PATH}/.rvm/gems/ruby-2.4.1/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/credential_provider_chain.rb:11:in `resolve'
    from ${SOME_PATH}/.rvm/gems/ruby-2.4.1/gems/bigshift-0.3.2/lib/bigshift/cli.rb:259:in `aws_credentials'
    from ${SOME_PATH}/.rvm/gems/ruby-2.4.1/gems/bigshift-0.3.2/lib/bigshift/cli.rb:174:in `redshift_unloader'
    from ${SOME_PATH}/.rvm/gems/ruby-2.4.1/gems/bigshift-0.3.2/lib/bigshift/cli.rb:51:in `unload'
    from ${SOME_PATH}/.rvm/gems/ruby-2.4.1/gems/bigshift-0.3.2/lib/bigshift/cli.rb:29:in `run'
    from ${SOME_PATH}/.rvm/gems/ruby-2.4.1/gems/bigshift-0.3.2/bin/bigshift:6:in `<top (required)>'
    from ${SOME_PATH}/.rvm/gems/ruby-2.4.1/bin/bigshift:22:in `load'
    from ${SOME_PATH}/.rvm/gems/ruby-2.4.1/bin/bigshift:22:in `<main>'
    from ${SOME_PATH}/.rvm/gems/ruby-2.4.1/bin/ruby_executable_hooks:15:in `eval'
    from ${SOME_PATH}/.rvm/gems/ruby-2.4.1/bin/ruby_executable_hooks:15:in `<main>'
iconara commented 7 years ago

Seems like the AWS SDK is not backwards compatible. Either you need to downgrade, or find a workaround.

irvifa commented 7 years ago

@iconara I see, so what version of aws-sdk-core should I install for this library?

iconara commented 7 years ago

Anything before v3

michael-erasmus commented 6 years ago

I just run into this issue too. @irvifa did you manage to get around this by installing a specific version?

gh-mlfowler commented 5 years ago

@michael-erasmus I've been able to use this with aws-sdk 3.0.1 by specifying the environment variables AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY and not using --aws-credentials.

I was initially using a role that also has an AWS_SESSION_TOKEN which gets you a different error (with redactions): 

PG::InternalError: ERROR:  S3ServiceException:The AWS Access Key Id you provided does not exist in our records.,Status 403,Error InvalidAccessKeyId,Rid ###############,ExtRid ##########################,CanRetry 1
DETAIL:  
  -----------------------------------------------
  error:  S3ServiceException:The AWS Access Key Id you provided does not exist in our records.,Status 403,Error InvalidAccessKeyId,Rid ###############,,ExtRid ##########################,,CanRetry 1
  code:      8001
  context:   Listing bucket=#### prefix=###/###/###/###
  query:     0
  location:  s3_utility.cpp:720
  process:   padbmaster [pid=28309]
  -----------------------------------------------

  ~/bigshift/lib/bigshift/redshift_unloader.rb:24:in `async_exec'
  ~/bigshift/lib/bigshift/redshift_unloader.rb:24:in `unload_to'
  ~/bigshift/lib/bigshift/cli.rb:51:in `unload'
  ~/bigshift/lib/bigshift/cli.rb:29:in `run'
  ~/bigshift/bin/bigshift:6:in `<top (required)>'
  ~/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bin/bigshift:23:in `load'
  ~/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bin/bigshift:23:in `<top (required)>'

It's possible to update the code to pass the token through however it then fails in GCP with the same error which isn't so correctable. In the end I created a dedicated user with only the minimum IAM permissions.