search

mitchellh / vagrant-aws

Use Vagrant to manage your EC2 and VPC instances.
MIT License
2.61k stars 574 forks source link

Exponential Backoff required to prevent RequestLimitExceeded error #483

Closed lawrenceong closed 6 years ago

lawrenceong commented 8 years ago

The vagrant-aws module (0.7.2) does not currently implement exponential backoff. This means there is a chance that we would have builds fail as a result of a RequestLimitExceeded error.

vagrant-aws module 0.7.2 uses fog 1.22 that uses the latest excon (0.51.0 as of this writing)

fog uses the excon module to talk to aws. It sets the idempotent flag when calling excon, that signifies that it should retry on error. With idempotent set, the excon module will retry 3 times when an error is encountered. It does not distinguish between the errors. The retry happens consecutively without any wait in between, then fail with the latest error (usually RequestLimitExceeded when build fails)

Sample stack trace from a build failure:

/root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/middlewares/expects.rb:6:in `response_call': RequestLimitExceeded => Request limit exceeded. (Fog::Compute::AWS::Error)
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/middlewares/response_parser.rb:8:in `response_call'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/connection.rb:389:in `response'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/connection.rb:253:in `request'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/middlewares/idempotent.rb:26:in `error_call'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/middlewares/base.rb:10:in `error_call'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/middlewares/base.rb:10:in `error_call'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/connection.rb:273:in `rescue in request'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/connection.rb:221:in `request'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/middlewares/idempotent.rb:26:in `error_call'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/middlewares/base.rb:10:in `error_call'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/middlewares/base.rb:10:in `error_call'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/connection.rb:273:in `rescue in request'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/connection.rb:221:in `request'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/middlewares/idempotent.rb:26:in `error_call'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/middlewares/base.rb:10:in `error_call'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/middlewares/base.rb:10:in `error_call'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/connection.rb:273:in `rescue in request'
    from /root/.vagrant.d/gems/gems/excon-0.51.0/lib/excon/connection.rb:221:in `request'
    from /root/.vagrant.d/gems/gems/fog-xml-0.1.2/lib/fog/xml/sax_parser_connection.rb:35:in `request'
    from /root/.vagrant.d/gems/gems/fog-xml-0.1.2/lib/fog/xml/connection.rb:7:in `request'
    from /root/.vagrant.d/gems/gems/fog-aws-0.10.0/lib/fog/aws/compute.rb:525:in `_request'
    from /root/.vagrant.d/gems/gems/fog-aws-0.10.0/lib/fog/aws/compute.rb:520:in `request'
    from /root/.vagrant.d/gems/gems/fog-aws-0.10.0/lib/fog/aws/requests/compute/describe_instances.rb:81:in `describe_instances'
    from /root/.vagrant.d/gems/gems/fog-aws-0.10.0/lib/fog/aws/models/compute/servers.rb:62:in `all'
    from /root/.vagrant.d/gems/gems/fog-aws-0.10.0/lib/fog/aws/models/compute/servers.rb:149:in `get'
    from /root/.vagrant.d/gems/gems/vagrant-aws-0.7.2/lib/vagrant-aws/action/read_state.rb:24:in `read_state'
    from /root/.vagrant.d/gems/gems/vagrant-aws-0.7.2/lib/vagrant-aws/action/read_state.rb:15:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/warden.rb:34:in `call'
    from /root/.vagrant.d/gems/gems/vagrant-aws-0.7.2/lib/vagrant-aws/action/connect_aws.rb:43:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/builder.rb:116:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/runner.rb:66:in `block in run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/util/busy.rb:19:in `busy'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/runner.rb:66:in `run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/machine.rb:225:in `action_raw'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/machine.rb:200:in `block in action'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/machine.rb:182:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/machine.rb:182:in `block in action'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/machine.rb:186:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/machine.rb:186:in `action'
    from /root/.vagrant.d/gems/gems/vagrant-aws-0.7.2/lib/vagrant-aws/provider.rb:32:in `state'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/machine.rb:504:in `state'
    from /root/.vagrant.d/gems/gems/vagrant-aws-0.7.2/lib/vagrant-aws/action/is_created.rb:12:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/builder.rb:116:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/runner.rb:66:in `block in run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/util/busy.rb:19:in `busy'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/runner.rb:66:in `run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/builtin/call.rb:43:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/builder.rb:116:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/runner.rb:66:in `block in run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/util/busy.rb:19:in `busy'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/runner.rb:66:in `run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/builtin/call.rb:53:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/warden.rb:34:in `call'
    from /root/.vagrant.d/gems/gems/vagrant-hostmanager-1.8.3/lib/vagrant-hostmanager/action/update_all.rb:27:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/builder.rb:116:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/runner.rb:66:in `block in run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/util/busy.rb:19:in `busy'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/action/runner.rb:66:in `run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/machine.rb:225:in `action_raw'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/machine.rb:200:in `block in action'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/environment.rb:561:in `lock'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/machine.rb:186:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/machine.rb:186:in `action'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/plugins/commands/destroy/command.rb:31:in `block in execute'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/plugin/v2/command.rb:235:in `block in with_target_vms'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/plugin/v2/command.rb:229:in `each'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/plugin/v2/command.rb:229:in `with_target_vms'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/plugins/commands/destroy/command.rb:30:in `execute'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/cli.rb:42:in `execute'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/lib/vagrant/environment.rb:302:in `cli'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.8.4/bin/vagrant:174:in `<main>'

About the RequestLimitExceeded error: http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html

lawrenceong commented 8 years ago

Related: #403

BlinkyStitt commented 7 years ago

@lawrenceong do you think we should be making this change in the excon package?

lawrenceong commented 7 years ago

@WyseNynja I would say that the implementation should either be done in fog's aws module or in the vagrant-aws plugin itself. excon is a connection library that is unrelated to aws.

VVMichaelSawyer commented 7 years ago

Pull request open for the fog-aws gem: fog/fog-aws#361