search

dtan4 / terraforming

Export existing AWS resources to Terraform style (tf, tfstate) / No longer actively maintained
http://terraforming.dtan4.net/
MIT License
4.29k stars 659 forks source link

Feature request: Implement retry with backoff #388

Open seren opened 6 years ago

seren commented 6 years ago

Hi, since terraforming can make many API requests in a short time, I've found that sometimes it will error out due to AWS throttling. Would it be possible to add retries with exponential backoff? This is the error I'm seeing:

/usr/local/bundle/gems/aws-sdk-core-3.6.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': Rate exceeded (Aws::Route53::Errors::Throttling)
    from /usr/local/bundle/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
    from /usr/local/bundle/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
    from /usr/local/bundle/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
    from /usr/local/bundle/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
    from /usr/local/bundle/gems/aws-sdk-core-3.6.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'
    from /usr/local/bundle/gems/aws-sdk-core-3.6.0/lib/seahorse/client/request.rb:70:in `send_request'
    from /usr/local/bundle/gems/aws-sdk-route53-1.3.0/lib/aws-sdk-route53/client.rb:3410:in `list_tags_for_resource'
    from /usr/local/bundle/gems/terraforming-0.16.0/lib/terraforming/resource/route53_zone.rb:56:in `tags_of'
    from (erb):12:in `block in apply_template'
    from (erb):1:in `each'
    from (erb):1:in `apply_template'
    from /usr/local/lib/ruby/2.4.0/erb.rb:896:in `eval'
    from /usr/local/lib/ruby/2.4.0/erb.rb:896:in `result'
    from /usr/local/bundle/gems/terraforming-0.16.0/lib/terraforming/util.rb:4:in `apply_template'
    from /usr/local/bundle/gems/terraforming-0.16.0/lib/terraforming/resource/route53_zone.rb:19:in `tf'
    from /usr/local/bundle/gems/terraforming-0.16.0/lib/terraforming/resource/route53_zone.rb:7:in `tf'
    from /usr/local/bundle/gems/terraforming-0.16.0/lib/terraforming/cli.rb:258:in `tf'
    from /usr/local/bundle/gems/terraforming-0.16.0/lib/terraforming/cli.rb:245:in `execute'
    from /usr/local/bundle/gems/terraforming-0.16.0/lib/terraforming/cli.rb:165:in `r53z'
    from /usr/local/bundle/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
    from /usr/local/bundle/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
    from /usr/local/bundle/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
    from /usr/local/bundle/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
    from /usr/local/bundle/gems/terraforming-0.16.0/bin/terraforming:5:in `<top (required)>'
    from /usr/local/bundle/bin/terraforming:21:in `load'
    from /usr/local/bundle/bin/terraforming:21:in `<main>'
apenney commented 6 years ago

Has anyone come across a way around this? I can't use terraforming because I constantly get rate exceeded. I was thinking of trying to hack a bunch of ugly sleep statements in to make it finish at least. @dtan4 have you seen this on larger accounts? I can't get route53, I can't get ELBs, etc, all give me "rate exceeded".