Potential bug in handling of rate-limit errors

[cyclotron3k]

I've been playing with this a bit and it generally works very well, but I get intermittent errors, and they seem to occur when I'm pushing it quite hard and (presumably) causing the AWS API gateway to throw rate limit errors.

I see errors about unexpected EOF.

Is this something you have seen too? Are you accepting PRs?

[saluzafa]

Hello @cyclotron3k ! I didn't experienced that, but I'm not using it in the context of a high volume of requests. Feel free to open a pull request, it will be received with pleasure 😄 .

Cheers!

[cyclotron3k]

I've been digging into this a bit recently. My notes so far:

• I can reliably recreate the issue by sending hundreds of "describe stack" requests to the cloud formation API.

• Rate limit exceptions are being thrown and handled correctly, but eventually I see the following

  4m error: Async::Task [oid=0x4ff24] [ec=0x45f10] [pid=7] [2021-11-22 01:25:04 +0000] | EOFError: Could not read line! | → /usr/local/bundle/gems/async-http-0.56.5/lib/async/http/protocol/http1/connection.rb:55 in read_line' | /usr/local/bundle/gems/protocol-http1-0.14.2/lib/protocol/http1/connection.rb:201 inread_response' | /usr/local/bundle/gems/async-http-0.56.5/lib/async/http/protocol/http1/response.rb:31 in read' | /usr/local/bundle/gems/async-http-0.56.5/lib/async/http/protocol/http1/client.rb:78 incall' | /usr/local/bundle/gems/protocol-http-0.22.5/lib/protocol/http/request.rb:53 in call' | /usr/local/bundle/gems/async-http-0.56.5/lib/async/http/client.rb:143 inmake_response' | /usr/local/bundle/gems/async-http-0.56.5/lib/async/http/client.rb:106 in call' | /usr/local/bundle/gems/async-aws-1.0.2/lib/async/aws/http_handler.rb:41 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/plugins/content_length.rb:24 in call' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/plugins/request_callback.rb:85 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/xml/error_handler.rb:10 in call' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/signature_v4.rb:58 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/transfer_encoding.rb:26 in call' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:349 in call' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:383 inretry_request' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:371 in retry_if_possible' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:360 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:383 in retry_request' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:371 inretry_if_possible' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:360 in call' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:383 inretry_request' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:371 in retry_if_possible' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:360 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:383 in retry_request' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:371 inretry_if_possible' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/retry_errors.rb:360 in call' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/http_checksum.rb:18 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30 in call' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/query/handler.rb:30 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/user_agent.rb:13 in call' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/endpoint_discovery.rb:80 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/plugins/endpoint.rb:47 in call' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/param_validator.rb:26 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/logging.rb:41 in call' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/plugins/raise_response_errors.rb:16 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22 in call' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/param_converter.rb:26 in call' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/plugins/request_callback.rb:71 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/aws-sdk-core/plugins/response_paging.rb:12 in call' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/plugins/response_target.rb:24 incall' | /usr/local/bundle/gems/aws-sdk-core-3.122.1/lib/seahorse/client/request.rb:72 in send_request' | /usr/local/bundle/gems/aws-sdk-cloudformation-1.61.0/lib/aws-sdk-cloudformation/client.rb:3051 indescribe_stacks' | stress_test.rb:106 in _get_stack_parameters' | stress_test.rb:62 inget_stack_parameters' | stress_test.rb:42 in block (4 levels) in run' | /usr/local/bundle/gems/async-1.30.1/lib/async/task.rb:260 inblock in make_fiber'

[cyclotron3k]

I've noticed that the issue usually happens 4-5 minutes into a stress test. It doesn't seem to be correlated with how many connections I'm allowing, and what the backoff parameters are - which is unusual. This could be a red herring though.