[BUG] Fails if encountering ThrottlingException from AWS for DescribeLogGroups request

[edruid]

Describe the bug If the DescribeLogGroups call encounters a ThrottlingException from AWS the logger will throw a CloudWatchLogsException 400 Bad Request {"__type":"ThrottlingException","message":"Rate exceeded"}

Expected behavior If exception is thrown, sleep 1 second and retry (at least once).

Please provide the steps to reproduce and if possible a minimal demo of the problem Run 6 parallel processes all trying to initialize the CloudWatch handler within the same second

Please tell about your environment:

• PHP Version: 7.2
• Operating system (distro): AWS Linux
• Application mode:
  • [X] Web app
  • [ ] CLI app
  • [ ] Daemon worker

trace: "/var/app/current/carrot/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:100", "/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:203", "/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:174", "/var/app/current/carrot/vendor/guzzlehttp/promises/src/RejectedPromise.php:40", "/var/app/current/carrot/vendor/guzzlehttp/promises/src/TaskQueue.php:47", "/var/app/current/carrot/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:104", "/var/app/current/carrot/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:131", "/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:246", "/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:223", "/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:267", "/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:225", "/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:267", "/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:225", "/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:62", "/var/app/current/carrot/vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58", "/var/app/current/carrot/vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86", "/var/app/current/carrot/vendor/maxbanton/cwh/src/Handler/CloudWatch.php:285", "/var/app/current/carrot/vendor/maxbanton/cwh/src/Handler/CloudWatch.php:248", "/var/app/current/carrot/vendor/maxbanton/cwh/src/Handler/CloudWatch.php:172", "/var/app/current/carrot/vendor/maxbanton/cwh/src/Handler/CloudWatch.php:150", "/var/app/current/carrot/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php:39", "/var/app/current/carrot/vendor/monolog/monolog/src/Monolog/Logger.php:344", "/var/app/current/carrot/vendor/monolog/monolog/src/Monolog/Logger.php:637", ...

[agheorghe-careplanner]

Hi.

We're experiencing this issue as well when a large number of parallel processes try to create CloudWatch logs at the same time. Would I be correct in thinking the following would work as a fix for it? From what I can see the throttling is handled in a similar way as described above through that.

diff --git a/src/Handler/CloudWatch.php b/src/Handler/CloudWatch.php
index 41c12f7..29c6492 100755
--- a/src/Handler/CloudWatch.php
+++ b/src/Handler/CloudWatch.php
@@ -299,6 +299,7 @@ class CloudWatch extends AbstractProcessingHandler

     private function initializeGroup(): void
     {
+        $this->checkThrottle();
         // fetch existing groups
         $existingGroups =
             $this

Thanks

[tuxoff]

I have the same problem. Also, regarding the fix, I think it should be not only checked throttling inside of the function, but it should also be additional retries as well.

[skrijeljhasib]

Hello,

I have the same issue on the latest version 2.0.2. Will there be a fix in the near future ?

[ihr-it-projekt]

Have a look to https://github.com/maxbanton/cwh/blob/master/src/Handler/CloudWatch.php#L130 . You do not need to check normally that the log group is created and create one. This is insufficient. So use here false instead.