search

quarkiverse / quarkus-logging-cloudwatch

Quarkus Amazon CloudWatch Logging extension
Apache License 2.0
13 stars 6 forks source link

Logs stop appearing in cloudwatch #629

Open mukshe01 opened 1 month ago

mukshe01 commented 1 month ago

Hi Team,

We are running a quarkus app with quarkus-logging-cloudwatch extention to push logs to cloudwatch. we are running the app with default quarkus-logging-cloudwatch extention configs,

https://github.com/quarkiverse/quarkus-logging-cloudwatch quarkus-logging-cloudwatch extention version: 5.4.0 our quarkus platform version: 2.13.0.Final

After some time the app stops pushing logs to cloudwatch, we see these errors in the app logs, would you let us how to fix this issue,

2024-06-03T08:43:51.133976517Z stdout F {"timestamp":"2024-06-03T08:43:51.133Z","sequence":1773,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.headers","level":" DEBUG","message":"http-outgoing-0 << HTTP/1.1 400 Bad Request","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","pro cessId":1} 2024-06-03T08:43:51.133988847Z stdout F {"timestamp":"2024-06-03T08:43:51.133Z","sequence":1774,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.headers","level":" DEBUG","message":"http-outgoing-0 << x-amzn-RequestId: 125decf5-51f6-46cd-be57-af4919a3abff","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","proce ssName":"quarkus-run.jar","processId":1} 2024-06-03T08:43:51.133991568Z stdout F {"timestamp":"2024-06-03T08:43:51.133Z","sequence":1775,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.headers","level":" DEBUG","message":"http-outgoing-0 << Content-Type: application/x-amz-json-1.1","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quark us-run.jar","processId":1} 2024-06-03T08:43:51.133998188Z stdout F {"timestamp":"2024-06-03T08:43:51.133Z","sequence":1776,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.headers","level":" DEBUG","message":"http-outgoing-0 << Content-Length: 107","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","processI d":1} 2024-06-03T08:43:51.134000398Z stdout F {"timestamp":"2024-06-03T08:43:51.133Z","sequence":1777,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.headers","level":" DEBUG","message":"http-outgoing-0 << Date: Mon, 03 Jun 2024 08:43:50 GMT","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quarkus-ru n.jar","processId":1} 2024-06-03T08:43:51.134064129Z stdout F {"timestamp":"2024-06-03T08:43:51.133Z","sequence":1778,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.headers","level":" DEBUG","message":"http-outgoing-0 << Connection: close","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","processId" :1} 2024-06-03T08:43:51.1345481Z stdout F {"timestamp":"2024-06-03T08:43:51.134Z","sequence":1779,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"software.amazon.awssdk.requestId","level":"DEBUG", "message":"Received failed response: 400, Request ID: 125decf5-51f6-46cd-be57-af4919a3abff, Extended Request ID: not available","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"r est-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","processId":1} 2024-06-03T08:43:51.134570519Z stdout F {"timestamp":"2024-06-03T08:43:51.134Z","sequence":1780,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"software.amazon.awssdk.request","level":"DEBUG", "message":"Received failed response: 400, Request ID: 125decf5-51f6-46cd-be57-af4919a3abff, Extended Request ID: not available","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"r est-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","processId":1} 2024-06-03T08:43:51.135589531Z stdout F {"timestamp":"2024-06-03T08:43:51.135Z","sequence":1781,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.wire","level":"DEB UG","message":"http-outgoing-0 << \"{\"__type\":\"InvalidParameterException\",\"message\":\"Upload too large: 1064089 bytes exceeds limit of 1048576\"}\"","threadName":"pool-12-thread-1","threadId":32,"mdc ":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","processId":1} 2024-06-03T08:43:51.135661173Z stdout F {"timestamp":"2024-06-03T08:43:51.135Z","sequence":1782,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.impl.conn.DefaultM anagedHttpClientConnection","level":"DEBUG","message":"http-outgoing-0: Close connection","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processN ame":"quarkus-run.jar","processId":1} 2024-06-03T08:43:51.135938688Z stdout F {"timestamp":"2024-06-03T08:43:51.135Z","sequence":1783,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"software.amazon.awssdk.http.apache.internal.net. SdkSslSocket","level":"DEBUG","message":"closing logs.us-east-1.amazonaws.com/3.236.94.242:443","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","pr ocessName":"quarkus-run.jar","processId":1} 2024-06-03T08:43:51.136365877Z stdout F {"timestamp":"2024-06-03T08:43:51.136Z","sequence":1784,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.impl.execchain.Mai nClientExec","level":"DEBUG","message":"Connection discarded","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","proc essId":1}

i have attached entire logfile.

Regards Shekhar

app_logfile.gz

gwenneg commented 1 month ago

Hey @mukshe01. There's something interesting in the errors you posted:

Upload too large: 1064089 bytes exceeds limit of 1048576

The extension sends a batch of log entries to CloudWatch. If the log entries are big enough, the size of a batch sometimes exceeds the maximum limit allowed by CloudWatch. Setting quarkus.log.cloudwatch.batch-size to a value lower than 10000 (default value) in your application.properties file might help fix your issue.

mukshe01 commented 4 weeks ago

Hi @gwenneg ,

Thank you for your response, we have tried updating batch size from 10000 to 5000 then to 1000, we still see the issue.

Below are the environment variables we are passing to our app, are these environment variables look correct?, 

QUARKUS_LOG_CLOUDWATCH_LOG_GROUP:                                /aws/containerinsights/one-source-qa-n5p1P1d1/application-new
      QUARKUS_LOG_CLOUDWATCH_LOG_STREAM_NAME:                          participant1-qa.rest-api-cw
      QUARKUS_LOG_CLOUDWATCH_BATCH_SIZE:                               1000
      QUARKUS_LOG_CLOUDWATCH_MAX_QUEUE_SIZE:                           50000000
      QUARKUS_LOG_CLOUDWATCH_BATCH_PERIOD:                             2s

we are still seeing the error 

","message":"Received failed response: 400, Request ID: 945f0bd9-0fbe-419d-8892-f2e549d93a9b, Extended Request ID: not available","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":
"rest-api-qa-5c5cc46dc-qlhbj","processName":"quarkus-run.jar","processId":1}
2024-06-04T13:36:45.503614412Z stdout F {"timestamp":"2024-06-04T13:36:45.503Z","sequence":1708,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"software.amazon.awssdk.request","level":"DEBUG",
"message":"Received failed response: 400, Request ID: 945f0bd9-0fbe-419d-8892-f2e549d93a9b, Extended Request ID: not available","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"r
est-api-qa-5c5cc46dc-qlhbj","processName":"quarkus-run.jar","processId":1}
2024-06-04T13:36:45.504961956Z stdout F {"timestamp":"2024-06-04T13:36:45.504Z","sequence":1709,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.wire","level":"DEB
UG","message":"http-outgoing-0 << \"{\"__type\":\"InvalidParameterException\",\"message\":\"Upload too large: 1063020 bytes exceeds limit of 1048576\"}\"","threadName":"pool-12-thread-1","threadId":32,"mdc
":{},"ndc":"","hostName":"rest-api-qa-5c5cc46dc-qlhbj","processName":"quarkus-run.jar","processId":1}
2024-06-04T13:36:45.505110638Z stdout F {"timestamp":"2024-06-04T13:36:45.505Z","sequence":1710,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.impl.conn.DefaultM
anagedHttpClientConnection","level":"DEBUG","message":"http-outgoing-0: Close connection","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-5c5cc46dc-qlhbj","processNa
me":"quarkus-run.jar","processId":1}
2024-06-04T13:36:45.505404201Z stdout F {"timestamp":"2024-06-04T13:36:45.505Z","sequence":1711,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"software.amazon.awssdk.http.apache.internal.net.
SdkSslSocket","level":"DEBUG","message":"closing logs.us-east-1.amazonaws.com/3.236.94.234:443","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-5c5cc46dc-qlhbj","pro
cessName":"quarkus-run.jar","processId":1}
2024-06-04T13:36:45.505830914Z stdout F {"timestamp":"2024-06-04T13:36:45.505Z","sequence":1712,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.impl.execchain.Mai
nClientExec","level":"DEBUG","message":"Connection discarded","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-5c5cc46dc-qlhbj","processName":"quarkus-run.jar","proce
ssId":1}
2024-06-04T13:36:45.505886748Z stdout F {"timestamp":"2024-06-04T13:36:45.505Z","sequence":1713,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.impl.conn.PoolingH
ttpClientConnectionManager","level":"DEBUG","message":"Connection released: [id: 0][route: {s}->https://logs.us-east-1.amazonaws.com:443][total available: 0; route allocated: 0 of 50; total allocated: 0 of
 50]","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-5c5cc46dc-qlhbj","processName":"quarkus-run.jar","processId":1}

I have uploaded entire log, would you suggest how to fix this.

cloudwatch_extention_logs_20240604.log.gz

Regards Shekhar