First of all, thank you for the very useful plugin!
Problem
Using in_cloudwatch_logs, we are ingesting logs from a few different AWS services, most notably Lambda and AppSync:
# AppSync config
...
log_group_name /aws/appsync/apis/
use_log_group_name_prefix true
# Lambda config
...
log_group_name /aws/lambda/
use_log_group_name_prefix true
use_todays_log_stream true # Get only log streams prefixed with today's YYYY/MM/DD
However, both of these services create a log stream for each request, which results in having a lot streams with few logs lines in each (even with very short retention configured). With the current configuration, get_log_events API calls are made for each stream, even though no new log lines are expected for older streams. After a while, the get_log_events quota is reached and throttling comes into play.
To mitigate that, I've been thinking about a few different approaches:
Option to enable deletion of the log stream after reading it. Maybe through matching a regex from the last line to know when we're done. E.g until ^REPORT RequestId: .*$ is matched, in the case of Lambda logs and ^.* End Request$ for AppSync.
Option to enable a rolling start date. E.g get only log streams that are up to 30 minutes old and check only them for new logs.
I am happy to work towards a solution, if any of that makes sense
First of all, thank you for the very useful plugin!
Problem
Using
in_cloudwatch_logs, we are ingesting logs from a few different AWS services, most notably Lambda and AppSync:However, both of these services create a log stream for each request, which results in having a lot streams with few logs lines in each (even with very short retention configured). With the current configuration,
get_log_eventsAPI calls are made for each stream, even though no new log lines are expected for older streams. After a while, the get_log_events quota is reached and throttling comes into play.To mitigate that, I've been thinking about a few different approaches:
^REPORT RequestId: .*$is matched, in the case of Lambda logs and^.* End Request$for AppSync.I am happy to work towards a solution, if any of that makes sense