log groups streams are not been collected by fluentd

[haiakab]

Problem

Hi, I'm using Fluentd agent to scrape streams from CloudWatch log groups. The issue I'm facing is that data is received only for specific log groups (those with names that start with /aws/lambda). These kinds of log groups have a specific template for log stream names, such as: 2024/07/04/[$LATEST]80cd86893da74d0783015c4173ce6eda

**We have permissions to fetch all log groups other than /aws/lambda using ROLE_ARN from AWS CLI

Attached is the Fluentd configuration:

system: log_level: error log_file: /var/log/fluent/fluentd.log

config:

• !include config.d/*.yaml
• match: $type: forward server: name: HOST host: HOST port: 3000
• source: $type: cloudwatch_logs tag: cloudwatch.input1 log_group_name: LOG_GROUP_NAME use_todays_log_stream: true include_metadata: true region: REGION aws_use_sts: true aws_sts_role_arn: ROLE_ARN parse: $type: none

** no errors in fluentd logs

Using Fluentd and CloudWatchLogs plugin versions

fluentd 1.16.3, within docker OS: NAME="Amazon Linux" VERSION="2"

• Dependent gem versions
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock:

aws-eventstream (1.3.0) aws-partitions (1.907.0, 1.867.0, 1.866.0) aws-sdk-cloudwatch (1.84.0) aws-sdk-cloudwatchlogs (1.76.0, 1.71.0) aws-sdk-core (3.191.6, 3.190.0) aws-sdk-kms (1.74.0) aws-sdk-s3 (1.141.0) aws-sigv4 (1.8.0)

[haiakab]

currently I'm getting this error in fluentd logs:

2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:293:in block in get_events' 2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:331:inthrottling_handler' 2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:279:in get_events' 2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:209:inblock (2 levels) in run' 2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:207:in each' 2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:207:inblock in run' 2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:201:in each' 2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:201:inrun' 2024-07-07 13:49:20 +0000 [error]: #0 /opt/fluent/lib/ruby/gems/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2024-07-07 13:49:20 +0000 [error]: Worker 0 exited unexpectedly with status 1

** I'm not sure if it's affecting the case I described, as I'm not receiving it consistently