Prefixed log stream(s) are not found

minh-nguyenquang commented 3 years ago

Problem

I try to pull logs from AWS Cloudwatch with log group prefix is /aws/lambda/prefix_string In every log group with prefix /aws/lambda/, there are many log streams with naming convention 2021/01/13[$LATEST]... So my case is I have to set both use_log_group_name_prefix and use_log_stream_name_prefix config parameters to true After deploy, I get this warning log

2021-01-08 07:12:24 +0000 [warn]: '2021' prefixed log stream(s) are not found

Steps to replicate

Use this config to replicate the issue

    <source>
      @type cloudwatch_logs
      tag lambda_events
      region ap-southeast-1
      aws_use_sts true
      aws_sts_role_arn "#{ENV['AWS_STS_ROLE_ARN']}"
      log_group_name /aws/lambda/prefix
      use_log_group_name_prefix true
      log_stream_name "20"
      use_log_stream_name_prefix true
      fetch_interval 180
      json_handler json
      include_metadata true
      <storage>
        @type local
        path /var/log/fluentd-buffers/fluentd-lambda.state
      </storage>
      <parse>
        @type none
      </parse>
    </source>

Expected Behavior or What you need to ask

I'm be able to collect log from all logs streams with both use_log_group_name_prefix and use_log_stream_name_prefix config parameters are set to true

Using Fluentd and CloudWatchLogs plugin versions

Kubernetes
Fluentd v1.12.0

minh-nguyenquang commented 3 years ago

I manage to solve the issue by updating code in this PR https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs/pull/218/files

PaulRudin commented 3 years ago

Funny coincidence - I came here to ask about this problem, and this issue was created a few hours ago.

Out of interest: is this a regression? Is it working in some earlier version?

minh-nguyenquang commented 3 years ago

@PaulRudin Config param log_group_name_prefix was added recently https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs/pull/215. This PR was merged on Dec 8, 2020.

So I guess the earlier versions doesn't work even the current latest release is v0.13.0

So when I start trying config param log_group_name_prefix with use_log_stream_name_prefix I get above issue.

fluent-plugins-nursery / fluent-plugin-cloudwatch-logs