search

fluent-plugins-nursery / fluent-plugin-cloudwatch-logs

CloudWatch Logs Plugin for Fluentd
MIT License
201 stars 141 forks source link

The in_cloudwatch_log doesn't pick up the newly created logstream after fluentd initialized #242

Open rendyfebry opened 2 years ago

rendyfebry commented 2 years ago

Problem

When I use the use_log_group_name_prefix and use_log_stream_name_prefix, this plugin works as expected and is able to fetch logs from all satisfied log_group and log_stream. But after the if there's a new log_stream created during fluentd run time, the plugin won't detect this new log_stream, therefor the logs not being ingested.

...

Steps to replicate

  1. Run Fluentd with this configuration

    <source>
@type cloudwatch_logs

log_group_name /group-a
use_log_group_name_prefix true
log_stream_name app
use_log_stream_name_prefix true
fetch_interval 5
</source>

  2. During the runtime, create a new log_stream that still satisfy the above configuration

  3. You'll notice the plugin won't detect this new stream

Expected Behavior or What you need to ask

It should detect new log_stream as long as it satisfy the configuration.

...

Using Fluentd and CloudWatchLogs plugin versions

2022-01-21 11:09:44 +0000 [info]: starting fluentd-1.14.4 pid=7 ruby="2.6.8"
2022-01-21 11:09:44 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "--plugin", "/fluentd/plugins", "--under-supervisor"]