Not every log makes it to CloudWatch

[lonnix]

Problem

Not every log is making it to cloudwatch. Some of my logs are not making it to cloudwatch. When I have fluentd log to stdout instead of sent to CW I see everything I expect but when switching back to cloudwatch I start losing messages. This does not happen all the time. I have an example below that has consistently reproduced the issue but I have multiple apps writing out on my cluster and they all get all or most of their logs into CW. It just is a few logs here and there that are missing. I know that the issue is with the CW output plugin because its all there when logged to stdout, even with my example that reproduces the issue 100% of the time. It seems to be related to logs having a time key. I tried "time": "asdf" as well as "time":"2020-08-31T03:36:16" and neither of them showed up in CW

Steps to replicate

1. Run fluentd with this plugin as a daemonset on kubernetes (I run on EKS) with this output plugin:

   <label @NORMAL>
     <match ** myjson mynotjson>
       @type cloudwatch_logs
       @id out_cloudwatch_logs_containers
       region "#{ENV.fetch('REGION')}"
       log_level trace
       log_group_name_key fvr_log_group
       log_stream_name_key stream_name
       remove_log_stream_name_key true
       auto_create_stream true
       include_time_key false
       <buffer>
         flush_interval 1
         chunk_limit_size 1m
         queued_chunks_limit_size 32
         retry_forever true
       </buffer>
     </match>
   </label>

2. Add these 3 lines to a file being watched

   {"log":{"key":"lonnix3-testing-with-time-key"},"stream":"stdout","time":"2020-08-31T03:36:16"}
   {"log":{"key":"lonnix3-testing-with-badtime-key"},"stream":"stdout","badtime":"asdf"}
   {"log":{"key":"lonnix3-testing"},"stream":"stdout"}

3. Check CloudWatch to see if all 3 lines showed up

Expected Behavior or What you need to ask

I expect to see all 3 of those entries in Cloudwatch but I only see the bottom 2.

Using Fluentd and CloudWatchLogs plugin versions

kubernetes version: 1.15 fluentd version: fluent/fluentd-kubernetes-daemonset:v1.11-debian-cloudwatch-1

[lonnix]

I tried the solution mentioned on https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs/issues/201 and it did not help

[lonnix]

I have talked with an AWS Support representative and it is leaning more towards an issue with the plugin than with CloudWatch. All logs are written to stdout so the kubernetes logging engine handles and formats them all the same way. Is the plugin doing some parsing that would affect only certain logs?

[cosmo0920]

I tried the solution mentioned on #201 and it did not help

I'm not a psychic. Your shared configuration does not include source section. fluent-plugin-cloudwatch-logs just handle source plugin consumed events. When source plugins (such as in_tail) do not handle or exhausted, fluent-plugin-cloudwatch-logs cannot see or handle lost events.

[lonnix]

I switched to fluent-bit to use that cloudwatch plugin, so this issue can be closed

[cosmo0920]

Sure, I'm closing this issue.