RDS logs as input doesn't work

[polycaster]

Problem

I want to use this plugin to input RDS logs from Cloudwatch which are not output in a JSON structure, however the plugin seems to only support JSON messages and i'm wondering why. I might be missing something here. ...

Steps to replicate

1. Export RDS logs to clodwatch
2. Setup input

   <source>
   @type cloudwatch_logs
   tag cloudwatch
   log_group_name /aws/rds/instance/instancesrds-skyfallperf-postings/postgresql
   log_stream_name instancesrds-skyfallperf-postings
   fetch_interval 20
   use_log_stream_name_prefix true
   json_handler json
   state_file /etc/fluent/rds_state.pos
   </source>

Now for every parsed log there will be the following:

2020-02-19 11:13:04 +0000 [error]: Invalid JSON encountered while parsing event.message
2020-02-19 11:13:04 +0000 [warn]: dump an error event: error_class=Yajl::ParseError error="parse error: unallowed token at this point in JSON text\n                          2020-02-18 18:50:53 UTC:10.137.147.148(47612\n                     (right here) ------^\n" location="/usr/lib/ruby/gems/2.5.0/gems/yajl-ruby-1.4.1/lib/yajl.rb:44:in `parse'" tag="cloudwatch" time=1582051853 record={:message=>"2020-02-18 18:50:53 UTC:10.137.147.148(47612):perf_skyfall_integrations@perf_skyfall_integrations:[5517]:STATEMENT:  SELECT pg_database.datname, pg_database_size(pg_database.datname) as size FROM pg_database"}

Expected Behavior or What you need to ask

I would expect messages to be parsed regardless of format within the "message" parameter coming from the cloudwatch event. Is there a specific reason why these are mandatory to be valid JSON? ...

Using Fluentd and CloudWatchLogs plugin versions

• Alpine - v1.7.4-1.0
• Kubernetes
• Fluentd 1.0

  / $ fluentd --version
  fluentd 1.9.2

• Dependent gem versions

  && fluent-gem install fluent-plugin-prometheus -v 1.7.0 \
  && fluent-gem install fluent-plugin-elasticsearch -v 3.8.0 \
  && fluent-gem install fluent-plugin-multi-format-parser -v 1.0.0 \
  && fluent-gem install fluent-plugin-cloudwatch-logs -v 0.8.0 \

[polycaster]

Got around this by adding an undocumented parameter to the input source.

It then became:

<source>
    @type cloudwatch_logs
    tag cloudwatch
    log_group_name /aws/rds/instance/instancesrds-skyfallperf-postings/postgresql
    log_stream_name instancesrds-skyfallperf-postings
    fetch_interval 20
    use_log_stream_name_prefix true
    json_handler json
        format /(?<message>.+)/
    state_file /etc/fluent/rds_state.pos
</source>

Noticed this after reading through the code and it can be found on the Configure STS credentials section as an example.