Can't send events as raw string only s JSON!

[boatmisser]

Problem

We do not want our log events to be JSON; we want the message to be the raw message! E.g. instead of {"message": "2020-09-22 10-23-41 INFO - Hello World"} we want 2020-09-22 10-23-41 INFO - Hello World ...

Steps to replicate

<source>
  @type tail
  @id in_tail_container_logs
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  read_from_head true
  <parse>
    @type json
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>
<filter kubernetes.**>
  @type kubernetes_metadata
</filter>
<filter kubernetes.**>
  @type record_transformer
  enable_ruby true
  <record>
    service /aws/eks/application-cluster/${record['kubernetes']['namespace_name']}/${record['kubernetes']['container_name']}
    pod ${record['kubernetes']['pod_name']}
  </record>
  remove_keys stream,docker,kubernetes
</filter>
<filter **>
  @type concat
  key log
  multiline_start_regexp /^[0-9]{4}-[0-9]{2}-[0-9]{2}|^\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b|^[0-9]{4}\/[0-9]{2}\/[0-9]{2}/
  separator ""
  flush_interval 5
</filter>
<filter kubernetes.**>
  @type parser
  key_name log
  reserve_data true
  remove_key_name_field true
  emit_invalid_record_to_error false
  <parse>
    @type none
  </parse>
</filter>
<match kubernetes.**>
  @type cloudwatch_logs
  format single_value
  region "#{ENV.fetch('REGION')}"
  log_group_name_key service
  log_stream_name_key pod
  remove_log_group_name_key true
  remove_log_stream_name_key true
  auto_create_stream true
  <buffer>
    flush_interval 5
    chunk_limit_size 2m
    queued_chunks_limit_size 32
    retry_forever true
  </buffer>
</match>

Expected Behavior or What you need to ask

If I replace the @type cloudwatch_logs match with an @type stdout; the output is formatted correctly I.e. as normal text not JSON. We want the CloudWatch event message to be the same raw message and not wrapped in a JSON object. E.g. instead of {"message": "2020-09-22 10-23-41 INFO - Hello World"} we want 2020-09-22 10-23-41 INFO - Hello World

Is this possible? Or does the plugin only ever send JSON regardless of the fluentd config? ...

Using Fluentd and CloudWatchLogs plugin versions

Kubernetes: 1.17 Fluentd: fluent/fluentd-kubernetes-daemonset:v1.7.3-debian-cloudwatch-1.0

[boatmisser]

Never mind, I worked it out. You need to set the message_keys.

[masterchop]

nooo please put a sample!!! i am on the same quest :dagger:

2021/07/27 23:10:33] [error] [config] cloudwatch_logs: unknown configuration property 'message_keys'. The following properties are allowed: region, log_group_name, log_stream_name, log_stream_prefix, log_key, extra_user_agent, log_format, role_arn, auto_create_group, log_retention_days, endpoint, sts_endpoint, metric_namespace, and metric_dimensions. [2021/07/27 23:10:33] [ help] try the command: /fluent-bit/bin/fluent-bit -o cloudwatch_logs -h