fluent-plugins-nursery / fluent-plugin-systemd

This is a fluentd input plugin. It reads logs from the systemd journal.
Apache License 2.0
153 stars 43 forks source link

Parser type filter not working with systemd logs #106

Open vipulbehalpande08 opened 1 year ago

vipulbehalpande08 commented 1 year ago

Hi Team,

I have an inline JSON string in message field of systemd logs that I want to parse but when I add filter @type parser it stops capturing the logs in cloudwatch. It works with other log sources. Also, there are no error logs in td-agent.log

{
"syslog_identifier": "vault-audit",
"message": "{\"field_1\": \"value1\", \"field_2\": \"value_2\"}"
}

I am trying to parse above message in JSON format like below

{
   "syslog_identifier":"vault-audit",
   "message":{
      "field_1":"value1",
      "field_2":"value_2"
   }
}

Config:

<source>
  @type systemd
  tag journal
  path /var/log/journal
  read_from_head true

  <storage>
    @type local
    path /var/log/td-agent/fluentd-journald-cursor.json
  </storage>

  <entry>
    fields_strip_underscores true
    fields_lowercase true
  </entry>
</source>

<filter vault-audit>
  @type parser

  key_name message
  reserve_data true
  hash_value_field app_message
  emit_invalid_record_to_error false
  <parse>
    @type json
  </parse>
</filter>

<match journal>
  @type rewrite_tag_filter

  <rule>
    key syslog_identifier
    pattern /^vault-audit$/
    tag vault-audit
  </rule>
 </match>

<match vault-audit>
  @type cloudwatch_logs
  region {{region}}
  log_group_name /ec2/vault/audit
  log_stream_name {{instance_id}}
  auto_create_stream true
  retention_in_days 7
</match>