Handle RFC 5424 syslog format for CF platform logs

[bandesz]

The latest syslog-release project introduced a new Syslog log format which is based on RFC 5424 [1]. This format is automatically parsed by the Logstash filters defined in [2] therefore the grok filters will fail in platform.conf.

We check for syslog_sd_id first and make sure the common fields and tags are set.

[1] https://github.com/cloudfoundry/syslog-release/blob/e58e88281429f0bac1313994db447fce22b29106/jobs/syslog_forwarder/templates/rsyslog.conf.erb#L54 [2] https://github.com/cloudfoundry-community/logsearch-boshrelease/blob/9c0376bfd403cc63df058a30e6e27e4f0b41c889/src/logsearch-config/src/logstash-filters/snippets/syslog_standard.conf#L3

[bandesz]

Note: https://github.com/cloudfoundry-community/logsearch-for-cloudfoundry/pull/292 should be merged before this. I tested the two PRs together and all tests passed.

[bandesz]

I rebased the branch as https://github.com/cloudfoundry-community/logsearch-for-cloudfoundry/pull/292 was merged.

Also I'd like to get your opinion about the extra fields added by the syslog-release. Do we want to parse them as additional @source fields?

For example we could add the following:

mutate {
  add_field => {
    "[@source][az]" => "%{[syslog_sd_params][az]}"
    "[@source][director]" => "%{[syslog_sd_params][director]}"
    "[@source][id]" => "%{[syslog_sd_params][id]}"
  }
}

Also do we want to remove any of the syslog_sd_params fields to avoid duplication? I'm not sure we can do that as people might already depend on them in their setups.

[bandesz]

I added an extra commit to add the missing fields, let me know what you think

[hannayurkevich]

@bandesz

Thank you for your PR! I'm merging your changes.

Also regarding your question about adding / removing fields.

I think we should not delete existing fields, because, you are right, some people can rely on them. Also, if you think that the new fields can be useful, then yes, you can add them.

Thanks again! Hanna