Mesosphere filter not working

[davidsayers]

Problem

Hi! I am trying to use the https://github.com/joshughes/fluent-plugin-mesosphere-filter filter to include Mesos specific data in the output sent to elasticsearch.

When I don't include the elasticsearch plugin and only add the Mesosphere plugin it works with other plugins like https://github.com/SumoLogic/fluentd-output-sumologic

When I add the elasticsearch plugin the Mesosphere plugin doesn't work with it and Sumologic. When I add some debug logging to the Mesosphere plugin it also stops printing this out. It's like the Mesosphere plugin stops being called

I am using the docker fluentd log driver to send data to fluentd.

Steps to replicate

Install https://github.com/joshughes/fluent-plugin-mesosphere-filter and Elasticsearch plugin.

Config:

<match **>
  @type elasticsearch
  @log_level debug
  hosts https://ES_HOST:443
  ssl_verify true
  # logstash_format true
  # logstash_prefix fluentd
</match>

<filter **> 
  @type mesosphere_filter
  cache_size 1000
  cache_ttl 3600
  merge_json_log true
  cronos_task_regex (?<app>[a-z0-9]([-a-z0-9]*[a-z0-9]))-(?<date>[^-]+)-(?<time>[^-]+)-(?<task_type>[^-]+)-(?<run>[^-]+)-(?<epoc>[^-]+)
</filter>

Expected Behavior or What you need to ask

Mesosphere docker data should be included in the output sent to elasticsearch.

Using Fluentd and ES plugin versions

• Fluentd v0.12 or v0.14
• ES plugin 2.x.y or 1.x.y
• mesosphere-filter 0.2.0

[cosmo0920]

fluent-plugin-elasticsearch does not emit any record latter plugin pipeline. Could you rearrange plugin pipeline in your configuration?

<filter **> 
  @type mesosphere_filter
  cache_size 1000
  cache_ttl 3600
  merge_json_log true
  cronos_task_regex (?<app>[a-z0-9]([-a-z0-9]*[a-z0-9]))-(?<date>[^-]+)-(?<time>[^-]+)-(?<task_type>[^-]+)-(?<run>[^-]+)-(?<epoc>[^-]+)
</filter>

<match **>
  @type elasticsearch
  @log_level debug
  hosts https://ES_HOST:443
  ssl_verify true
  # logstash_format true
  # logstash_prefix fluentd
</match>

[davidsayers]

Thanks!!!! This has caused me to lose two days of my life. I was close to writing my own plugin and would have probably still hit this.

Strange decision to make the order of the config file important.