GoogleCloudPlatform / fluent-plugin-detect-exceptions

A fluentd plugin that scans line-oriented log streams and combines exceptions stacks into a single log entry.
Apache License 2.0
192 stars 69 forks source link

No logs parsed #63

Closed laghao closed 4 years ago

laghao commented 4 years ago

Hi, I'm using fluentd-plugin-detect-exception to parse and ship logs to AWS-Cloudwatch using a patched version of "fluentd-kubernetes-daemonset" Docker Hub tag: adessoasaservice / fluentd-kubernetes-daemonset:v1.7.3-debian-cloudwatch-exception-1.0

having following gems:

gem "fluentd", "1.7.3"
gem "oj", "3.8.1"
gem "fluent-plugin-multi-format-parser", "~> 1.0.0"
gem "fluent-plugin-concat", "~> 2.3.0"
gem "fluent-plugin-grok-parser", "~> 2.5.0"
gem "fluent-plugin-prometheus", "~> 1.5.0"
gem 'fluent-plugin-json-in-json-2', ">= 1.0.2"
gem "fluent-plugin-record-modifier", "~> 2.0.0"
gem "fluent-plugin-rewrite-tag-filter", "~> 2.2.0"
gem "aws-sdk-cloudwatchlogs", "~> 1.0"
gem "fluent-plugin-detect-exceptions", "~> 0.0.12"
gem "fluent-plugin-cloudwatch-logs", "~> 0.7.4"
gem "fluent-plugin-kubernetes_metadata_filter", "~> 2.3.0"
gem "ffi"
gem "fluent-plugin-systemd", "~> 1.0.1"

and the configuration of fluentd as follow:

data:
  fluent.conf: |
    <match fluent.**>
      @type null
    </match>

    <source>
      @type tail
      enable_stat_watcher false
      path /var/log/containers/*.log
      exclude_path ["/var/log/containers/kube-*.log", "/var/log/containers/calico-*.log", "/var/log/containers/heapster-*.log", "/var/log/containers/etcd-*.log", "/var/log/containers/kubernetes-*.log", "/var/log/containers/fluentd-*.log", "/var/log/containers/monitoring-*.log"]
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      format json
      keep_time_key true
    </source>

    <match kubernetes.**>
      @type detect_exceptions
      message log
      remove_tag_prefix kubernetes
      languages all
      multiline_flush_interval 0.5
    </match>

    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>

    <match **>
      @type cloudwatch_logs
      log_group_name "#{ENV['LOG_GROUP_NAME']}"
      auto_create_stream true
      use_tag_as_stream true
    </match>

the logs of fluentd shows that the plugin is used, no errors but there's no logs parsed and shipped to Cloudwatch:

2019-10-10 07:13:30 +0000 [info]: starting fluentd-1.7.3 pid=6 ruby="2.6.4"
2019-10-10 07:13:30 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.6.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "--under-supervisor"]
2019-10-10 07:13:31 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.7.4'
2019-10-10 07:13:31 +0000 [info]: gem 'fluent-plugin-concat' version '2.3.0'
2019-10-10 07:13:31 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.12'
2019-10-10 07:13:31 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.5.1'
2019-10-10 07:13:31 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2019-10-10 07:13:31 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.3.0'
2019-10-10 07:13:31 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2019-10-10 07:13:31 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.5.0'
2019-10-10 07:13:31 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.0.1'
2019-10-10 07:13:31 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0'
2019-10-10 07:13:31 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.2'
2019-10-10 07:13:31 +0000 [info]: gem 'fluentd' version '1.7.3'
2019-10-10 07:13:31 +0000 [info]: adding match pattern="fluent.**" type="null"
2019-10-10 07:13:31 +0000 [info]: adding match pattern="kubernetes.**" type="detect_exceptions"
2019-10-10 07:13:31 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2019-10-10 07:13:31 +0000 [info]: adding match pattern="**" type="cloudwatch_logs"
2019-10-10 07:13:31 +0000 [info]: adding source type="tail"
2019-10-10 07:13:31 +0000 [info]: adding source type="tail"
2019-10-10 07:13:31 +0000 [info]: adding source type="tail"
2019-10-10 07:13:31 +0000 [info]: adding source type="tail"
2019-10-10 07:13:31 +0000 [info]: adding source type="tail"
2019-10-10 07:13:31 +0000 [info]: adding source type="tail"
2019-10-10 07:13:31 +0000 [info]: adding source type="tail"
jkohen commented 4 years ago

Thanks for the report. This plugin is intended to be used with Stackdriver. We don't know enough about CloudWatch to help.