search

atomita / fluent-plugin-aws-elasticsearch-service

This repository is extends of https://github.com/uken/fluent-plugin-elasticsearch/ which made connectable to Amazon Elassticsearch Service using Aws Signers V4. Please check the requirements to connect to Amazon Elassticsearch Service without using Aws Signers V4 for before starting use. (reference https://groups.google.com/forum/#!msg/fluentd/uW87VAOqxeE/cfhenicEBAAJ)
MIT License
121 stars 49 forks source link

emit transaction failed: error_class=NoMethodError error="undefined method `full_pack' #56

Open sagimann opened 5 years ago

sagimann commented 5 years ago

Problem

Suddenly, in recent days, td-agent stops sending events via the ES plugin, showing the following for each event in its log file:

2019-08-27 06:57:24 +0000 [warn]: #0 emit transaction failed: error_class=NoMethodError error="undefined method `full_pack' for #<MessagePack::Packer:0x007f3de657dbb0>" location="/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-1.7.0/lib/fluent/event.rb:63:in `to_msgpack_stream'" tag="**************"

From what I could gather, this did not occur on machines where td-agent --version showed "1.6.3", but did start recently in machines with td-agent showing version 1.7.0. Also, looking at the logs, it showed fluentd 1.7.0 instead of 1.6.3 and fluentd elastic search plugin changed from 3.5.4 to 3.5.5. We do not install the plugin directly, but rather via fluent-plugin-aws-elasticsearch-service.

Steps to replicate

in ubuntu:

curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent2.sh | sh
td-agent-gem install fluent-plugin-aws-elasticsearch-service

The td-agent conf file is:

<source>
  @type syslog
  tag syslog
</source>

<source> 
  @type tail
  format multiline
  format_firstline /^\d{4}-\d{2}-\d{2}/
  format1 /^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+)\s+\[(?<pid>.+)\]\s+(?<level>[^ ]+)\s+(?<message>.*)/
  time_format %Y-%m-%d %H:%M:%S,%N
  time_key timestamp
  path /var/log/xxxx/x*
  tag xxxx
</source>

<filter **>
  @type record_transformer
  enable_ruby
  remove_keys ["error"]
  <record>
    hostname "#{Socket.gethostname}"
    environment dev
  </record>
</filter>

<match fluent.**>
  @type null
</match>

<match **>
  @type "aws-elasticsearch-service"
  type_name "access_log"
  logstash_format true
  include_tag_key true
  tag_key "@log_name"
  reload_connections false
  flush_interval 1s
  <endpoint>
   url xxxxxxx
   region xxxxx
  </endpoint>
  <secondary>
    @type file
    path /var/log/td-agent/failed_records
  </secondary>
</match>

Expected Behavior or What you need to ask

error should not be issued, and the event should be emitted correctly to ES

Using Fluentd and ES plugin versions

boot log:

2019-08-27 08:56:08 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2019-08-27 08:56:08 +0000 [warn]: secondary type should be same with primary one primary="Fluent::TreasureDataLogOutput" secondary="Fluent::Plugin::FileOutput"
2019-08-27 08:56:08 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type syslog
    tag "syslog"
  </source>
  <source>
    @type tail
    format multiline
    format_firstline /^\d{4}-\d{2}-\d{2}/
    format1 /^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+)\s+\[(?<pid>.+)\]\s+(?<level>[^ ]+)\s+(?<message>.*)/
    time_format %Y-%m-%d %H:%M:%S,%N
    time_key timestamp
    path "/var/log/xxxx/x*"
    tag "xxx.xxx"
    <parse>
      time_key timestamp
      time_format %Y-%m-%d %H:%M:%S,%N
      format_firstline /^\d{4}-\d{2}-\d{2}/
      @type multiline
      format1 /^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+)\s+\[(?<pid>.+)\]\s+(?<level>[^ ]+)\s+(?<message>.*)/
    </parse>
  </source>
  <filter **>
    @type record_transformer
    enable_ruby 
    remove_keys ["error"]
    <record>
      hostname 1fffcfd5bb61
      environment staging
    </record>
  </filter>
  <match fluent.**>
    @type null
  </match>
  <match **>
    @type aws-elasticsearch-service
    type_name "access_log"
    logstash_format true
    include_tag_key true
    tag_key "@log_name"
    reload_connections false
    flush_interval 1s
    <endpoint>
      url "xxxxx"
      region "xxxxx"
    </endpoint>
    <secondary>
      @type "file"
      path "/var/log/td-agent/failed_records"
      <buffer time>
        path /var/log/td-agent/failed_records
      </buffer>
    </secondary>
    <buffer>
      flush_interval 1s
    </buffer>
  </match>
</ROOT>
2019-08-27 08:56:08 +0000 [info]: starting fluentd-1.7.0 pid=105 ruby="2.1.10"
2019-08-27 08:56:08 +0000 [info]: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.1.0'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '3.5.5'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-kafka' version '0.6.1'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-mongo' version '0.8.1'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.6'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-s3' version '0.8.5'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-td' version '0.10.29'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.3'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.7.1'
2019-08-27 08:56:08 +0000 [info]: gem 'fluentd' version '1.7.0'
2019-08-27 08:56:08 +0000 [info]: gem 'fluentd' version '0.12.40'
2019-08-27 08:56:08 +0000 [info]: adding match pattern="td.*.*" type="tdlog"
2019-08-27 08:56:08 +0000 [warn]: #0 secondary type should be same with primary one primary="Fluent::TreasureDataLogOutput" secondary="Fluent::Plugin::FileOutput"
2019-08-27 08:56:08 +0000 [info]: adding match pattern="debug.**" type="stdout"
2019-08-27 08:56:08 +0000 [info]: adding source type="forward"
2019-08-27 08:56:08 +0000 [info]: adding source type="http"
2019-08-27 08:56:08 +0000 [info]: adding source type="debug_agent"
2019-08-27 08:56:08 +0000 [info]: #0 starting fluentd worker pid=115 ppid=110 worker=0
2019-08-27 08:56:08 +0000 [info]: #0 listening dRuby uri="druby://127.0.0.1:24230" object="Fluent::Engine" worker=0
2019-08-27 08:56:08 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
2019-08-27 08:56:08 +0000 [info]: #0 fluentd worker is now running worker=0

LOCAL GEMS

actionmailer (4.2.8) actionpack (4.2.8) actionview (4.2.8) activejob (4.2.8) activemodel (4.2.8) activerecord (4.2.8) activesupport (4.2.8) addressable (2.5.2, 2.5.1) arel (6.0.4) aws-eventstream (1.0.3) aws-partitions (1.206.0) aws-sdk (2.10.45) aws-sdk-core (3.64.0, 2.10.45) aws-sdk-resources (2.10.45) aws-sigv4 (1.1.0, 1.0.2) bigdecimal (default: 1.2.4) bson (4.1.1) builder (3.2.3) bundler (1.14.5) celluloid (0.15.2) concurrent-ruby (1.1.5) cool.io (1.5.1) crass (1.0.2) diff-lcs (1.3) dig_rb (1.0.1) draper (1.4.0) elasticsearch (7.3.0) elasticsearch-api (7.3.0) elasticsearch-transport (7.3.0) erubis (2.7.0) excon (0.66.0) faraday (0.15.4) faraday_middleware-aws-sigv4 (0.2.4) fluent-logger (0.7.1) fluent-mixin-plaintextformatter (0.2.6) fluent-plugin-aws-elasticsearch-service (2.1.0) fluent-plugin-elasticsearch (3.5.5) fluent-plugin-kafka (0.6.1) fluent-plugin-mongo (0.8.1) fluent-plugin-rewrite-tag-filter (1.5.6) fluent-plugin-s3 (0.8.5) fluent-plugin-scribe (0.10.14) fluent-plugin-td (0.10.29) fluent-plugin-td-monitoring (0.2.3) fluent-plugin-webhdfs (0.7.1) fluentd (1.7.0, 0.12.40) fluentd-ui (0.4.4) font-awesome-rails (4.7.0.1) globalid (0.4.0) haml (4.0.7) haml-rails (0.5.3) hike (1.2.3) hirb (0.7.3) http_parser.rb (0.6.0) httpclient (2.8.2.4) i18n (0.8.1) io-console (default: 0.4.3) ipaddress (0.8.3) jbuilder (2.6.3) jmespath (1.3.1) jquery-rails (3.1.4) json (default: 1.8.1) kramdown (1.13.2) kramdown-haml (0.0.3) loofah (2.1.1, 2.0.3) ltsv (0.1.0) mail (2.6.4) mime-types (3.1) mime-types-data (3.2016.0521) mini_portile2 (2.3.0, 2.1.0) minitest (5.10.1, default: 4.7.5) mixlib-cli (1.7.0) mixlib-config (2.2.4) mixlib-log (1.7.1) mixlib-shellout (2.2.7) mongo (2.2.7) msgpack (1.1.0) multi_json (1.12.1) multipart-post (2.1.1) nokogiri (1.8.1) ohai (6.20.0) oj (2.18.1) parallel (1.8.0) psych (default: 2.0.5) public_suffix (3.0.0, 2.0.5) puma (3.8.2) rack (1.6.5) rack-test (0.6.3) rails (4.2.8) rails-deprecated_sanitizer (1.0.3) rails-dom-testing (1.0.8) rails-html-sanitizer (1.0.3) railties (4.2.8) rake (default: 10.1.0) rdoc (default: 4.1.0) request_store (1.3.2) ruby-kafka (0.4.2) ruby-progressbar (1.8.3) rubyzip (1.2.1, 1.1.7) sass (3.2.19) sass-rails (4.0.5) serverengine (2.1.1) settingslogic (2.0.9) sigdump (0.2.4) sprockets (2.12.4) sprockets-rails (2.3.3) string-scrub (0.0.5) strptime (0.2.3) sucker_punch (1.0.5) systemu (2.5.2) td (0.15.2) td-client (0.8.85) td-logger (0.3.27) test-unit (default: 2.1.10.0) thor (0.19.4) thread_safe (0.3.6) thrift (0.8.0) tilt (1.4.1) timers (1.1.0) tzinfo (2.0.0, 1.2.3) tzinfo-data (1.2017.2) uuidtools (2.1.5) webhdfs (0.8.0) yajl-ruby (1.3.0) zip-zip (0.3)

sagimann commented 5 years ago

One workaround I found in the meantime. Call this before installing the fluent-plugin-aws-elasticsearch-service gem: td-agent-gem install fluentd -v 1.6.3 .. now install fluent-plugin-aws-elasticsearch-service...

cosmo0920 commented 4 years ago

Contaminated usage for fluentd 1.7.0 and 0.12.40 is dangerous. Please uninstall Fluentd v0.12.40 when using Fluentd v1.7 or later.

cosmo0920 commented 4 years ago

And td-agent 2 is already EOL. We don't provide any issue triage for td-agent 2. Could you use td-agent 3 with this installation step?

$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh