Unable to start fluentd using this plugin w/ prefer_oj_serializer enabled

[renegaderyu]

Problem

I was hoping to test/enable the oj serializer for better performance; however fluentd refuses to start when I set prefer_oj_serializer true using this plugin. This does not happen with the base plugin, uken/fluent-plugin-elasticsearch.

Steps to replicate

1. Use Dockerfile and test.conf below, build container: docker build -t fluentd:test .
2. Run test: docker run --rm -v $(pwd)/test.conf:/etc/fluent/fluent.conf -it fluentd:test -c /etc/fluent/fluent.conf --dry-run
3. With prefer_oj_serializer false dry run is successful, with prefer_oj_serializer true dry run dumps the traceback below.

traceback

Traceback (most recent call last):
    18: from /usr/bin/fluentd:23:in `<main>'
    17: from /usr/bin/fluentd:23:in `load'
    16: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/bin/fluentd:8:in `<top (required)>'
    15: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    14: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    13: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/lib/fluent/command/fluentd.rb:345:in `<top (required)>'
    12: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/lib/fluent/supervisor.rb:648:in `run_supervisor'
    11: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/lib/fluent/engine.rb:80:in `run_configure'
    10: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/lib/fluent/engine.rb:105:in `configure'
     9: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/lib/fluent/root_agent.rb:143:in `configure'
     8: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/lib/fluent/root_agent.rb:143:in `each'
     7: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/lib/fluent/root_agent.rb:143:in `block in configure'
     6: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/lib/fluent/label.rb:31:in `configure'
     5: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/lib/fluent/agent.rb:64:in `configure'
     4: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/lib/fluent/agent.rb:64:in `each'
     3: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/lib/fluent/agent.rb:74:in `block in configure'
     2: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/lib/fluent/agent.rb:132:in `add_match'
     1: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.12.4/lib/fluent/plugin.rb:178:in `configure'
/usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.0/lib/fluent/plugin/out_elasticsearch.rb:296:in `configure': uninitialized constant Fluent::Plugin::ElasticsearchOutput::Elasticsearch::API (NameError)

Dockerfile

FROM fluent/fluentd:v1.12.4-1.3

USER root

RUN apk upgrade --update --no-cache \
    && apk add --update --no-cache \
    build-base \
    && apk add --no-cache --update --virtual .build-deps sudo ruby-dev \
    && sudo gem install elasticsearch -v 7.13.3 \
    && sudo gem install fluent-plugin-elasticsearch \
    && sudo gem install fluent-plugin-aws-elasticsearch-service \
    && sudo gem sources --clear-all \
    && apk del .build-deps \
    && rm -rf /var/cache/apk/* \
    && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

USER fluent

test.conf

<system>
  log_level warn
  root_dir /fluentd/log
</system>

<source>
  @type forward
  @id "in-forward"
  @label @test
  @log_level warn
  port 24224
  bind 0.0.0.0
  add_tag_prefix test
</source>

<label @test>
  <match **>
    @type "aws-elasticsearch-service"
    @id "out-aws-es"
    @log_level warn
    logstash_format true
    logstash_prefix test
    logstash_dateformat %Y-%m-%d
    include_timestamp true
    include_tag_key true
    time_key timestamp
    reconnect_on_error true
    reload_on_failure true
    reload_connections false
    prefer_oj_serializer true
    <endpoint>
      url https://127.0.0.1
      region us-east-2
      assume_role_arn none
    </endpoint>
  </match>
</label>

Expected Behavior or What you need to ask

Can you help me understand...

1. Is the oj serializer supported by this plugin? Why or why not?
2. Am I missing something?

Using Fluentd and ES plugin versions

• Docker fluent/fluentd:v1.12.4-1.3
• ES plugin fluent-plugin-elasticsearch-5.1.0
• ES version 7.10

[renegaderyu]

I think I might have found the problem. Closing as PICNIC/RTFM

NOTE: If you want to enable json_parser oj by default, The oj gem must be installed separately. This is because oj gem is not required from fluentd by default. If oj gem is not installed, yajl is used as a fallback.

https://docs.fluentd.org/parser/json