Elasticsearch 8.0 release breaks plugin (cannot load such file -- elasticsearch/transport/transport/connections/selector)

nnul commented 2 years ago

(check apply)

[x] read the contribution guideline
[ ] (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Problem

The release of the Elasticsearch 8.0 ruby package has broken this plugin. The Gemspec does not version the elasticsearch dependency and v8.0 has breaking changes related to the transport code. The easy fix is to version the elasticsearch gem dependency to v7.x until v8.x support is added.

Callstack:

 /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:85:in `require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)
      from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:85:in `require'
      from /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/elasticsearch_fallback_selector.rb:1:in `<top (required)>'
      from /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/out_elasticsearch.rb:32:in `require_relative'
      from /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/out_elasticsearch.rb:32:in `<top (required)>'
      from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:85:in `require'
      from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:85:in `require'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/registry.rb:102:in `block in search'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/registry.rb:99:in `each'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/registry.rb:99:in `search'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/registry.rb:44:in `lookup'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/plugin.rb:169:in `new_impl'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/plugin.rb:114:in `new_output'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:108:in `block in configure'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:99:in `each'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:99:in `configure'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/plugin/out_copy.rb:39:in `configure'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/plugin.rb:187:in `configure'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/agent.rb:132:in `add_match'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/agent.rb:74:in `block in configure'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/agent.rb:64:in `each'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/agent.rb:64:in `configure'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/root_agent.rb:149:in `configure'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/engine.rb:105:in `configure'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/engine.rb:80:in `run_configure'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/supervisor.rb:665:in `run_supervisor'
      from /usr/local/bundle/gems/fluentd-1.14.0/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
      from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:85:in `require'
      from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:85:in `require'
      from /usr/local/bundle/gems/fluentd-1.14.0/bin/fluentd:15:in `<top (required)>'
      from /usr/local/bundle/bin/fluentd:23:in `load'
      from /usr/local/bundle/bin/fluentd:23:in `<main>'

Steps to replicate

Run the fluentd plugin from a new install.

Expected Behavior or What you need to ask

The plugin runs as expected.

Using Fluentd and ES plugin versions

Affects all versions and all platforms

nnul commented 2 years ago

As a work around you can force uninstall the elasticsearch gem and install v7.17.0.

gem uninstall -I elasticsearch
gem install elasticsearch -v 7.17.0

nnul commented 2 years ago

I believe this is the root cause and should instead be versioned: https://github.com/uken/fluent-plugin-elasticsearch/blob/9cafce0c01fa9e21b36784ac82e9526d2f4767e4/fluent-plugin-elasticsearch.gemspec#L27

nnul commented 2 years ago

Additional information on the release that triggered this: https://github.com/elastic/elasticsearch-ruby/releases/tag/v8.0.0

PhotoTeeborChoka commented 2 years ago

Hi @nnul can you prepare a PR? If this plugin however wants to be backwards compatible with pre 8.0 releases of elasticsearch, there should be a single version pinning it down and then go up to elasticsearch 8.0.0 anyway.

As a work around you can force uninstall the elasticsearch gem and install v7.17.0.
gem uninstall -I elasticsearch
gem install elasticsearch -v 7.17.0

This is how I did it in my custom image, works like a charm. :+1:

tumbl3w33d commented 2 years ago

My error was similar, but seems related: cannot load such file -- elasticsearch/xpack (LoadError)

Inspired by the hint above this worked in my Dockerfile:

FROM fluent/fluentd:edge

USER root

RUN apk add --no-cache --update --virtual .build-deps \
        sudo build-base ruby-dev \
 && sudo gem install 'elasticsearch:7.17.0' \
                     'elasticsearch-api:7.17.0' \
                     'elasticsearch-xpack:7.17.0' \
                     fluent-plugin-elasticsearch \
                     oj \
                     fluent-plugin-rewrite-tag-filter \
 && sudo gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

USER fluent

nnul commented 2 years ago

Hi @nnul can you prepare a PR?

Unfortunately, I am not familiar enough with Ruby gems to be able to make the needed changes to the gemspec file and test it currently. I was hoping this issue would provide enough information for another maintainer to make the required changes to this plugin.

In the meantime, I'm glad to see the workaround I provided is helping people get unstuck. :smile:

kvokka commented 2 years ago

Pls, keep in mind that gem CI is falsely green on master, cos elasticsearch-ruby gem was released after the latest CI execution. Yet another possible workaround https://github.com/bitnami/bitnami-docker-fluentd/issues/35#issuecomment-1046420417

nnul commented 2 years ago

I noticed that the fix in #949 adds support for v8.0 which has fixed the issue, however it does not address the underlying time bomb of relying on an unversioned elasticsearch dependency in the gemspec file. I'm unaware of how semver packages work in Ruby/Gems but ideally this package should be versioned and only allow implicit minor upgrades w/o forcing a major version upgrade upon installation. If this package remains unversioned, then this issue has the potential to arise again in the future when another breaking change is introduced in that dependency.

uken / fluent-plugin-elasticsearch