search

uken / fluent-plugin-elasticsearch

Apache License 2.0
890 stars 310 forks source link

plugin compatibility between Elasticsearch and clients #971

Closed jcantrill closed 2 years ago

jcantrill commented 2 years ago

(check apply)

Problem

Upon upgrade to latest version available, my tests are now failing with:

The client noticed that the server is not Elasticsearch and we do not support this unknown product.

My lock file: https://github.com/ViaQ/logging-fluentd/blob/578feef8741e2a21c1c8559f9a082303e92d4f37/fluentd/Gemfile.lock

I'm testing against a 7.x server and would like to understand if I should expect compatibility across 6.x->8x ? Are specific versions of this plugin needed to push to specific versions of Elasticsearch?

jcantrill commented 2 years ago

cc @cosmo0920

cosmo0920 commented 2 years ago

The dependent elasticsearch gem and its transport gem should be upgraded. From Elasticsearch 6 to 8, you must upgrade elasticsearch and its transport gem. Note that, for Elasticsearch 8, you must use elastic-tansport gem instead of elasticsearch-transport gem. elasticsearch-transport gem can be dropped from Gemfile.

cosmo0920 commented 2 years ago

And, same of Elasticsearch server and elasticsearch gem versions should be used.

cosmo0920 commented 2 years ago

When using elasticsearch 8 gem, elasticsearch gem will complains here:

https://github.com/elastic/elasticsearch-ruby/blob/ce84322759ff494764bbd096922faff998342197/elasticsearch/lib/elasticsearch.rb#L110-L119

When using elasticsearch 7.14 gem or above, elasticsearch will complains here: https://github.com/elastic/elasticsearch-ruby/blob/7.17/elasticsearch/lib/elasticsearch.rb#L70-L90

cosmo0920 commented 2 years ago

So, the compatibilities should be:

Elasticsearch gem Elasticsearch server
7 7.0.0-SNAPSHOT or above with 'You Know, for Search' and header: x-elastic-product == 'Elasticsearch' or version.build_flavor == "default"
8 8..0.0-SNAPSHOT or above with header: x-elastic-product == 'Elasticsearch'
jcantrill commented 2 years ago

@cosmo0920 in summary, is it possible to support shipping logs to v6, v7, and v8 with one set of these gems, Can I:

cosmo0920 commented 2 years ago

@cosmo0920 in summary, is it possible to support shipping logs to v6, v7, and v8 with one set of these gems, Can I:

  • use latest fluent elasticsearch plugin
  • vendor all the referenced gems from this thread
  • add the headers as needed in the previous comment

We can't do that with default Fluentd options. If we do the different versions of Elasticsearch server, we should use the different fluentd instances which are using the different elasticsearch gem.

Or, we should provide/create gemfiles to bundle the different versions of elasticsearch gems and launch with --gemfile. But I'm not tested yet for this solution.

jcantrill commented 2 years ago

@cosmo0920 Thank you for response. We will need to isolate the gems used depending upon version of Elasticsearch being used.