kube-logging / fluentd-images

Custom-built Fluentd images for the Logging operator
Apache License 2.0
2 stars 13 forks source link

Fluentd doesn't work with elasticsearch 7.x #42

Closed sylvainOL closed 2 months ago

sylvainOL commented 1 year ago

Hello, I'm trying to use ghcr.io/kube-logging/fluentd:v1.15-build.84 or ghcr.io/kube-logging/fluentd:v1.14-build.84 with an elasticsearch cluster v7 as output. I've also tested ghcr.io/kube-logging/fluentd:v1.15-staging-build.86 and the issue is still here

I'm getting this error:

  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch_data_stream.rb:64:in `rescue in configure'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch_data_stream.rb:54:in `configure'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:187:in `configure'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:132:in `add_match'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2023-05-22 13:04:45 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
  2023-05-22 13:04:45 +0000 [debug]: /usr/bin/fluentd:23:in `load'
  2023-05-22 13:04:45 +0000 [debug]: /usr/bin/fluentd:23:in `<main>'
2023-05-22 13:04:59 +0000 [info]: init supervisor logger path="/fluentd/log/out" rotate_age=10 rotate_size=10485760
2023-05-22 13:04:59 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.2'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.3.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-kafka' version '0.19.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-mysqlslowquery' version '0.0.9'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.2'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.2'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.2'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-vmware-log-intelligence' version '2.0.6'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-vmware-loginsight' version '1.4.1'
2023-05-22 13:04:59 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-05-22 13:04:59 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-05-22 13:04:59 +0000 [info]: [clusterflow:logging:fluent-cluster-flow:0] DeDot will recurse nested hashes and arrays
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'host elasticsearch-es-http.elasticsearch.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: elasticsearch-es-http.elasticsearch.svc.cluster.local' doesn't have timestamp placeholders for timekey 60
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'host elasticsearch-es-http.elasticsearch.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: elasticsearch-es-http.elasticsearch.svc.cluster.local' doesn't have tag placeholder
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'index_name fluentd.%Y-%m-%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd.%Y-%m-%d' doesn't have timestamp placeholder for day('%d') for timekey 60
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'index_name fluentd.%Y-%m-%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd.%Y-%m-%d' doesn't have tag placeholder
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have timestamp placeholders for timekey 60
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have timestamp placeholders for timekey 60
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have timestamp placeholder for day('%d') for timekey 60
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have timestamp placeholders for timekey 60
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have timestamp placeholders for timekey 60
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have timestamp placeholders for timekey 60
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] Need substitution: false
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'host_placeholder elasticsearch-es-http.elasticsearch.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: elasticsearch-es-http.elasticsearch.svc.cluster.local' doesn't have timestamp placeholders for timekey 60
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'host_placeholder elasticsearch-es-http.elasticsearch.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: elasticsearch-es-http.elasticsearch.svc.cluster.local' doesn't have tag placeholder
2023-05-22 13:05:00 +0000 [warn]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] Consider to specify log_level with @log_level.
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'data_stream_name_placeholder fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: fluentd' doesn't have timestamp placeholders for timekey 60
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] 'data_stream_name_placeholder fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: fluentd' doesn't have tag placeholder
2023-05-22 13:05:00 +0000 [info]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] GET http://fluentd:*********@elasticsearch-es-http.elasticsearch.svc.cluster.local:9200/ [status:200, request:0.029s, query:n/a]
2023-05-22 13:05:00 +0000 [debug]: [clusterflow:logging:fluent-cluster-flow:clusteroutput:logging:fluent-cluster-out] < {
  "name" : "elasticsearch-es-default-0",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "oA7uOED7RFW_R1IzW5mLoA",
  "version" : {
    "number" : "7.17.10",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "fecd68e3150eda0c307ab9a9d7557f5d5fd71349",
    "build_date" : "2023-04-23T05:33:18.138275597Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

2023-05-22 13:05:00 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Failed to create data stream: <fluentd> The client noticed that the server is not Elasticsearch and we do not support this unknown product."
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch_data_stream.rb:64:in `rescue in configure'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch_data_stream.rb:54:in `configure'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:187:in `configure'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:132:in `add_match'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2023-05-22 13:05:00 +0000 [debug]: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
  2023-05-22 13:05:00 +0000 [debug]: /usr/bin/fluentd:23:in `load'
  2023-05-22 13:05:00 +0000 [debug]: /usr/bin/fluentd:23:in `<main>'

I've tried adding :

spec:
  elasticsearch:
    default_elasticsearch_version: '7'
    validate_client_version: false
    verify_es_version_at_startup: false

but it doesn't change anything.

DIgging into the container, I see that elasticsearch gem is present two times (one in version 7.13.3 as aked in Dockerfile, one in version 8.7.1) and we hit this lines: https://github.com/elastic/elasticsearch-ruby/blob/v8.7.1/elasticsearch/lib/elasticsearch.rb#L112-L115

using the "old" ghcr.io/banzaicloud/fluentd:v1.14.6-alpine-666, it works without any issues

pepov commented 1 year ago

hey, thanks for reporting! Do you have an idea what could help fixing the issue? Have you tried playing around with gem version upgrades perhaps? I would specifically prefer to look at the v1.15-staging image as we are going to release a new operator version with that fluentd image as the default.

sylvainOL commented 1 year ago

Hi @pepov, no I didn't for now (to be honest, we plan to move out from elasticsearch to loki in a few weeks...) I'll try to see if I can do something but looking at elastic-ruby, it seems complex to have a solution satisfying both elasticsearch < 7 and >= 8 :(

sylvainOL commented 1 year ago

I've just figured out that you set elasticsearch-ruby to 8.7.1 in v1.15-staging.

So it won't work with elasticsearch < 8.0.0 because of https://github.com/elastic/elasticsearch-ruby/blob/v8.7.1/elasticsearch/lib/elasticsearch.rb#L112-L115

pepov commented 1 year ago

I'm thinking of creating a separate image that works with legacy elasticsearch versions, can you help me with the correct gem version to help with that?

kikisp commented 10 months ago

hey @pepov just encountered same issue.. this is gem version thats used in ghcr.io/banzaicloud/fluentd:v1.14.6-alpine-5 which we use with Elasticsearch 7.10 elasticsearch (7.13.3) elasticsearch-api (7.13.3) elasticsearch-transport (7.13.3) elasticsearch-xpack (7.13.3)

kikisp commented 10 months ago

to add even using yours 1.14 image which according to this https://github.com/kube-logging/fluentd-images/blob/5a36a885bed87b3ab803b2cf8cd626684abda18f/v1.14/Dockerfile#L40 should work still fails..as one user mentioned also above

pepov commented 3 months ago

If anyone would still need this we have a filters image which does not contain output plugins and you can add your own specific version of elasticsearch dependencies to use.

Similar to this one, but with your desired elasticsearch version: https://github.com/kube-logging/logging-operator/issues/1706#issuecomment-2020445093