search

kube-logging / logging-operator

Logging operator for Kubernetes
https://kube-logging.dev
Apache License 2.0
1.53k stars 327 forks source link

Upgrade from v3.17.10 to v4.2.0 problem #1367

Closed karloramone closed 1 year ago

karloramone commented 1 year ago

Bugs should be filed for issues encountered whilst operating logging-operator. You should first attempt to resolve your issues through the community support channels, e.g. Slack, in order to rule out individual configuration errors. #logging-operator Please provide as much detail as possible.

Describe the bug: I use banzai-cloud/logging-operator 3.17.10 and banzai-cloud/logging-operator-logging 3.17.10 configured to send the logs to OpenSearch v2.5.0 with ElasticSearch plugin. I would like to upgrade from v3.17.10 to v4.2.0.

When I try to upgrade with my current elasticsearch cluster output config I get an error that server is not Elasticsearch.

             elasticsearch:
               buffer:
                 flush_interval: 1s
                 flush_mode: interval
                 flush_thread_count: 2
                 retry_max_interval: 60s
                 retry_timeout: 72h
               default_elasticsearch_version: "8"
               hosts: log-opensearch1.myserver.com,log-opensearch2.myserver.com
               include_timestamp: true
               index_name: cluster-a
               log_es_400_reason: true
               password:
                 valueFrom:
                   secretKeyRef:
                     name: opensearch-credentials
                     key: password
               port: 9200
               reconnect_on_error: true
               scheme: https
               ssl_verify: false
               suppress_type_name: true
               target_index_key: 'cluster-a'
               user: myuser
               verify_es_version_at_startup: false
  2023-04-17 16:03:59 +0000 [warn]: #0 suppressed same stacktrace
2023-04-17 16:03:59 +0000 [warn]: #0 [flow:logging:events:clusteroutput:logging:opensearch] failed to flush the buffer. retry_times=7 next_retry_time=2023-04-17 16:05:00 +0000 chunk="5f98a4f8e4d99797074acfbc8f0a54fc" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"log-opensearch1.myserver.com\", :port=>9200, :scheme=>\"https\", :user=>\"myuser\", :password=>\"obfuscated\"}, {:host=>\"log-opensearch1.myserver.com\", :port=>9200, :scheme=>\"https\", :user=>\"myuser\", :password=>\"obfuscated\"}): The client noticed that the server is not Elasticsearch and we do not support this unknown product.

When I change the output to opensearch and config to:

             opensearch:
               buffer:
                 flush_interval: 1s
                 flush_mode: interval
                 flush_thread_count: 2
                 retry_max_interval: 60s
                 retry_timeout: 72h
               default_opensearch_version: 2
               hosts: log-opensearch1.myserver.com,log-opensearch2.myserver.com
               include_timestamp: true
               index_name: cluster-a
               log_os_400_reason: true
               password:
                 valueFrom:
                   secretKeyRef:
                     name: opensearch-credentials
                     key: password
               port: 9200
               reconnect_on_error: true
               scheme: https
               ssl_verify: false
               suppress_type_name: true
               target_index_key: 'cluster-a'
               user: myuser
               verify_os_version_at_startup: false

I get an error of broken ruby gems in logging-operator-logging-fluentd-configcheck pod:

fluentd -c /fluentd/etc/fluent.conf --dry-run
2023-06-20 08:17:34 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-06-20 08:17:34 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-06-20 08:17:35 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.2'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.3.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.22'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-mysqlslowquery' version '0.0.9'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.2'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.10'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.2'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.2'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-vmware-log-intelligence' version '2.0.6'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-vmware-loginsight' version '1.4.0'
2023-06-20 08:17:35 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-06-20 08:17:35 +0000 [info]: starting fluentd-1.15.3 as dry run mode ruby="3.1.4"
2023-06-20 08:17:35 +0000 [info]: [flow:acme-website-dev:logs:2] DeDot will recurse nested hashes and arrays
/usr/lib/ruby/3.1.0/rubygems/specification.rb:2288:in `raise_if_conflicts': Unable to activate faraday_middleware-aws-sigv4-0.6.1, because faraday-2.7.6 conflicts with faraday (>= 1.8, < 2) (Gem::ConflictError)
        from /usr/lib/ruby/3.1.0/rubygems/specification.rb:1419:in `activate'
        from /usr/lib/ruby/3.1.0/rubygems.rb:211:in `rescue in try_activate'
        from /usr/lib/ruby/3.1.0/rubygems.rb:204:in `try_activate'
        from <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:153:in `rescue in require'
        from <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
        from /usr/lib/ruby/gems/3.1.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:58:in `<top (required)>'
        from <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
        from <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:25:in `load'
        from /usr/bin/fluentd:25:in `<main>'
/usr/lib/ruby/3.1.0/rubygems/specification.rb:2288:in `raise_if_conflicts': Unable to activate faraday_middleware-aws-sigv4-0.6.1, because faraday-2.7.6 conflicts with faraday (>= 1.8, < 2) (Gem::ConflictError)

Expected behaviour: Fluentd config check passes and logs are sent to OpenSearch successfully.

Steps to reproduce the bug: Try to upgrade from version v3.17.10 to v4.2.0 with both configurations from above.

Environment details:

/kind bug

vanveele commented 1 year ago

Conflict exists in fluentd image release. I have proposed a fix in this PR

pepov commented 1 year ago

That fix is already published, did that resolve your problem?

pepov commented 1 year ago

Closing for now, please reopen if needed