search

kube-logging / fluentd-images

Custom-built Fluentd images for the Logging operator
Apache License 2.0
2 stars 13 forks source link

missing net_http_persistent faraday extension? #100

Open sylvainOL opened 11 months ago

sylvainOL commented 11 months ago

Hello, after failure to use 1.14 and 1.15, I decided to give a try to 1.16 but no luck :(

I've got an issue with fluent-plugin-enhance-k8s-metadataplugin:

2023-12-19 15:07:03 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-12-19 15:07:03 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-12-19 15:07:04 +0000 [info]: gem 'fluentd' version '1.16.3'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-aliyun-oss' version '0.0.1'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-datapoint' version '2.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.4.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-events' version '2.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-kafka' version '0.19.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-logzio' version '0.2.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.1.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-mysqlslowquery' version '0.0.9'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.4'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.1.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-prometheus-format' version '2.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-protobuf' version '2.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.2.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.3'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-ua-parser' version '1.2.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-vmware-log-intelligence' version '2.0.8'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-vmware-loginsight' version '1.4.2'
2023-12-19 15:07:04 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-12-19 15:07:04 +0000 [info]: gem 'fluentd' version '1.16.2'
2023-12-19 15:07:04 +0000 [info]: [clusterflow:logging:fluent-cluster-flow:0] DeDot will recurse nested hashes and arrays
2023-12-19 15:07:05 +0000 [info]: [globalFilter:0] Kubernetes URL is not set - inspecting environment
2023-12-19 15:07:05 +0000 [info]: [globalFilter:0] cache_ttl: 7200, cache_size: 1000, cache_refresh: 3600, cache_refresh_variation: 900
2023-12-19 15:07:05 +0000 [info]: [globalFilter:0] Initializing Kubernetes API clients
2023-12-19 15:07:05 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:07:05 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 2 second(s)...
2023-12-19 15:07:07 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:07:07 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 4 second(s)...
2023-12-19 15:07:11 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:07:11 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 8 second(s)...
2023-12-19 15:07:19 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:07:19 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 16 second(s)...
2023-12-19 15:07:35 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:07:35 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 32 second(s)...
2023-12-19 15:08:07 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:08:07 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 64 second(s)...
2023-12-19 15:09:11 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
2023-12-19 15:09:11 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 128 second(s)...
2023-12-19 15:11:19 +0000 [info]: [globalFilter:0] create client with URL: https://10.43.0.1:443/api and apiVersion: v1
/usr/local/bundle/gems/faraday-2.7.12/lib/faraday/middleware_registry.rb:57:in `lookup_middleware': :net_http_persistent is not registered on Faraday::Adapter (Faraday::Error)
    from /usr/local/bundle/gems/faraday-2.7.12/lib/faraday/rack_builder.rb:113:in `adapter'
    from /usr/local/lib/ruby/3.2.0/forwardable.rb:240:in `adapter'
    from /usr/local/bundle/gems/fluent-plugin-enhance-k8s-metadata-2.0.0/lib/sumologic/kubernetes/connector.rb:55:in `create_client'
    from /usr/local/bundle/gems/fluent-plugin-enhance-k8s-metadata-2.0.0/lib/sumologic/kubernetes/connector.rb:16:in `block in core_clients'
    from /usr/local/bundle/gems/fluent-plugin-enhance-k8s-metadata-2.0.0/lib/sumologic/kubernetes/connector.rb:15:in `map'
    from /usr/local/bundle/gems/fluent-plugin-enhance-k8s-metadata-2.0.0/lib/sumologic/kubernetes/connector.rb:15:in `core_clients'
    from /usr/local/bundle/gems/fluent-plugin-enhance-k8s-metadata-2.0.0/lib/sumologic/kubernetes/connector.rb:11:in `connect_kubernetes'
    from /usr/local/bundle/gems/fluent-plugin-enhance-k8s-metadata-2.0.0/lib/fluent/plugin/filter_enhance_k8s_metadata.rb:59:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:187:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:154:in `add_filter'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:72:in `block in configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:149:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/bundle/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
    from /usr/local/bundle/bin/fluentd:25:in `load'
    from /usr/local/bundle/bin/fluentd:25:in `<main>'

I tried to see the issue and it seems the gem faraday-net_http_persistent is missing.

```output *** LOCAL GEMS *** abbrev (default: 0.1.1) activemodel (7.1.1) activesupport (7.1.1) addressable (2.8.5) aes_key_wrap (1.1.0) aliyun-sdk (0.7.0) async (1.30.3) async-http (0.60.2) async-io (1.37.0) async-pool (0.4.0) attr_required (1.0.1) aws-eventstream (1.2.0) aws-partitions (1.851.0, 1.842.0) aws-sdk-cloudwatchlogs (1.71.0) aws-sdk-core (3.186.0, 3.185.1) aws-sdk-firehose (1.58.0) aws-sdk-kinesis (1.52.0) aws-sdk-kms (1.72.0) aws-sdk-s3 (1.136.0) aws-sdk-sqs (1.67.0, 1.64.0) aws-sigv4 (1.6.1) base64 (0.2.0, default: 0.1.1) benchmark (default: 0.2.1) bigdecimal (3.1.4, default: 3.1.3, 1.4.4) bindata (2.4.15) bundler (2.4.22, default: 2.4.10) cgi (default: 0.3.6) concurrent-ruby (1.2.2, 1.1.10) connection_pool (2.4.1) console (1.23.2) cool.io (1.8.0) csv (default: 3.2.6) date (default: 3.3.3) debug (1.7.1) declarative (0.0.20) delegate (default: 0.3.0) did_you_mean (default: 1.6.3) dig_rb (1.0.1) digest (default: 3.1.1) digest-crc (0.6.5) domain_name (0.6.20231109, 0.5.20190701) drb (default: 2.1.1) elastic-transport (8.3.0) elasticsearch (8.11.0) elasticsearch-api (8.11.0) english (default: 0.7.2) erb (default: 4.0.2) error_highlight (default: 0.5.1) etc (default: 1.4.2) excon (0.104.0) faraday (2.7.12, 1.10.3) faraday-em_http (1.0.0) faraday-em_synchrony (1.0.0) faraday-excon (2.1.0, 1.1.0) faraday-follow_redirects (0.3.0) faraday-httpclient (1.0.1) faraday-multipart (1.0.4) faraday-net_http (3.0.2, 1.0.1) faraday-net_http_persistent (1.2.0) faraday-patron (1.0.0) faraday-rack (1.0.0) faraday-retry (1.0.3) faraday_middleware (1.2.0) faraday_middleware-aws-sigv4 (1.0.1) fcntl (default: 1.0.2) ffi (1.16.3) ffi-compiler (1.0.1) fiber-annotation (0.2.0) fiber-local (1.0.0) fiddle (default: 1.1.1) fileutils (default: 1.7.0) find (default: 0.1.1) fluent-config-regexp-type (1.0.0) fluent-mixin-config-placeholders (0.4.0) fluent-plugin-aliyun-oss (0.0.1) fluent-plugin-cloudwatch-logs (0.14.3) fluent-plugin-concat (2.5.0) fluent-plugin-datadog (0.14.2) fluent-plugin-datapoint (2.0.0) fluent-plugin-dedot_filter (1.0.0) fluent-plugin-detect-exceptions (0.0.15) fluent-plugin-elasticsearch (5.4.2) fluent-plugin-enhance-k8s-metadata (2.0.0) fluent-plugin-events (2.0.0) fluent-plugin-gcs (0.4.0) fluent-plugin-gelf-hs (1.0.8) fluent-plugin-geoip (1.3.2) fluent-plugin-grafana-loki (1.2.20) fluent-plugin-grok-parser (2.6.2) fluent-plugin-kafka (0.19.2) fluent-plugin-kinesis (3.4.2) fluent-plugin-kube-events-timestamp (0.1.3) fluent-plugin-kubernetes-metadata-filter (2.5.3) fluent-plugin-kubernetes-sumologic (2.0.0) fluent-plugin-label-router (0.2.10) fluent-plugin-logzio (0.2.2) fluent-plugin-mattermost (0.2.2) fluent-plugin-multi-format-parser (1.1.0) fluent-plugin-mysqlslowquery (0.0.9) fluent-plugin-newrelic (1.2.2) fluent-plugin-opensearch (1.1.4) fluent-plugin-parser-logfmt (0.0.0) fluent-plugin-prometheus (2.1.0) fluent-plugin-prometheus-format (2.0.0) fluent-plugin-protobuf (2.0.0) fluent-plugin-record-modifier (2.2.0, 2.1.1) fluent-plugin-redis (0.3.5) fluent-plugin-remote-syslog (1.1) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-s3 (1.7.2) fluent-plugin-splunk-hec (1.3.3) fluent-plugin-sqs (3.0.0) fluent-plugin-sumologic_output (1.8.0) fluent-plugin-syslog_rfc5424 (0.9.0.rc.8) fluent-plugin-systemd (1.0.5) fluent-plugin-tag-normaliser (0.1.2) fluent-plugin-throttle (0.0.5) fluent-plugin-ua-parser (1.2.0) fluent-plugin-vmware-log-intelligence (2.0.8) fluent-plugin-vmware-loginsight (1.4.2) fluent-plugin-webhdfs (1.5.0) fluentd (1.16.3, 1.16.2) forwardable (default: 1.3.3) gelf (3.1.0) geoip-c (0.9.1) geoip2_c (0.3.4) getoptlong (default: 0.2.0) google-apis-core (0.11.2) google-apis-iamcredentials_v1 (0.17.0) google-apis-storage_v1 (0.29.0) google-cloud-core (1.6.1) google-cloud-env (2.0.1) google-cloud-errors (1.3.1) google-cloud-storage (1.45.0) google-protobuf (3.24.4, 3.21.12 ruby x86_64-linux) googleauth (1.9.0) http (5.1.1, 4.4.1) http-accept (1.7.0) http-cookie (1.0.5) http-form_data (2.3.0) http-parser (1.2.3) http_parser.rb (0.8.0) httpclient (2.8.3) i18n (1.14.1) io-console (default: 0.6.0) io-nonblock (default: 0.2.0) io-wait (default: 0.3.0) ipaddr (default: 1.2.5) irb (default: 1.6.2) jmespath (1.6.2) json (2.6.3) json-jwt (1.15.3) jwt (2.7.1) kubeclient (4.10.1, 4.9.3) llhttp-ffi (0.4.0) logfmt (0.0.10) logger (default: 1.5.3) lru_redux (1.1.0) ltsv (0.1.2) mail (2.8.1) matrix (0.4.2) mime-types (3.5.1) mime-types-data (3.2023.1003) mini_mime (1.1.5) mini_portile2 (2.8.5) minitest (5.20.0, 5.16.3) msgpack (1.7.2) multi_json (1.15.0) multipart-post (2.3.0) mutex_m (default: 0.1.2) myslog (0.1.1) net-ftp (0.2.0) net-http (default: 0.3.2) net-http-persistent (4.0.2) net-imap (0.4.2, 0.3.4) net-pop (0.1.2) net-protocol (default: 0.2.1) net-smtp (0.4.0, 0.3.3) netrc (0.11.0) nio4r (2.6.0) nkf (default: 0.1.2) nokogiri (1.15.4) observer (default: 0.1.1) oj (3.15.0) open-uri (default: 0.3.0) open3 (default: 0.1.2) openid_connect (1.1.8) opensearch-ruby (3.0.1) openssl (default: 3.1.0) optparse (default: 0.3.1) os (1.1.4) ostruct (default: 0.5.5) pathname (default: 0.2.1) power_assert (2.0.3) pp (default: 0.4.0) prettyprint (default: 0.1.1) prime (0.1.2) prometheus-client (4.2.2, 4.2.1) protocol-hpack (1.4.2) protocol-http (0.24.7) protocol-http1 (0.15.1) protocol-http2 (0.15.1) pstore (default: 0.1.2) psych (default: 5.0.1) public_suffix (5.0.3) racc (1.7.1, default: 1.6.2) rack (3.0.8) rack-oauth2 (1.21.3) rake (13.0.6) rbs (2.8.2) rdoc (default: 6.5.0) readline (default: 0.0.3) readline-ext (default: 0.1.5) recursive-open-struct (1.1.3) redis (3.3.5) reline (default: 0.3.2) representable (3.2.0) resolv (default: 0.2.2) resolv-replace (default: 0.1.1) rest-client (2.1.0) retriable (3.1.2) rexml (3.2.5) rinda (default: 0.1.1) rss (0.2.9) ruby-kafka (1.5.0) ruby2_keywords (default: 0.0.5) securerandom (default: 0.2.2) serverengine (2.3.2) set (default: 1.0.3) shellwords (default: 0.1.0) sigdump (0.2.5) signet (0.18.0) singleton (default: 0.1.1) snappy (0.3.0) specific_install (0.3.8) stringio (default: 3.0.4) strptime (0.2.5) strscan (default: 3.0.5) swd (2.0.2) syntax_suggest (default: 1.0.2) syslog (default: 0.1.1) syslog_protocol (0.9.2) systemd-journal (1.4.2) tempfile (default: 0.1.3) test-unit (3.5.7) time (default: 0.2.2) timeout (0.4.0, default: 0.3.1) timers (4.3.5) tmpdir (default: 0.1.3) traces (0.11.1) trailblazer-option (0.1.2) tsort (default: 0.1.1) typeprof (0.21.3) tzinfo (2.0.6) tzinfo-data (1.2023.3) uber (0.1.0) un (default: 0.2.1) unf (0.1.4) unf_ext (0.0.8.2) uri (0.12.2, default: 0.12.1) user_agent_parser (2.16.0) uuidtools (2.2.0) validate_email (0.1.6) validate_url (1.0.15) weakref (default: 0.1.2) webfinger (2.1.2) webhdfs (0.10.2) webrick (1.8.1) yajl-ruby (1.4.3) yaml (default: 0.2.1) zlib (default: 3.0.0) ```

My configuration is "basic":

---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: logging
  namespace: logging
spec:
  controlNamespace: logging
  enableRecreateWorkloadOnImmutableFieldChange: true
  flowConfigCheckDisabled: true
  fluentd:
    annotations:
      proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
      sidecar.istio.io/proxyMemory: '2Gi'
      sidecar.istio.io/proxyMemoryLimit: '5Gi'
    bufferStorageVolume:
      pvc:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 3Gi
          storageClassName: fast
          volumeMode: Filesystem
    configCheckAnnotations:
      sidecar.istio.io/inject: 'false'
    extraVolumes:
      - containerName: fluentd
        path: /fluentd/log
        volume:
          emptyDir:
            sizeLimit: 1Gi
        volumeName: log-out
    image:
      pullPolicy: Always
      # renovate: datasource=docker
      repository: ghcr.io/kube-logging/fluentd
      tag: v1.16-full-build.122
    logLevel: info
      enabled: false
  globalFilters:
    - enhanceK8s: {}
pepov commented 11 months ago

I tried adding that gem, but it still fails for me with the same message