search

fluent / fluentd-kubernetes-daemonset

Fluentd daemonset for Kubernetes and it Docker image
Apache License 2.0
1.26k stars 982 forks source link

elasticsearch7/v1.15: NoMethodError error="undefined method '+' for nil:NilClass datum[:headers]['Host'] ||= datum[:host] + port_string(datum) at gems/excon-0.93.1/lib/excon/connection.rb:247:in `request' #1407

Closed matthewapeters closed 1 year ago

matthewapeters commented 1 year ago

Description: Daemonset in repeating crash loop due to error. Using Helm to deploy -- significant parts of chart included below.

The docker image builds fine. The image posts to AWS repository without issue. The Helm chart deploys to AES EKS without error.
The pods deploy but will not go ready - logs show error below - and enter CrashLoopBackOff

Please advise is some required parameter is missing: I would expect an error akin to "FLUENT_ELASTICSEARCH_HOST required"

Branch: Master Commit: 449777964f839bddcee2743fe463fce22103735d Image build source: docker-image/v1.15/debian-elasticsearch7 Please Note: Same results using image fluent/fluentd-kubernetes-daemonset:elasticsearch

Kubernetes Status: 

root@xxxxxxxx:~# k get daemonsets
NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fluentd-fluentd-daemonset   3         3         0       3            0           <none>          38s
root@xxxxxxxx:~# k get pods
NAME                                  READY   STATUS    RESTARTS      AGE
fluentd-fluentd-daemonset-5qxqz       0/1     Running   1 (19s ago)   50s
fluentd-fluentd-daemonset-gb92m       0/1     Running   1 (19s ago)   50s
fluentd-fluentd-daemonset-kz4zk       0/1     Running   1 (19s ago)   50s
....
root@xxxxxxxx:~# k get pods
NAME                                  READY   STATUS             RESTARTS      AGE
fluentd-fluentd-daemonset-5qxqz       0/1     CrashLoopBackOff   5 (45s ago)   4m36s
fluentd-fluentd-daemonset-gb92m       0/1     CrashLoopBackOff   5 (55s ago)   4m36s
fluentd-fluentd-daemonset-kz4zk       0/1     CrashLoopBackOff   5 (55s ago)   4m36s

Error:

2023-01-22 01:22:27 +0000 [info]: adding match pattern="**" type="elasticsearch"
2023-01-22 01:22:27 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `+' for nil:NilClass\n\n        datum[:headers]['Host']   ||= datum[:host] + port_string(datum)\n                                                   ^"
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/excon-0.93.1/lib/excon/connection.rb:247:in `request'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/faraday-excon-1.1.0/lib/faraday/adapter/excon.rb:31:in `block in call'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/faraday-1.10.2/lib/faraday/adapter.rb:50:in `connection'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/faraday-excon-1.1.0/lib/faraday/adapter/excon.rb:31:in `call'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/faraday-1.10.2/lib/faraday/rack_builder.rb:154:in `build_response'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/faraday-1.10.2/lib/faraday/connection.rb:516:in `run_request'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elasticsearch-transport-7.17.7/lib/elasticsearch/transport/transport/http/faraday.rb:50:in `block in perform_request'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elasticsearch-transport-7.17.7/lib/elasticsearch/transport/transport/base.rb:288:in `perform_request'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elasticsearch-transport-7.17.7/lib/elasticsearch/transport/transport/http/faraday.rb:36:in `perform_request'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elasticsearch-transport-7.17.7/lib/elasticsearch/transport/client.rb:197:in `perform_request'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elasticsearch-7.17.7/lib/elasticsearch.rb:93:in `elasticsearch_validation_request'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elasticsearch-7.17.7/lib/elasticsearch.rb:51:in `verify_elasticsearch'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elasticsearch-7.17.7/lib/elasticsearch.rb:40:in `method_missing'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elasticsearch-api-7.17.7/lib/elasticsearch/api/actions/info.rb:37:in `info'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/out_elasticsearch.rb:498:in `detect_es_major_version'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/out_elasticsearch.rb:490:in `block in handle_last_seen_es_major_version'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/elasticsearch_index_template.rb:52:in `retry_operate'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/out_elasticsearch.rb:487:in `handle_last_seen_es_major_version'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/out_elasticsearch.rb:339:in `configure'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:187:in `configure'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:132:in `add_match'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:149:in `configure'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:782:in `block in run_worker'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:1056:in `main_process'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:774:in `run_worker'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:370:in `<top (required)>'
  2023-01-22 01:22:27 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-01-22 01:22:27 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `load'
  2023-01-22 01:22:27 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `<main>'
2023-01-22 01:22:27 +0000 [info]: Worker 0 finished with status 1

Helm: templates/deployment.yaml 

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: "{{ .Values.namespace }}"

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
  namespace: "{{ .Values.namespace }}"
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: "{{ .Values.namespace }}"

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: {{ include "fluentd-daemonset.fullname" . }}
  namespace: "{{ .Values.namespace }}"
  labels:
    {{- include "fluentd-daemonset.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "fluentd-daemonset.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "fluentd-daemonset.selectorLabels" . | nindent 8 }}
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      serviceAccountName: fluentd
      #{{ include "fluentd-daemonset.serviceAccountName" . }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.global.aciRegistry }}/{{ .Values.image.repository }}:{{ .Values.global.fluentd.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          env:
            - name:  FLUENT_ELASTICSEARCH_HOST
              value: "{{ .Values.elasticsearch.host }}"
            - name:  FLUENT_ELASTICSEARCH_PORT
              value: "{{ .Values.elasticsearch.port }}"
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName

          volumeMounts:
            {{- toYaml .Values.volumeMounts | nindent 12 }}
          ports:
            - name: http
              containerPort: {{ .Values.service.port }}
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      volumes:
        {{- toYaml .Values.volumes | nindent 8 }}

values.yaml

**# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

image:
  repository: fluentd-kubernetes-elasticsearch7
  # Overrides the image tag whose default is the chart appVersion.
#  tag: "v1.15.3-debian-elasticsearch7-1.0"
  pullPolicy: IfNotPresent

volumeMounts:
- name: varlog
  mountPath: /var/log
- name: dockerlogs
  mountPath: /var/lib/docker/containers
  readOnly: true

volumes:
- name: varlog
  hostPath:
    path: /var/log
- name: dockerlogs
  hostPath:
    path: /var/lib/docker/containers

labels:
  k8s-app: fluentd-logger

elasticsearch:
  host: ""
  port: "9200"
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
#
serviceAccount:
#  # Specifies whether a service account should be created
  create: false
#  # Annotations to add to the service account
  annotations: {}
#  # The name of the service account to use.
#  # If not set and create is true, a name is generated using the fullname template
  name: ""
#
podAnnotations: {}
#
podSecurityContext: {}
  # fsGroup: 2000
#
securityContext: {}
#  # capabilities:
#  #   drop:
#  #   - ALL
#  # readOnlyRootFilesystem: true
#  # runAsNonRoot: true
#  # runAsUser: 1000
#
service:
  type: ClusterIP
  port: "80"
#
ingress:
  enabled: false
#  className: ""
#  annotations: {}
#    # kubernetes.io/ingress.class: nginx
#    # kubernetes.io/tls-acme: "true"
#  hosts:
#    - host: chart-example.local
#      paths:
#        - path: /
#          pathType: ImplementationSpecific
#  tls: []
#  #  - secretName: chart-example-tls
#  #    hosts:
#  #      - chart-example.local
#
#resources: {}
#  # We usually recommend not to specify default resources and to leave this as a conscious
#  # choice for the user. This also increases chances charts run on environments with little
#  # resources, such as Minikube. If you do want to specify resources, uncomment the following
#  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
#  # limits:
#  #   cpu: 100m
#  #   memory: 128Mi
#  # requests:
#  #   cpu: 100m
#  #   memory: 128Mi
#
autoscaling:
  enabled: false
#  minReplicas: 1
#  maxReplicas: 100
#  targetCPUUtilizationPercentage: 80
#  # targetMemoryUtilizationPercentage: 80
#
namespace: default

nodeSelector: {}
#
tolerations: []
#
affinity: {}
#

Invocation to build:

    helm ls | grep ^fluentd || helm install \
        --set "elasticsearch_host=${ELASTICSEARCH_ADDR}" \
        --set "global.aciRegistry=${AWS_ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com" \
        --set "global.fluentd.tag=$(FLUENTD_VERSION)" \
        fluentd fluentd-daemonset
matthewapeters commented 1 year ago

Found error in deployment.yaml -- variable name for host incorrectly matched name of set value.