null-valued setting found for key [node.attr.zone] found at line number [20], column number [11]

veerendra2 commented 3 weeks ago

Bug Report

What did you do?

Followed Using node topology labels, Kubernetes topology spread constraints, and Elasticsearch shard allocation awareness doc to make elasticsearch on AKS aware zones.

Added node.attr.zone: "${ZONE}" in Elasticsearch CR. Full spec below

What did you expect to see? I would expect elasticsearch should accept the config, up and running

What did you see instead? Under which circumstances?

Elasticsearch data nodes(pod) is CrashLooping with below error

Exception in thread "main" org.elasticsearch.ElasticsearchParseException: null-valued setting found for key [node.attr.zone] found at line number [20], column number [11]
    at org.elasticsearch.common.settings.Settings.validateValue(Settings.java:817)
    at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:792)
    at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:775)
    at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:775)
    at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:736)
    at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1245)
    at org.elasticsearch.node.InternalSettingsPreparer.loadConfigWithSubstitutions(InternalSettingsPreparer.java:126)
    at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:52)
    at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:110)
    at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
    at org.elasticsearch.cli.Command.main(Command.java:50)
    at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
Stream closed EOF for search/elasticsearch-es-data-v3-1 (elasticsearch)

Environment

ECK version:

Elasticsearch verion: 8.12.2 eck-operator helm chart version: 2.6.1

Kubernetes information:

insert any information about your Kubernetes environment that could help us:

Cloud: AKS
Kubernetes version: v1.28.5

for each of them please give us the version you are using

$ k version
Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.5
WARNING: version difference between client (1.30) and server (1.28) exceeds the supported minor version skew of +/-1

Resource definition:

---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
annotations:
eck.k8s.elastic.co/downward-node-labels: "topology.kubernetes.io/zone"
name: elasticsearch
namespace: search
spec:
version: 8.12.2
image: <REDACTED>/images/cust-images/elasticsearch:8.12.2
updateStrategy:
changeBudget:
  maxUnavailable: 1

nodeSets:
- name: master-v3
  count: 3
  config:
    node.attr.zone: "${ZONE}"
    cluster.routing.allocation.awareness.attributes: "k8s_node_name,zone"
    node.roles: [master]
    node.store.allow_mmap: true
    action.auto_create_index: "sync-marker-*,media_ingest,.security*,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*"
    xpack.security.authc:
      anonymous:
        username: anonymous
        roles: superuser
        authz_exception: false
  podTemplate:
    metadata:
      labels:
        app: elasticsearch
      annotations:
        sidecar.istio.io/proxyCPU: "100m"
        sidecar.istio.io/proxyMemory: "480Mi"
        sidecar.istio.io/proxyMemoryLimit: "480Mi"
        traffic.sidecar.istio.io/includeInboundPorts: "*"
        traffic.sidecar.istio.io/excludeOutboundPorts: "9300"
        traffic.sidecar.istio.io/excludeInboundPorts: "9300"
    spec:
      automountServiceAccountToken: true
      imagePullSecrets:
        - name: [REDACTED]
      containers:
        - name: elasticsearch
          env:
            - name: ZONE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.annotations['topology.kubernetes.io/zone']
          resources:
            requests:
              cpu: 1
              memory: 2Gi
            limits:
              memory: 2Gi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: false
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
      initContainers:
        - name: elastic-internal-init-filesystem
          resources:
            limits:
              memory: 50Mi
            requests:
              cpu: 100m
              memory: 50Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: false
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
        - name: elastic-internal-suspend
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
      securityContext:
        fsGroup: 1000
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              elasticsearch.k8s.elastic.co/cluster-name: es-visibility
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              elasticsearch.k8s.elastic.co/cluster-name: es-visibility
  volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: "managed-premium-xfs-noatime"
        resources:
          requests:
            storage: 1Gi

- name: data-v3
  count: 5
  config:
    node.attr.zone: "${ZONE}"
    cluster.routing.allocation.awareness.attributes: "k8s_node_name,zone"
    node.roles: [data]
    node.store.allow_mmap: true
    action.auto_create_index: "media_ingest,.security*,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*"
    xpack.security.authc:
      anonymous:
        username: anonymous
        roles: superuser
        authz_exception: false
  podTemplate:
    metadata:
      labels:
        app: elasticsearch
      annotations:
        sidecar.istio.io/proxyCPU: "100m"
        sidecar.istio.io/proxyMemory: "480Mi"
        sidecar.istio.io/proxyMemoryLimit: "480Mi"
        traffic.sidecar.istio.io/includeInboundPorts: "*"
        traffic.sidecar.istio.io/excludeOutboundPorts: "9300"
        traffic.sidecar.istio.io/excludeInboundPorts: "9300"
    spec:
      automountServiceAccountToken: true
      imagePullSecrets:
        - name: [REDACTED]
      containers:
        - name: elasticsearch
          env:
            - name: ZONE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.annotations['topology.kubernetes.io/zone']
          resources:
            requests:
              cpu: 1
              memory: 2Gi
            limits:
              memory: 2Gi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: false
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
      initContainers:
        - name: elastic-internal-init-filesystem
          resources:
            limits:
              memory: 50Mi
            requests:
              cpu: 100m
              memory: 50Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: false
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
        - name: elastic-internal-suspend
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
      securityContext:
        fsGroup: 1000
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              elasticsearch.k8s.elastic.co/cluster-name: es-visibility
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              elasticsearch.k8s.elastic.co/cluster-name: es-visibility
  volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: "managed-premium-xfs-noatime"
        resources:
          requests:
            storage: 100Gi

- name: ingest-v3
  count: 3
  config:
    node.attr.zone: "${ZONE}"
    cluster.routing.allocation.awareness.attributes: "k8s_node_name,zone"
    node.roles: [ingest]
    node.store.allow_mmap: true
    action.auto_create_index: "media_ingest,.security*,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*"
    xpack.security.authc:
      anonymous:
        username: anonymous
        roles: superuser
        authz_exception: false
  podTemplate:
    metadata:
      labels:
        app: elasticsearch
      annotations:
        sidecar.istio.io/proxyCPU: "100m"
        sidecar.istio.io/proxyMemory: "480Mi"
        sidecar.istio.io/proxyMemoryLimit: "480Mi"
        traffic.sidecar.istio.io/includeInboundPorts: "*"
        traffic.sidecar.istio.io/excludeOutboundPorts: "9300"
        traffic.sidecar.istio.io/excludeInboundPorts: "9300"
    spec:
      automountServiceAccountToken: true
      imagePullSecrets:
        - name: [REDACTED]
      containers:
        - name: elasticsearch
          env:
            - name: ZONE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.annotations['topology.kubernetes.io/zone']
          resources:
            requests:
              cpu: 1
              memory: 2Gi
            limits:
              memory: 2Gi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: false
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
      initContainers:
        - name: elastic-internal-init-filesystem
          resources:
            limits:
              memory: 50Mi
            requests:
              cpu: 100m
              memory: 50Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: false
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
        - name: elastic-internal-suspend
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
      securityContext:
        fsGroup: 1000
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              elasticsearch.k8s.elastic.co/cluster-name: es-visibility
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              elasticsearch.k8s.elastic.co/cluster-name: es-visibility
  volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: "managed-premium-xfs-noatime"
        resources:
          requests:
            storage: 1Gi

podDisruptionBudget:
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      elasticsearch.k8s.elastic.co/cluster-name: elasticsearch

http:
tls:
  selfSignedCertificate:
    disabled: true

Logs:

Exception in thread "main" org.elasticsearch.ElasticsearchParseException: null-valued setting found for key [node.attr.zone] found at line number [20], column number [11]
at org.elasticsearch.common.settings.Settings.validateValue(Settings.java:817)
at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:792)
at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:775)
at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:775)
at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:736)
at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1245)
at org.elasticsearch.node.InternalSettingsPreparer.loadConfigWithSubstitutions(InternalSettingsPreparer.java:126)
at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:52)
at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:110)
at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
at org.elasticsearch.cli.Command.main(Command.java:50)
at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
Stream closed EOF for search/elasticsearch-es-data-v3-1 (elasticsearch)

barkbay commented 3 weeks ago

Could you please provide the operator configuration? Maybe with k get cm -n elastic-system elastic-operator -o yaml.

Thanks

veerendra2 commented 3 weeks ago

@barkbay Please check

$ k get cm -n elastic-system elastic-operator -o yaml
apiVersion: v1
data:
  eck.yaml: log-verbosity: 0\nmetrics-port: 6060\ncontainer-registry: docker.elastic.co\ncontainer-suffix:
    \nmax-concurrent-reconciles: 3\nca-cert-validity: 8760h\nca-cert-rotate-before:
    24h\ncert-validity: 8760h\ncert-rotate-before: 24h\nexposed-node-labels: [topology.kubernetes.io/.*,failure-domain.beta.kubernetes.io/.*]\nset-default-security-context:
    auto-detect\nkube-client-timeout: 60s\nelasticsearch-client-timeout: 180s\ndisable-telemetry:
    false\ndistribution-channel: helm\nvalidate-storage-class: true\nenable-webhook:
    true\nwebhook-name: elastic-operator.elastic-system.k8s.elastic.co\nenable-leader-election:
    true\nelasticsearch-observation-interval: 10s
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: elastic-operator
    meta.helm.sh/release-namespace: elastic-system
  creationTimestamp: "2021-12-22T19:45:12Z"
  labels:
    app.kubernetes.io/instance: elastic-operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: elastic-operator
    app.kubernetes.io/version: 2.6.1
    helm.sh/chart: eck-operator-2.6.1
    helm.toolkit.fluxcd.io/name: elastic-operator
    helm.toolkit.fluxcd.io/namespace: elastic-system
  name: elastic-operator
  namespace: elastic-system

We use flux to sync the resources, below is Helm values in HelmRelease

---
apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: elastic-operator
  namespace: elastic-system
spec:
  interval: 5m
  releaseName: elastic-operator
  chart:
    spec:
      chart: eck-operator
      version: 2.6.1
      sourceRef:
        kind: HelmRepository
        name: elastic
        namespace: flux-system
  install:
    crds: Skip
  upgrade:
    crds: Skip
  values:
    imagePullSecrets:
      - name: [REDACTED]
    image:
      repository: [REDACTED]-elastic-proxy/eck/eck-operator
    installCRDs: false
    resources:
      limits:
        cpu: null
        memory: 512Mi
      requests:
        cpu: 50m
        memory: 256Mi
    podLabels:
      app: elastic-operator
    config:
      metricsPort: 6060
    podMonitor:
      enabled: false
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      privileged: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault

barkbay commented 3 weeks ago

I can't apply this configuration:

error converting YAML to JSON: yaml: line 3: mapping values are not allowed in this context

I would expect the eck.yaml key to looks like the following:

apiVersion: v1
data:
  eck.yaml: |-
    log-verbosity: 0
    metrics-port: 0
    container-registry: docker.elastic.co
    max-concurrent-reconciles: 3
    ca-cert-validity: 8760h
    ca-cert-rotate-before: 24h
    cert-validity: 8760h
    cert-rotate-before: 24h
    disable-config-watch: false
    exposed-node-labels: [topology.kubernetes.io/.*,failure-domain.beta.kubernetes.io/.*]
    set-default-security-context: auto-detect
    kube-client-timeout: 60s
    elasticsearch-client-timeout: 180s
    disable-telemetry: false
    distribution-channel: helm
    validate-storage-class: true
    enable-webhook: true
    webhook-name: elastic-operator.elastic-system.k8s.elastic.co
    webhook-port: 9443
    operator-namespace: elastic-system
    enable-leader-election: true
    elasticsearch-observation-interval: 10s
    ubi-only: false
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: elastic-operator
    meta.helm.sh/release-namespace: elastic-system
  creationTimestamp: "2024-06-10T11:32:05Z"
  labels:
    app.kubernetes.io/instance: elastic-operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: elastic-operator
    app.kubernetes.io/version: 2.13.0
    helm.sh/chart: eck-operator-2.13.0
  name: elastic-operator
  namespace: elastic-system
  resourceVersion: "127910"
  uid: aa350f80-9ff6-4f1a-8c8c-dcbab204bb45

veerendra2 commented 3 weeks ago

Thanks, I used the configmap you provided to test, restarted operator and elasticsearch cluster, getting same error

barkbay commented 3 weeks ago

The config map was not overwritten by Flux? (I'm not familiar with flux)
Did the operator restart following the config map change?
Do the nodes have the expected label?
Do the Pods have the expected label?
Is there any error in the operator logs?

elastic / cloud-on-k8s

null-valued setting found for key [node.attr.zone] found at line number [20], column number [11] #7888

Bug Report