Kibana install using ECK - [server].port]: expected value of type [number] but got [string]

[plutley]

Bug Report

What did you do?

Tried deploying kibana instance following quickstart quide - https://www.elastic.co/guide/en/cloud-on-k8s/2.2/k8s-deploy-kibana.html Literally copy/pasting instructions from the quickstart.

• The ES install was fine.

What did you expect to see?

kibana pod running without error

What did you see instead? Under which circumstances?

kibana pod in CrashLoopBackOff

Environment

• ECK version:

  8.2.3

• Kubernetes information:

  insert any information about your Kubernetes environment that could help us:

  • Cloud: EKS (v1.20)

  for each of them please give us the version you are using

  
  $ kubectl version

Client Version: v1.24.1 Kustomize Version: v4.5.4 Server Version: v1.20.15-eks-a64ea69

* Resource definition:

apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: quickstart spec: version: 8.2.3 count: 1 elasticsearchRef: name: quickstart

* Logs:

From the kibana pod:

[2022-06-27T15:54:56.327+00:00][FATAL][root] Error: [config validation of [server].port]: expected value of type [number] but got [string] at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:95:13) at ConfigService.validateAtPath (/usr/share/kibana/node_modules/@kbn/config/target_node/config_service.js:228:19) at MapSubscriber.project (/usr/share/kibana/node_modules/@kbn/config/target_node/config_service.js:236:169) at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35) at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18) at DistinctUntilChangedSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30) at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18) at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26) at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18) at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:80:28) at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:44:25) at ReplaySubject.Subject._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Subject.js:102:51) at ReplaySubject.Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:30:22) at MapSubscriber.shareReplayOperation (/usr/share/kibana/node_modules/rxjs/internal/operators/shareReplay.js:48:32) at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:25:31) at MapOperator.call (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:32:23) at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:25:31) at DistinctUntilChangedOperator.call (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:27:23) at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:25:31) at MapOperator.call (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:32:23) at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:25:31) at TakeOperator.call (/usr/share/kibana/node_modules/rxjs/internal/operators/take.js:38:23) at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:25:31) at /usr/share/kibana/node_modules/rxjs/internal/Observable.js:99:19 at new Promise () at Observable.toPromise (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:97:16) at EnvironmentService.preboot (/usr/share/kibana/src/core/server/environment/environment_service.js:49:234) at Server.preboot (/usr/share/kibana/src/core/server/server.js:157:55) at Root.preboot (/usr/share/kibana/src/core/server/root/index.js:48:32) at processTicksAndRejections (node:internal/process/task_queues:96:5) at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:99:9) at Command. (/usr/share/kibana/src/cli/serve/serve.js:216:5)

FATAL Error: [config validation of [server].port]: expected value of type [number] but got [string]

describing kibana pod I noticed a port variable - unsure if related

k describe po quickstart-kb-7dfb4b799f-hwjbx 1 err | Name: quickstart-kb-7dfb4b799f-hwjbx Namespace: default Priority: 0 Node: Start Time: Mon, 27 Jun 2022 16:38:52 +0100 Labels: common.k8s.elastic.co/type=kibana kibana.k8s.elastic.co/name=quickstart kibana.k8s.elastic.co/version=8.2.3 pod-template-hash=7dfb4b799f Annotations: co.elastic.logs/module: kibana kibana.k8s.elastic.co/config-hash: 3192103546 kubernetes.io/psp: eks.privileged Status: Running IP: 10.216.136.37 IPs: IP: 10.216.136.37 Controlled By: ReplicaSet/quickstart-kb-7dfb4b799f Init Containers: elastic-internal-init-config: Container ID: docker://3cf61b37887daba9cf81784cdc59652ec97419846c1f059ccdaf3ea42d69b552 Image: docker.elastic.co/kibana/kibana:8.2.3 Image ID: docker-pullable://docker.elastic.co/kibana/kibana@sha256:b102c86e17311a1e936c51aa787751de3806702522ea2cbdcbc18eb9359601c9 Port: Host Port:

[thbkrkr]

Seems similar to #1987. Do you have a Kubernetes Service named server in the same namespace?

If so, that is the cause of the problem. When a Pod is run on a Node, the kubelet adds a set of environment variables for each active Service (doc). With a server Service, the Kibana Pod gets a SERVER_PORT=tcp://<ip>:<port> env var which is then used by Kibana to configure its port but this value isn't a valid port.

You have 3 options to avoid this:

• disable enableServiceLinks to prevent the automatic injection of Service environment variables:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 8.2.3
  count: 1
  elasticsearchRef:
    name: quickstart
  podTemplate:
     spec:
        enableServiceLinks: false

• rename the server Service in something else
• override the injected env var by statically setting it:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 8.2.3
  count: 1
  elasticsearchRef:
    name: quickstart
  podTemplate:
    spec:
      containers:
      - name: kibana
        env:
        - name: SERVER_PORT
          value: "5601"

We planned to set enableServiceLinks to false by default in all ECK managed Pod specs (#2030).