bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.85k stars 9.14k forks source link

[bitnami/kibana] Unable to retrieve version information from Elasticsearch nodes. unable to get local issuer certificate #12907

Closed PetrusZ closed 1 year ago

PetrusZ commented 1 year ago

Name and Version

bitnami/kibana 10.2.5

What steps will reproduce the bug?

  1. I use bitnami/kibana with bitnami/elasticsearch
  2. With values.yaml
  3. Run helm install -n database --create-namespace -f kibana.values.yaml kibana bitnami/kibana
  4. See error Unable to retrieve version information from Elasticsearch nodes. unable to get local issuer certificate

Are you using any custom parameters or values?

plugins:
  - https://github.com/pjhampton/kibana-prometheus-exporter/releases/download/7.14.0/kibana-prometheus-exporter-7.14.0.zip

ingress:
  enabled: true
  hostname: kibana.codeplayer.org
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  tls: true
  ingressClassName: "nginx"

metrics:
  enabled: true
  serviceMonitor:
    enabled: true

tls:
  enabled: true
  autoGenerated: true

elasticsearch:
  hosts:
    - elastic.codeplayer.org
  port: 443
  security:
    auth:
      enabled: true
      existingSecret: "kibana-secret"
      createSystemUser: true
      elasticsearchPasswordSecret: "elastic-security"
    tls:
      enabled: true
      existingSecret: "elasticsearch-coordinating-crt"
      usePemCerts: true

What is the expected behavior?

No response

What do you see instead?

I have no name!@kibana-8754974fd-twpvp:/$ /opt/bitnami/scripts/kibana/entrypoint.sh /opt/bitnami/scripts/kibana/run.sh
kibana 15:47:13.09
kibana 15:47:13.09 Welcome to the Bitnami kibana container
kibana 15:47:13.09 Subscribe to project updates by watching https://github.com/bitnami/containers
kibana 15:47:13.10 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kibana 15:47:13.10
kibana 15:47:13.10 INFO  ==> ** Starting Kibana setup **
kibana 15:47:13.11 DEBUG ==> Validating settings in KIBANA_* environment variables...
kibana 15:47:13.12 INFO  ==> Configuring/Initializing Kibana...
kibana 15:47:13.12 DEBUG ==> Ensuring expected directories/files exist...
kibana 15:47:13.14 INFO  ==> Found mounted configuration directory
kibana 15:47:13.21 INFO  ==> Waiting for Elasticsearch to be ready.
kibana 15:47:13.23 DEBUG ==> Attempted to connect with Elasticserach. Status code: 401
kibana 15:47:13.25 INFO  ==> Skipping 'kibana_system' user creation. User already exists. Status code: 200
kibana 15:47:13.26 INFO  ==> Starting Kibana in background
kibana 15:47:13.26 INFO  ==> Waiting for Kibana to be started and ready
kibana 15:47:13.27 DEBUG ==> Waiting for Kibana server: 30 remaining attempts...
[2022-10-11T15:47:15.211+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:15.34 DEBUG ==> Waiting for Kibana server: 29 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:17.37 DEBUG ==> Waiting for Kibana server: 28 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:19.41 DEBUG ==> Waiting for Kibana server: 27 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:21.44 DEBUG ==> Waiting for Kibana server: 26 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:23.47 DEBUG ==> Waiting for Kibana server: 25 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:25.50 DEBUG ==> Waiting for Kibana server: 24 remaining attempts...
[2022-10-11T15:47:26.437+00:00][INFO ][http.server.Preboot] http server running at https://127.0.0.1:5601
[2022-10-11T15:47:26.483+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2022-10-11T15:47:26.524+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2022-10-11T15:47:26.751+00:00][INFO ][plugins-system.standard] Setting up [121] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,newsfeed,fieldFormats,expressions,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,lists,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,telemetry,licenseManagement,eventLog,actions,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,unifiedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,eventAnnotation,dataViewFieldEditor,triggersActionsUi,transform,stackAlerts,ruleRegistry,discover,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,aiops,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,osquery,maps,dataVisualizer,ml,cases,timelines,sessionView,kubernetesSecurity,securitySolution,visTypeGauge,sharedUX,observability,synthetics,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,dataViewManagement]
[2022-10-11T15:47:26.770+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 82aabb39-6b94-419e-8991-14e3e1da5d29
[2022-10-11T15:47:26.856+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2022-10-11T15:47:26.880+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2022-10-11T15:47:26.888+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2022-10-11T15:47:26.905+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2022-10-11T15:47:27.006+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2022-10-11T15:47:27.013+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2022-10-11T15:47:27.054+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2022-10-11T15:47:27.104+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
[2022-10-11T15:47:27.722+00:00][WARN ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, but is not supported for Linux Debian 11.5 OS. Automatically setting 'xpack.screenshotting.browser.chromium.disableSandbox: true'.
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:27.76 DEBUG ==> Waiting for Kibana server: 23 remaining attempts...
[2022-10-11T15:47:27.785+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. unable to get local issuer certificate
[2022-10-11T15:47:28.139+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /opt/bitnami/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:29.92 DEBUG ==> Waiting for Kibana server: 22 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:31.96 DEBUG ==> Waiting for Kibana server: 21 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:34.00 DEBUG ==> Waiting for Kibana server: 20 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:36.03 DEBUG ==> Waiting for Kibana server: 19 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:38.07 DEBUG ==> Waiting for Kibana server: 18 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:40.11 DEBUG ==> Waiting for Kibana server: 17 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:42.13 DEBUG ==> Waiting for Kibana server: 16 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:44.18 DEBUG ==> Waiting for Kibana server: 15 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:46.22 DEBUG ==> Waiting for Kibana server: 14 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:48.26 DEBUG ==> Waiting for Kibana server: 13 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:50.29 DEBUG ==> Waiting for Kibana server: 12 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:52.33 DEBUG ==> Waiting for Kibana server: 11 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:54.37 DEBUG ==> Waiting for Kibana server: 10 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:56.41 DEBUG ==> Waiting for Kibana server: 9 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:47:58.45 DEBUG ==> Waiting for Kibana server: 8 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:48:00.47 DEBUG ==> Waiting for Kibana server: 7 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:48:02.56 DEBUG ==> Waiting for Kibana server: 6 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:48:04.59 DEBUG ==> Waiting for Kibana server: 5 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:48:06.63 DEBUG ==> Waiting for Kibana server: 4 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:48:08.68 DEBUG ==> Waiting for Kibana server: 3 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:48:10.71 DEBUG ==> Waiting for Kibana server: 2 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:48:12.74 DEBUG ==> Waiting for Kibana server: 1 remaining attempts...
Error: cannot pick indicies from type !!null (status.overall)
kibana 15:48:14.78 ERROR ==> Kibana is not available after 30 retries

Additional information

No response

javsalgar commented 1 year ago

Hi,

Could you add more details on how you created the certificate?

PetrusZ commented 1 year ago

The certificate is auto generated by bitnami/elasticsearch, I use those values in bitnami/elasticsearch:

global:
  kibanaEnabled: false

security:
  enabled: true
  existingSecret: "elastic-security"
  tls:
    autoGenerated: true

ingress:
  enabled: true
  hostname: elastic.codeplayer.org
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  tls: true
  ingressClassName: "nginx"
fmulero commented 1 year ago

It seems there is something missing in the certificate, the intermediate issue or something like that. Have you tried relaxing the verification mode?

      ## @param elasticsearch.security.tls.verificationMode Verification mode for SSL communications.
      ## Supported values: full, certificate, none.
      ## Ref: https://www.elastic.co/guide/en/kibana/7.x/settings.html#elasticsearch-ssl-verificationmode
      verificationMode: "full"
PetrusZ commented 1 year ago

Hi @fmulero, even I set verificationMode to none, I still got the same error.

fmulero commented 1 year ago

Hi Petrus I've tested 2 different scenarios with autogenerated certs:

  1. Without ingress configuration.
  2. With autogenerated certs in ingress.

Without ingress configuration

For the first scenario I deployed elasticsearch:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm install elasticsearch bitnami/elasticsearch -f elasticsearch.yml
$ helm install kibana bitnami/kibana -f kibana.yml

This is the content of the elasticsearch.yml file (the secret is included to facilitate testing):

global:
  kibanaEnabled: false

security:
  enabled: true
  existingSecret: elastic-security
  tls:
    autoGenerated: true
    restEncryption: true

extraDeploy:
  - kind: Secret
    apiVersion: v1
    type: Opaque
    metadata:
      name: elastic-security
      namespace: default
    data:
      elasticsearch-password: SGVsbG9Xb3JsZCE=

And this is the configuration for kibana. Here I have to configure kibana to trust on the certificate created by the elasticsearch deployment:

metrics:
  enabled: true
  serviceMonitor:
    enabled: true
image:
  debug: true
tls:
  enabled: true
  autoGenerated: true
elasticsearch:
  hosts:
    - elasticsearch
  port: 9200
  security:
    auth:
      enabled: true
      existingSecret: kibana-secret
      createSystemUser: true
      elasticsearchPasswordSecret: elastic-security
    tls:
      enabled: true
      existingSecret: elasticsearch-coordinating-crt
      usePemCerts: true
extraDeploy:
  - kind: Secret
    apiVersion: v1
    type: Opaque
    metadata:
      name: kibana-secret
      namespace: default
    data:
      kibana-password: SGVsbG9Xb3JsZCE=

With autogenerated certs in ingress

In this scenario I've deployed elasticsearch and kibana enabling the ingress for both.

$ helm install elasticsearch bitnami/elasticsearch -f elasticsearch-ingress.yml
$ helm install kibana bitnami/kibana -f kibana-ingress.yml

This is the content of the elasticsearch-ingress.yml file:

global:
  kibanaEnabled: false
security:
  enabled: true
  existingSecret: "elastic-security"
  tls:
    autoGenerated: true
    restEncryption: true
ingress:
  enabled: true
  hostname: elastic.codeplayer.org
  tls: true
  selfSigned: true
  ingressClassName: "nginx"
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
extraDeploy:
  - kind: Secret
    apiVersion: v1
    type: Opaque
    metadata:
      name: elastic-security
      namespace: default
    data:
      elasticsearch-password: SGVsbG9Xb3JsZCE=

And this other code block has the content of kibana-ingress.yml file:

metrics:
  enabled: true
  serviceMonitor:
    enabled: true
image:
  debug: true
ingress:
  enabled: true
  hostname: kibana.codeplayer.org
  tls: true
  selfSigned: true
  ingressClassName: "nginx"
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
tls:
  enabled: true
  autoGenerated: true
elasticsearch:
  hosts:
    - elastic.codeplayer.org
  port: 443
  security:
    auth:
      enabled: true
      existingSecret: "kibana-secret"
      createSystemUser: true
      elasticsearchPasswordSecret: "elastic-security"
    tls:
      enabled: true
      existingSecret: "elastic.codeplayer.org-tls"
      usePemCerts: true
extraDeploy:
  - kind: Secret
    apiVersion: v1
    type: Opaque
    metadata:
      name: kibana-secret
      namespace: default
    data:
      kibana-password: SGVsbG9Xb3JsZCE=

In this scenario I've configured kibana to use the ingress endpoint (elastic.codeplayer.org:443) instead of the service (elasticsearch:9200) and now kibana must trust on the certificate exposed by the ingress.

PetrusZ commented 1 year ago

OMG, I forgot I should use the certificate exposed by the ingress when I use t he ingress endpoint as host. I'm stupid.

Thank you very much fmulero!