Certificate ca_sha256 ignored in Fleet initialisation config file

[marcanpilami]

Kibana version:

8.3.3

Elasticsearch version:

8.3.3

Server OS version:

Linux (Ubuntu 22.04, Kernel 5.10.102.1) + Docker CE (20.10.17) (also reproduced outside Docker)

Browser version:

Any.

Browser OS version:

Any

Original install method (e.g. download page, yum, from source, etc.):

Docker compose file (provided below)

Describe the bug:

The xpack.fleet.agents.elasticsearch.ca_sha256 value inside a Kibana config file is supposed according to https://www.elastic.co/guide/en/kibana/current/fleet-settings-kb.html to allow setting the certificate thumbprint for agent communications with elasticsearch.

When set manually inside the Kibana UI, it works perfectly.

However, when set using the configuration file, it seems ignored. The other parameters (like xpack.fleet.agents.elasticsearch.hosts) are correctly imported, but ca_sha256 is not. It does not appear inside the Kibana UI, and it is not usable by agents until manually set in the UI so it is likely not a display bug.

Steps to reproduce:

Just run the following docker compose file, simplified from by the official doc. The script inside the first container creates a Kibana config file with the correct fingerprint.

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.3.3
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
          echo "Creating xpack fleet config file" ;
          openssl x509 -fingerprint -sha256 -in config/certs/ca/ca.crt  | grep -i sha256 | sed "s/^.*=\(.*\)$$/\1/g" | tr -d ":" > config/certs/ca/ca.sha256 ;
          printf "xpack.fleet.agents.fleet_server.hosts: [ \"https://fleet:8220\" ]\nxpack.fleet.agents.elasticsearch.hosts: [ \"https://es01:9200\" ]\nxpack.fleet.agents.elasticsearch.ca_sha256: " > config/certs/ca/xpack.ca.yml ;
          cat config/certs/ca/ca.sha256 >> config/certs/ca/xpack.ca.yml ;
          echo "" >> config/certs/ca/xpack.ca.yml ;  
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:8.3.3
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    environment:
      - node.name=es01
      - cluster.name=none
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - discovery.type=single-node
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=basic
    mem_limit: 1010612736
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:8.3.3
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT:-5701}:5601
    command: /usr/local/bin/kibana-docker -c /usr/share/kibana/config/certs/ca/xpack.ca.yml -c config/kibana.yml
    environment:
      SERVERNAME: kibana
      ELASTICSEARCH_HOSTS: https://es01:9200
      ELASTICSEARCH_USERNAME: kibana_system
      ELASTICSEARCH_PASSWORD: ${KIBANA_PASSWORD}
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: config/certs/ca/ca.crt
    mem_limit: 1010612736
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local

Check the configuration file was correctly generated (the fingerprint will change in your results):

xpack.fleet.agents.fleet_server.hosts: [ "https://fleet:8220" ]
xpack.fleet.agents.elasticsearch.hosts: [ "https://es01:9200" ]
xpack.fleet.agents.elasticsearch.ca_sha256: 04883E122794C3248BB8673E0CC38D1780067E6BFCF67CF2717EEE83093876E3

After that go to http://localhost:5701/app/fleet/settings/outputs/fleet-default-output and check that the "Elasticsearch CA trusted fingerprint" field is empty instead of populated (while the "hosts" field is populated as expected).

Expected behavior:

Field is populated by certificate fingerprint.

Screenshots (if relevant):

N/A

Errors in browser console (if relevant):

N/A

Provide logs and/or server output (if relevant):

Kibana startup log:

[2022-08-24T16:48:55.098+00:00][INFO ][plugins-service] Plugin "cloudSecurityPosture" is disabled.
[2022-08-24T16:48:55.188+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
[2022-08-24T16:48:55.228+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2022-08-24T16:48:55.277+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2022-08-24T16:48:55.482+00:00][INFO ][plugins-system.standard] Setting up [118] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,share,embeddable,uiActionsEnhanced,screenshotMode,banners,newsfeed,fieldFormats,expressions,eventAnnotation,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,lists,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,telemetry,licenseManagement,kibanaUsageCollection,eventLog,actions,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,aiops,unifiedSearch,savedObjects,triggersActionsUi,transform,stackAlerts,ruleRegistry,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,sharedUX,discover,lens,maps,dataVisualizer,ml,cases,timelines,sessionView,observability,fleet,synthetics,osquery,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,discoverEnhanced,dataViewManagement]
[2022-08-24T16:48:55.496+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 09ffcfab-8e3f-4fb6-a6f1-17f5010eabbd
[2022-08-24T16:48:55.549+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2022-08-24T16:48:55.549+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2022-08-24T16:48:55.572+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2022-08-24T16:48:55.573+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2022-08-24T16:48:55.579+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2022-08-24T16:48:55.599+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2022-08-24T16:48:55.680+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2022-08-24T16:48:55.682+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[2022-08-24T16:48:55.686+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2022-08-24T16:48:55.711+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2022-08-24T16:48:56.262+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
[2022-08-24T16:48:56.844+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2022-08-24T16:48:56.845+00:00][INFO ][savedobjects-service] Starting saved objects migrations
[2022-08-24T16:48:56.881+00:00][INFO ][savedobjects-service] [.kibana] INIT -> CREATE_NEW_TARGET. took: 23ms.
[2022-08-24T16:48:56.884+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 24ms.
[2022-08-24T16:48:57.009+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
[2022-08-24T16:48:57.018+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 134ms.
[2022-08-24T16:48:57.038+00:00][INFO ][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 157ms.
[2022-08-24T16:48:57.070+00:00][INFO ][savedobjects-service] [.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 52ms.
[2022-08-24T16:48:57.070+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 210ms
[2022-08-24T16:48:57.089+00:00][INFO ][savedobjects-service] [.kibana] MARK_VERSION_INDEX_READY -> DONE. took: 51ms.
[2022-08-24T16:48:57.089+00:00][INFO ][savedobjects-service] [.kibana] Migration completed after 231ms
[2022-08-24T16:48:57.093+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2022-08-24T16:48:57.094+00:00][INFO ][plugins-system.standard] Starting [118] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,share,embeddable,uiActionsEnhanced,screenshotMode,banners,newsfeed,fieldFormats,expressions,eventAnnotation,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,lists,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,telemetry,licenseManagement,kibanaUsageCollection,eventLog,actions,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,aiops,unifiedSearch,savedObjects,triggersActionsUi,transform,stackAlerts,ruleRegistry,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,sharedUX,discover,lens,maps,dataVisualizer,ml,cases,timelines,sessionView,observability,fleet,synthetics,osquery,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,discoverEnhanced,dataViewManagement]
[2022-08-24T16:48:58.626+00:00][INFO ][plugins.monitoring.monitoring] config sourced from: production cluster
[2022-08-24T16:48:59.817+00:00][INFO ][http.server.Kibana] http server running at http://0.0.0.0:5601
[2022-08-24T16:48:59.903+00:00][INFO ][status] Kibana is now degraded
[2022-08-24T16:49:00.132+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Starting monitoring stats collection
[2022-08-24T16:49:00.133+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2022-08-24T16:49:00.186+00:00][INFO ][plugins.ruleRegistry] Installed common resources shared between all indices
[2022-08-24T16:49:00.186+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.uptime.alerts
[2022-08-24T16:49:00.187+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-security.alerts
[2022-08-24T16:49:00.188+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .preview.alerts-security.alerts
[2022-08-24T16:49:00.188+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.logs.alerts
[2022-08-24T16:49:00.188+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.metrics.alerts
[2022-08-24T16:49:00.188+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.apm.alerts
[2022-08-24T16:49:00.276+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.logs.alerts
[2022-08-24T16:49:00.277+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.metrics.alerts
[2022-08-24T16:49:00.279+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.uptime.alerts
[2022-08-24T16:49:00.279+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-security.alerts
[2022-08-24T16:49:00.280+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.apm.alerts
[2022-08-24T16:49:00.590+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .preview.alerts-security.alerts
[2022-08-24T16:49:01.016+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: scheduled with interval 1h
[2022-08-24T16:49:02.268+00:00][INFO ][plugins.fleet] Fleet setup completed
[2022-08-24T16:49:02.276+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask
[2022-08-24T16:49:06.141+00:00][INFO ][plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1] no endpoint installation found
[2022-08-24T16:49:07.061+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: 1 ML saved object synced
[2022-08-24T16:49:07.215+00:00][INFO ][status] Kibana is now available (was degraded)
[2022-08-24T16:49:07.234+00:00][INFO ][plugins.reporting.store] Creating ILM policy for managing reporting indices: kibana-reporting
[2022-08-24T16:49:12.123+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic)
[2022-08-24T16:49:20.358+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2022-08-24T16:49:20.417+00:00][INFO ][plugins.fleet] Fleet setup completed
[2022-08-24T17:04:10.357+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic)
[2022-08-24T17:04:16.134+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2022-08-24T17:04:16.167+00:00][INFO ][plugins.fleet] Fleet setup completed

Any additional context:

N/A

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[antoineco]

@marcanpilami the docs mention

The pin is a base64-encoded string of the SHA-256 fingerprint.

However, your value is the raw fingerprint with colons removed (why?), not its base64-encoded value.

---

Regardless, I was having the same issue. While poking around the console I understood the reason.

Below, the first output was created statically via Kibana's configuration. The second output was created via the UI. Notice that both are setting a different field: ca_sha256 vs. ca_trusted_fingerprint.

(Please note that I mixed up values formats in my example, so both a wrong, please ignore.)

The solution for me was to set the output via xpack.fleet.outputs, where all fields are exposed.

Related issue (and linked PR): https://github.com/elastic/kibana/issues/120608

[marcanpilami]

I had removed the columns because that was what the API returned when inputing the hash through the UI.

I can confirm that setting the output with ca_trusted_fingerprint works perfectly.

So thanks a lot!