Logstash 401 Unauthorized errors

[impovich]

New installation. I see 401 Unauthorized error in logs. Also, Kibana asks for credentials.

To Reproduce Steps to reproduce the behavior:

1. Install
2. check logs

   logstash    | [WARN ] 2022-04-26 11:28:56.121 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://pfelk_logstash:xxxxxx@es01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://es01:9200/'"}

Screenshots If applicable, add screenshots to help explain your problem.

Operating System (please complete the following information):

• Linux 5.15.0-27-generic x86_64
  PRETTY_NAME="Ubuntu 22.04 LTS"
  NAME="Ubuntu"
  VERSION_ID="22.04"
  VERSION="22.04 (Jammy Jellyfish)"
  VERSION_CODENAME=jammy
  ID=ubuntu
  ID_LIKE=debian
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  UBUNTU_CODENAME=jammy

• Version of Docker (docker version 20.10.12, build 20.10.12-0ubuntu4)
• Version of Docker-Compose (docker-compose version 1.29.2, build unknown)

Elasticsearch, Logstash, Kibana (please complete the following information):

• Version of ELK (ELK_VERSION=8.1.1)

  **Service logs

• docker-compose logs pfelk01

  {"@timestamp":"2022-04-26T11:39:31.318Z", "log.level": "INFO", "message":"Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#3]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"sghCi4faTMq-0HWfuHE6Wg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-docker-cluster"}

• docker-compose logs pfelk02

  {"@timestamp":"2022-04-26T11:27:10.187Z", "log.level": "INFO", "message":"added {{es03}{he_uTsJLRVqVdt94ZJZedQ}{XIGGdlwDTu6hmdcuNqYScQ}{172.18.0.5}{172.18.0.5:9300}{cdfhilmrstw}}, term: 2, version: 30, reason: ApplyCommitRequest{term=2, version=30, sourceNode={es01}{sghCi4faTMq-0HWfuHE6Wg}{29Cab6DxSUeO4rLOk3WPYw}{172.18.0.3}{172.18.0.3:9300}{cdfhilmrstw}{ml.machine_memory=16754929664, xpack.installed=true, ml.max_jvm_size=536870912}}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.ClusterApplierService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"LW-kJ81hQICwVKDyxvp-lA","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:14.199Z", "log.level": "INFO", "message":"license [9d9bf110-6edb-43d5-80ed-ef06331a806b] mode [basic] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.LicenseService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"LW-kJ81hQICwVKDyxvp-lA","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:14.200Z", "log.level": "INFO", "message":"license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"LW-kJ81hQICwVKDyxvp-lA","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:17.829Z", "log.level": "INFO", "message":"retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-8501814693674185395/geoip-databases/LW-kJ81hQICwVKDyxvp-lA/GeoLite2-ASN.mmdb.tmp.gz]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"LW-kJ81hQICwVKDyxvp-lA","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:18.677Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#3]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"LW-kJ81hQICwVKDyxvp-lA","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:34.902Z", "log.level": "INFO", "message":"retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-8501814693674185395/geoip-databases/LW-kJ81hQICwVKDyxvp-lA/GeoLite2-City.mmdb.tmp.gz]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"LW-kJ81hQICwVKDyxvp-lA","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:36.504Z", "log.level": "INFO", "message":"retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-8501814693674185395/geoip-databases/LW-kJ81hQICwVKDyxvp-lA/GeoLite2-Country.mmdb.tmp.gz]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"LW-kJ81hQICwVKDyxvp-lA","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:36.716Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#4]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"LW-kJ81hQICwVKDyxvp-lA","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:36.974Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"LW-kJ81hQICwVKDyxvp-lA","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"es-docker-cluster"}

• docker-compose logs pfelk03

  {"@timestamp":"2022-04-26T11:27:10.767Z", "log.level": "INFO", "message":"publish_address {172.18.0.5:9200}, bound_addresses {0.0.0.0:9200}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.http.AbstractHttpServerTransport","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"he_uTsJLRVqVdt94ZJZedQ","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:10.768Z", "log.level": "INFO", "message":"started", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"he_uTsJLRVqVdt94ZJZedQ","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:14.211Z", "log.level": "INFO", "message":"license [9d9bf110-6edb-43d5-80ed-ef06331a806b] mode [basic] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es03][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.LicenseService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"he_uTsJLRVqVdt94ZJZedQ","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:14.212Z", "log.level": "INFO", "message":"license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es03][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"he_uTsJLRVqVdt94ZJZedQ","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:17.828Z", "log.level": "INFO", "message":"retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-18255990313510465210/geoip-databases/he_uTsJLRVqVdt94ZJZedQ/GeoLite2-ASN.mmdb.tmp.gz]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es03][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"he_uTsJLRVqVdt94ZJZedQ","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:18.665Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es03][generic][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"he_uTsJLRVqVdt94ZJZedQ","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:34.906Z", "log.level": "INFO", "message":"retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-18255990313510465210/geoip-databases/he_uTsJLRVqVdt94ZJZedQ/GeoLite2-City.mmdb.tmp.gz]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es03][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"he_uTsJLRVqVdt94ZJZedQ","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:36.516Z", "log.level": "INFO", "message":"retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-18255990313510465210/geoip-databases/he_uTsJLRVqVdt94ZJZedQ/GeoLite2-Country.mmdb.tmp.gz]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es03][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"he_uTsJLRVqVdt94ZJZedQ","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:36.693Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es03][generic][T#3]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"he_uTsJLRVqVdt94ZJZedQ","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"es-docker-cluster"}
  {"@timestamp":"2022-04-26T11:27:37.062Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es03][generic][T#2]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ynFu1V-JT7S131Cr1K0OSw","elasticsearch.node.id":"he_uTsJLRVqVdt94ZJZedQ","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"es-docker-cluster"}

• docker-compose logs logstash

  [WARN ] 2022-04-26 11:41:31.580 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] licensereader - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@es01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://es01:9200/'"}
  [WARN ] 2022-04-26 11:41:31.996 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://pfelk_logstash:xxxxxx@es01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://es01:9200/'"}
  [WARN ] 2022-04-26 11:41:37.001 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://pfelk_logstash:xxxxxx@es01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://es01:9200/'"}
  [WARN ] 2022-04-26 11:41:42.016 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://pfelk_logstash:xxxxxx@es01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://es01:9200/'"}
  [WARN ] 2022-04-26 11:41:47.021 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://pfelk_logstash:xxxxxx@es01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://es01:9200/'"}

• docker-compose logs kibana

  [2022-04-26T11:26:36.862+00:00][INFO ][plugins-service] Plugin "metricsEntities" is disabled.
  [2022-04-26T11:26:36.978+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
  [2022-04-26T11:26:37.028+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
  [2022-04-26T11:26:37.061+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
  [2022-04-26T11:26:37.269+00:00][INFO ][plugins-system.standard] Setting up [112] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,sharedUX,share,embeddable,uiActionsEnhanced,screenshotMode,screenshotting,banners,telemetry,newsfeed,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,lists,fileUpload,ingestPipelines,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,savedObjectsManagement,console,controls,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,maps,dashboardEnhanced,expressionTagcloud,expressionPie,visTypePie,expressionMetricVis,expressionHeatmap,expressionGauge,dataViewFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,dataViewManagement]
  [2022-04-26T11:26:37.308+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: c40fdeb9-0c5c-4299-b81a-7a6e03fc4c30
  [2022-04-26T11:26:37.556+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
  [2022-04-26T11:26:37.557+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
  [2022-04-26T11:26:37.579+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
  [2022-04-26T11:26:37.583+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
  [2022-04-26T11:26:37.619+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
  [2022-04-26T11:26:37.626+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
  [2022-04-26T11:26:37.636+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
  [2022-04-26T11:26:37.654+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
  [2022-04-26T11:26:37.671+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
  [2022-04-26T11:26:37.714+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
  [2022-04-26T11:26:38.549+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
  [2022-04-26T11:26:42.165+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 172.18.0.3:9200
  [2022-04-26T11:26:44.118+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
  [2022-04-26T11:27:07.675+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
  [2022-04-26T11:27:23.854+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
  [2022-04-26T11:27:23.855+00:00][INFO ][savedobjects-service] Starting saved objects migrations
  [2022-04-26T11:27:24.626+00:00][INFO ][savedobjects-service] [.kibana] INIT -> CREATE_NEW_TARGET. took: 94ms.
  [2022-04-26T11:27:24.673+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 139ms.
  [2022-04-26T11:27:34.506+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 9833ms.
  [2022-04-26T11:27:34.668+00:00][INFO ][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 10042ms.
  [2022-04-26T11:27:34.756+00:00][INFO ][savedobjects-service] [.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 250ms.
  [2022-04-26T11:27:34.756+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 10222ms
  [2022-04-26T11:27:34.825+00:00][INFO ][savedobjects-service] [.kibana] MARK_VERSION_INDEX_READY -> DONE. took: 157ms.
  [2022-04-26T11:27:34.825+00:00][INFO ][savedobjects-service] [.kibana] Migration completed after 10293ms
  [2022-04-26T11:27:35.306+00:00][INFO ][status] Kibana is now unavailable
  [2022-04-26T11:27:35.306+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
  [2022-04-26T11:27:35.307+00:00][INFO ][plugins-system.standard] Starting [112] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,sharedUX,share,embeddable,uiActionsEnhanced,screenshotMode,screenshotting,banners,telemetry,newsfeed,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,lists,fileUpload,ingestPipelines,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,savedObjectsManagement,console,controls,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,maps,dashboardEnhanced,expressionTagcloud,expressionPie,visTypePie,expressionMetricVis,expressionHeatmap,expressionGauge,dataViewFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,dataViewManagement]
  [2022-04-26T11:27:36.864+00:00][INFO ][plugins.fleet] Beginning fleet setup
  [2022-04-26T11:27:36.882+00:00][INFO ][plugins.monitoring.monitoring] config sourced from: production cluster
  [2022-04-26T11:27:37.827+00:00][INFO ][http.server.Kibana] http server running at http://0.0.0.0:5601
  [2022-04-26T11:27:38.412+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Starting monitoring stats collection
  [2022-04-26T11:27:39.366+00:00][INFO ][status] Kibana is now degraded (was unavailable)
  [2022-04-26T11:27:40.399+00:00][INFO ][plugins.ruleRegistry] Installed common resources shared between all indices
  [2022-04-26T11:27:40.399+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.uptime.alerts
  [2022-04-26T11:27:40.400+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-security.alerts
  [2022-04-26T11:27:40.400+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .preview.alerts-security.alerts
  [2022-04-26T11:27:40.400+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.logs.alerts
  [2022-04-26T11:27:40.400+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.metrics.alerts
  [2022-04-26T11:27:40.400+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.apm.alerts
  [2022-04-26T11:27:40.838+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.uptime.alerts
  [2022-04-26T11:27:40.980+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.logs.alerts
  [2022-04-26T11:27:41.107+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.metrics.alerts
  [2022-04-26T11:27:41.436+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-security.alerts
  [2022-04-26T11:27:41.714+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.apm.alerts
  [2022-04-26T11:27:43.429+00:00][INFO ][plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1] no endpoint installation found
  [2022-04-26T11:27:43.484+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .preview.alerts-security.alerts
  [2022-04-26T11:27:45.059+00:00][INFO ][plugins.fleet] Fleet setup completed
  [2022-04-26T11:27:45.103+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask
  [2022-04-26T11:27:47.738+00:00][INFO ][status] Kibana is now available (was degraded)
  [2022-04-26T11:27:47.757+00:00][INFO ][plugins.reporting.store] Creating ILM policy for managing reporting indices: kibana-reporting

Additional context Add any other context about the problem here.

[a3ilson]

@AlexanderHamza

• [ ] Did you set the variables (credentials) within the .env file?
• [ ] Did you set the credentials within line 59 of the 50-outputs.conf file?
• [ ] Did you configure the user defined within line 58 of the 50-outputs.conf file?

[impovich]

@a3ilson 1) yes 2 and 3) they were set by default as:

    user => "pfelk_logstash"
    password => "pf31k-l0g$tas#-p@sSw0Rd"

[a3ilson]

Odd...My setup is the same except I'm running Ubuntu Server v20.04. Let me install a fresh instance of v22.04 and see if I can replicate the issue - might be later on in the week when I have time to get around to it.

[impovich]

@a3ilson

Please take a look at this once you get your hands on the issue:

[a3ilson]

@impovich this issue should be mitigated with the #40 Fix provided by @braunbearded

Please confirm/close

[impovich]

Hi, @a3ilson I will check and provide you with feedback, thank you for the notification.