Closed Mugibor closed 1 year ago
@Mugibor did you set a password for the required users? If not, please do so and run the setup
container again according to the instructions from the README.
https://github.com/deviantony/docker-elk/blob/be973ec7397e7524579f74194a664811c3d7e2d9/.env#L24-L42
@antoineco Yeah, everything is up to date. Every roles are created, every internal users are created in kibana.
I tried to reproduce the issue, but I'm not running into this error. The error also doesn't show in our automated tests.
With the limited amount of information and logs available in the description I'm not able to provide any further advice unfortunately. Probably Kibana's logs will contain more context.
One thing is for sure: the passwords in the .env
file most likely don't match the ones currently stored in Elasticsearch. If passwords were (re)set while Metricbeat was running, Metricbeat needs to be restarted to pick up the change.
I attached the logs. It was a clean build with the docker-compose command what I attached into the issue description.
Setup log: [+] Waiting for availability of Elasticsearch. This can take several minutes. ⠿ Elasticsearch is running [+] Waiting for initialization of built-in users ⠿ Built-in users were initialized [+] Role 'heartbeat_writer' ⠿ Creating/updating [+] Role 'metricbeat_writer' ⠿ Creating/updating [+] Role 'filebeat_writer' ⠿ Creating/updating [+] Role 'logstash_writer' ⠿ Creating/updating [+] User 'filebeat_internal' ⠿ User does not exist, creating [+] User 'kibana_system' ⠿ User exists, setting password [+] User 'logstash_internal' ⠿ User does not exist, creating [+] User 'heartbeat_internal' ⠿ User does not exist, creating [+] User 'metricbeat_internal' ⠿ User does not exist, creating [+] User 'monitoring_internal' ⠿ User does not exist, creating [+] User 'beats_system' ⠿ User exists, setting password
Metricbeat log: {"log.level":"info","@timestamp":"2022-12-22T13:54:50.920Z","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-12-22T13:54:50.932Z","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: 7ff3f668-5284-4588-9c94-49b469fc8f34","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2022-12-22T13:54:50.941Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-12-22T13:54:50.956Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":63},"message":"Starting stats endpoint","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-12-22T13:54:50.958Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":65},"message":"Metrics endpoint listening on: [::]:5066 (configured: 0.0.0.0)","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-12-22T13:54:50.957Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":102},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-12-22T13:54:50.959Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-12-22T13:54:50.961Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1082},"message":"Beat info","service.name":"metricbeat","system_info":{"beat":{"path":{"config":"/usr/share/metricbeat","data":"/usr/share/metricbeat/data","home":"/usr/share/metricbeat","logs":"/usr/share/metricbeat/logs"},"type":"metricbeat","uuid":"7ff3f668-5284-4588-9c94-49b469fc8f34"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2022-12-22T13:54:50.962Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1091},"message":"Build info","service.name":"metricbeat","system_info":{"build":{"commit":"1ebd0940bd56943642ea8d63d1fe8227f86e7435","libbeat":"8.5.2","time":"2022-11-15T20:38:34.000Z","version":"8.5.2"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2022-12-22T13:54:50.962Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1094},"message":"Go runtime info","service.name":"metricbeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":12,"version":"go1.18.7"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2022-12-22T13:54:50.966Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1098},"message":"Host info","service.name":"metricbeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-12-22T10:27:53Z","containerized":true,"name":"6afe24dc19aa","ip":["127.0.0.1/8","172.20.0.4/16"],"kernel_version":"5.10.16.3-microsoft-standard-WSL2","mac":["02:42:ac:14:00:04"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.5 LTS (Focal Fossa)","major":20,"minor":4,"patch":5,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2022-12-22T13:54:50.968Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1127},"message":"Process info","service.name":"metricbeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/metricbeat","exe":"/usr/share/metricbeat/metricbeat","name":"metricbeat","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-12-22T13:54:49.290Z"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2022-12-22T13:54:50.968Z","log.origin":{"file.name":"instance/beat.go","file.line":294},"message":"Setup Beat: metricbeat; Version: 8.5.2","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2022-12-22T13:54:50.985Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-12-22T13:54:50.990Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://elasticsearch:9200","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-12-22T13:54:50.996Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: metricbeat","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-12-22T13:54:51.146Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":67},"message":"Stats endpoint ([::]:5066) finished: accept tcp [::]:5066: use of closed network connection","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-12-22T13:54:51.146Z","log.origin":{"file.name":"instance/beat.go","file.line":427},"message":"metricbeat stopped.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2022-12-22T13:54:51.146Z","log.origin":{"file.name":"instance/beat.go","file.line":1057},"message":"Exiting: 4 errors: error making http request: Get \"http://kibana:5601/api/status\": dial tcp 172.20.0.5:5601: connect: connection refused; error making http request: Get \"http://kibana:5601/api/status\": dial tcp 172.20.0.5:5601: connect: connection refused; error making http request: Get \"http://kibana:5601/api/status\": dial tcp 172.20.0.5:5601: connect: connection refused; error making http request: Get \"http://kibana:5601/api/status\": dial tcp 172.20.0.5:5601: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"} Exiting: 4 errors: error making http request: Get "http://kibana:5601/api/status": dial tcp 172.20.0.5:5601: connect: connection refused; error making http request: Get "http://kibana:5601/api/status": dial tcp 172.20.0.5:5601: connect: connection refused; error making http request: Get "http://kibana:5601/api/status": dial tcp 172.20.0.5:5601: connect: connection refused; error making http request: Get "http://kibana:5601/api/status": dial tcp 172.20.0.5:5601: connect: connection refused
Kibana log: [2022-12-22T13:53:53.819+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui] [2022-12-22T13:54:05.735+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled. [2022-12-22T13:54:05.745+00:00][INFO ][plugins-service] Plugin "profiling" is disabled. [2022-12-22T13:54:05.817+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601 [2022-12-22T13:54:54.026+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui] [2022-12-22T13:55:11.179+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled. [2022-12-22T13:55:11.194+00:00][INFO ][plugins-service] Plugin "profiling" is disabled. [2022-12-22T13:55:11.359+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601 [2022-12-22T13:55:11.448+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup] [2022-12-22T13:55:11.904+00:00][INFO ][plugins-system.standard] Setting up [125] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,guidedOnboarding,fieldFormats,expressions,dataViews,embeddable,uiActionsEnhanced,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,lists,files,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,telemetry,licenseManagement,eventLog,actions,stackConnectors,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,aiops,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,eventAnnotation,dataViewFieldEditor,triggersActionsUi,transform,stackAlerts,ruleRegistry,discover,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,observability,osquery,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement] [2022-12-22T13:55:11.961+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: a66a96d1-50b7-4432-bf22-35a0cd9a26df [2022-12-22T13:55:12.081+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended. [2022-12-22T13:55:12.126+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended. [2022-12-22T13:55:12.140+00:00][INFO ][plugins.encryptedSavedObjects] Hashed 'xpack.encryptedSavedObjects.encryptionKey' for this instance: 3frBilz1iPdwINfR2SoDPqZVU57g5zUje50W+oy94FY= [2022-12-22T13:55:12.396+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices [2022-12-22T13:55:12.460+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task] [2022-12-22T13:55:13.746+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox. [2022-12-22T13:55:14.204+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes..version%2Cnodes..http.publish_address%2Cnodes.*.ip] [2022-12-22T13:55:15.102+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell [2022-12-22T13:55:19.037+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations... [2022-12-22T13:55:19.038+00:00][INFO ][savedobjects-service] Starting saved objects migrations [2022-12-22T13:55:19.137+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 64ms. [2022-12-22T13:55:19.148+00:00][INFO ][savedobjects-service] [.kibana] INIT -> CREATE_NEW_TARGET. took: 80ms. [2022-12-22T13:55:19.747+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 611ms. [2022-12-22T13:55:19.922+00:00][INFO ][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 774ms. [2022-12-22T13:55:20.135+00:00][INFO ][savedobjects-service] [.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 388ms. [2022-12-22T13:55:20.136+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 1064ms [2022-12-22T13:55:20.349+00:00][INFO ][savedobjects-service] [.kibana] MARK_VERSION_INDEX_READY -> DONE. took: 427ms. [2022-12-22T13:55:20.351+00:00][INFO ][savedobjects-service] [.kibana] Migration completed after 1283ms [2022-12-22T13:55:20.364+00:00][INFO ][plugins-system.preboot] Stopping all plugins. [2022-12-22T13:55:20.368+00:00][INFO ][plugins-system.standard] Starting [125] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,guidedOnboarding,fieldFormats,expressions,dataViews,embeddable,uiActionsEnhanced,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,lists,files,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,telemetry,licenseManagement,eventLog,actions,stackConnectors,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,aiops,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,eventAnnotation,dataViewFieldEditor,triggersActionsUi,transform,stackAlerts,ruleRegistry,discover,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,observability,osquery,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement] [2022-12-22T13:55:22.773+00:00][INFO ][plugins.monitoring.monitoring] config sourced from: production cluster [2022-12-22T13:55:24.079+00:00][INFO ][http.server.Kibana] http server running at http://0.0.0.0:5601 [2022-12-22T13:55:24.300+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Starting monitoring stats collection [2022-12-22T13:55:24.301+00:00][INFO ][plugins.fleet] Beginning fleet setup [2022-12-22T13:55:24.321+00:00][INFO ][status] Kibana is now degraded [2022-12-22T13:55:24.822+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: scheduled with interval 1h [2022-12-22T13:55:25.242+00:00][INFO ][plugins.ruleRegistry] Installed common resources shared between all indices [2022-12-22T13:55:25.243+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.uptime.alerts [2022-12-22T13:55:25.244+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-security.alerts [2022-12-22T13:55:25.244+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .preview.alerts-security.alerts [2022-12-22T13:55:25.245+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.logs.alerts [2022-12-22T13:55:25.246+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.metrics.alerts [2022-12-22T13:55:25.246+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.apm.alerts [2022-12-22T13:55:25.374+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.logs.alerts [2022-12-22T13:55:25.375+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.uptime.alerts [2022-12-22T13:55:25.376+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-security.alerts [2022-12-22T13:55:25.378+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.apm.alerts [2022-12-22T13:55:25.379+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.metrics.alerts [2022-12-22T13:55:25.656+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .preview.alerts-security.alerts [2022-12-22T13:55:30.840+00:00][INFO ][plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1] no endpoint installation found [2022-12-22T13:55:31.626+00:00][INFO ][status] Kibana is now available (was degraded) [2022-12-22T13:55:31.644+00:00][INFO ][plugins.reporting.store] Creating ILM policy for managing reporting indices: kibana-reporting [2022-12-22T13:55:31.834+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: 1 ML saved object synced [2022-12-22T13:55:48.563+00:00][INFO ][plugins.fleet] Fleet setup completed [2022-12-22T13:55:48.573+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask [2022-12-22T13:55:51.118+00:00][INFO ][plugins.synthetics] Installed synthetics index templates
It looks like a timing issue in the logs above. Kibana starts serving roughly a minute after Metricbeat. The errors are thrown during that time frame, and then they disappear (last error is at 13:54:51, after that nothing).
Okay, it was a fail from my side. I thought it stopped out somehow.
No worries, glad everything works in the end!
Problem description
I tried to run my elastic stack with the metricbeat extension. After everything was set up, the metricbeat sent an error that cannot reach the kibana endpoint. I also tried to run your configuration without any changes. The problem is also exists. What should I change in the default settings?
Extra information
Stack configuration
docker-compose -f docker-compose.yml -f extensions/metricbeat/metricbeat-compose.yml up
Docker setup
Container logs
elastic-stack-metricbeat-1 | {"log.level":"info","@timestamp":"2022-12-21T12:39:14.736Z","log.origin":{"file.name":"instance/beat.go","file.line":427},"message":"metricbeat stopped.","service.name":"metricbeat","ecs.version":"1.6.0"} elastic-stack-metricbeat-1 | {"log.level":"error","@timestamp":"2022-12-21T12:39:14.736Z","log.origin":{"file.name":"instance/beat.go","file.line":1057},"message":"Exiting: 4 errors: error making http request: Get \"http://kibana:5601/api/status\": dial tcp 172.23.0.5:5601: connect: connection refused; error making http request: Get \"http://kibana:5601/api/status\": dial tcp 172.23.0.5:5601: connect: connection refused; error making http request: Get \"http://kibana:5601/api/status\": dial tcp 172.23.0.5:5601: connect: connection refused; error making http request: Get \"http://kibana:5601/api/status\": dial tcp 172.23.0.5:5601: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"} elastic-stack-metricbeat-1 | Exiting: 4 errors: error making http request: Get "http://kibana:5601/api/status": dial tcp 172.23.0.5:5601: connect: connection refused; error making http request: Get "http://kibana:5601/api/status": dial tcp 172.23.0.5:5601: connect: connection refused; error making http request: Get "http://kibana:5601/api/status": dial tcp 172.23.0.5:5601: connect: connection refused; error making http request: Get "http://kibana:5601/api/status": dial tcp 172.23.0.5:5601: connect: connection refused