search

deviantony / docker-elk

The Elastic stack (ELK) powered by Docker and Compose.
MIT License
17.06k stars 6.74k forks source link

Kibana server is not ready yet: security_exception: unable to authenticate user [kibana_system] for REST request #996

Closed wilsonakbar closed 3 months ago

wilsonakbar commented 3 months ago
image image image image

I have a problem, where the Kibana server is not ready yet, even though I have followed the instructions according to the instructions, please help me in deploying elk

Problem description

Extra information

Stack configuration

Docker setup

$ docker version

[OUTPUT HERE]
$ docker-compose version

[OUTPUT HERE]

Container logs

$ docker-compose logs

[OUTPUT HERE]
wilsonakbar commented 3 months ago
image image
antoineco commented 3 months ago

Could you please share the output of docker compose up setup?

[!TIP] You can select the output in your terminal window with your mouse, then press Cmd+c on your Mac's keyboard to copy it to your clipboard. Then Cmd+v to copy it to GitHub. It is much easier to read that way. Thanks!

wilsonakbar commented 3 months ago
image image

docker compose up setup only runs the elk-elasticsearch-1 and elk-setup-1 containers, I don't find the elk-kibana-1 container, what should I do next? to be able to use elk?

antoineco commented 3 months ago

The setup initializes the users and roles required by docker-elk inside Elasticsearch.

The README clearly mentions that the setup needs to be run before the rest of the Elastic components can start.

In your case, it seems like you updated the password of the elastic user without completing the initial setup. Please update your passwords inside the .env file and try again.

wilsonakbar commented 3 months ago
image image
wilsonakbar commented 3 months ago
image

the result is still the same as not being able to run kibana

wilsonakbar commented 3 months ago
image

I didn't change anything you made in .env

antoineco commented 3 months ago

As you can see, the setup didn't complete. It failed with "exit code 28".

Please clear your data with docker compose down -v (notice the -v flag) and try again from the beginning.

wilsonakbar commented 3 months ago

I've started it from the beginning and the results are still the same, when I open Kibana

antoineco commented 3 months ago

In this case I don't know.

It would be fantastic if you could write the logs to a file with docker compose logs > logs.txt, then share that file here. Please wait ~2 min after starting the stack before collecting the logs. (No screenshot please)

wilsonakbar commented 3 months ago

this is docker log kibana

Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see https://www.elastic.co/guide/en/kibana/8.14/production.html#openssl-legacy-provider
{"log.level":"info","@timestamp":"2024-06-13T11:21:57.347Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","agentVersion":"4.5.0","env":{"pid":6,"proctitle":"/usr/share/kibana/bin/../node/bin/node","os":"linux 4.15.0-213-generic","arch":"x64","host":"08d0f8276a2f","timezone":"UTC+00","runtime":"Node.js v20.13.1"},"config":{"active":{"source":"start","value":true},"breakdownMetrics":{"source":"start","value":false},"captureBody":{"source":"start","value":"off","commonName":"capture_body"},"captureHeaders":{"source":"start","value":false},"centralConfig":{"source":"start","value":false},"contextPropagationOnly":{"source":"start","value":true},"environment":{"source":"start","value":"production"},"globalLabels":{"source":"start","value":[["git_rev","3bc2979d1d65982aee7d13ebd65434c3470dc808"]],"sourceValue":{"git_rev":"3bc2979d1d65982aee7d13ebd65434c3470dc808"}},"logLevel":{"source":"default","value":"info","commonName":"log_level"},"metricsInterval":{"source":"start","value":120,"sourceValue":"120s"},"serverUrl":{"source":"start","value":"https://kibana-cloud-apm.apm.us-east-1.aws.found.io/","commonName":"server_url"},"transactionSampleRate":{"source":"start","value":0.1,"commonName":"transaction_sample_rate"},"captureSpanStackTraces":{"source":"start","sourceValue":false},"secretToken":{"source":"start","value":"[REDACTED]","commonName":"secret_token"},"serviceName":{"source":"start","value":"kibana","commonName":"service_name"},"serviceVersion":{"source":"start","value":"8.14.0","commonName":"service_version"}},"activationMethod":"require","message":"Elastic APM Node.js Agent v4.5.0"}
Native global console methods have been overridden in production environment.
[2024-06-13T11:22:01.005+00:00][INFO ][root] Kibana is starting
[2024-06-13T11:22:01.142+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
[2024-06-13T11:22:17.251+00:00][INFO ][plugins-service] The following plugins are disabled: "cloudChat,cloudExperiments,cloudFullStory,profilingDataAccess,profiling,securitySolutionServerless,serverless,serverlessObservability,serverlessSearch".
[2024-06-13T11:22:17.393+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
[2024-06-13T11:22:17.672+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2024-06-13T11:22:17.823+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2024-06-13T11:22:18.369+00:00][INFO ][plugins-system.standard] Setting up [158] plugins: [devTools,translations,share,searchConnectors,screenshotMode,usageCollection,telemetryCollectionManager,telemetryCollectionXpack,taskManager,kibanaUsageCollection,cloud,newsfeed,savedObjectsFinder,noDataPage,monitoringCollection,licensing,mapsEms,globalSearch,globalSearchProviders,features,guidedOnboarding,banners,licenseApiGuard,customBranding,ftrApis,fieldFormats,expressions,screenshotting,esUiShared,customIntegrations,contentManagement,dataViews,home,searchprofiler,painlessLab,management,spaces,security,telemetry,licenseManagement,snapshotRestore,lists,files,encryptedSavedObjects,eventLog,actions,observabilityAIAssistant,notifications,cloudDataMigration,aiAssistantManagementSelection,advancedSettings,grokdebugger,console,searchNotebooks,bfetch,data,savedObjectsTagging,savedObjectsManagement,unifiedSearch,navigation,graph,embeddable,uiActionsEnhanced,savedSearch,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,alerting,fileUpload,ingestPipelines,ecsDataQualityDashboard,dataViewFieldEditor,dataViewManagement,charts,watcher,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,visTypeHeatmap,inputControlVis,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,eventAnnotation,expressionXY,lens,dashboard,triggersActionsUi,transform,stackConnectors,searchPlayground,stackAlerts,ruleRegistry,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,metricsDataAccess,logsShared,aiops,links,discover,reporting,canvas,fleet,osquery,logsExplorer,datasetQuality,cloudSecurityPosture,cloudDefend,observability,slo,observabilityLogsExplorer,observabilityOnboarding,discoverEnhanced,maps,dataVisualizer,ml,uptime,synthetics,observabilityAIAssistantApp,indexManagement,textBasedLanguages,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,observabilityAiAssistantManagement,elasticAssistant,securitySolution,securitySolutionEss,dashboardEnhanced,apmDataAccess,infra,upgradeAssistant,monitoring,logstash,assetManager,apm,ux]
[2024-06-13T11:22:19.128+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 5bb2f18f-f79e-4749-a789-8455cba3a5f0
[2024-06-13T11:22:20.506+00:00][INFO ][custom-branding-service] CustomBrandingService registering plugin: customBranding
[2024-06-13T11:22:22.124+00:00][WARN ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, but is not supported for Linux Ubuntu 20.04 OS. Automatically setting 'xpack.screenshotting.browser.chromium.disableSandbox: true'.
[2024-06-13T11:22:22.850+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2024-06-13T11:22:22.850+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2024-06-13T11:22:22.924+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2024-06-13T11:22:22.925+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2024-06-13T11:22:23.353+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2024-06-13T11:22:23.633+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2024-06-13T11:22:23.950+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
[2024-06-13T11:22:24.921+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2024-06-13T11:22:24.921+00:00][INFO ][plugins.alerting] using indexes and aliases for persisting alerts
[2024-06-13T11:22:29.718+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2024-06-13T11:22:31.311+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
[2024-06-13T11:22:38.449+00:00][INFO ][plugins.securitySolution.endpoint:user-artifact-packager:1.0.0] Registering endpoint:user-artifact-packager task with timeout of [20m], interval of [60s] and policy update batch size of [25]
[2024-06-13T11:22:38.450+00:00][INFO ][plugins.securitySolution.endpoint:complete-external-response-actions] Registering task [endpoint:complete-external-response-actions] with timeout of [5m] and run interval of [60s]
[2024-06-13T11:22:39.260+00:00][INFO ][plugins.assetManager] Server is NOT enabled
[2024-06-13T11:22:40.061+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception
    Root causes:
        security_exception: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
[2024-06-13T11:22:42.049+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell
antoineco commented 3 months ago

All the logs in a file please, I shared the command just above. To understand why Kibana cannot authenticate, or why the setup container fails, I need to see the full log output of Elasticsearch, not of Kibana.

wilsonakbar commented 3 months ago

logs.txt

wilsonakbar commented 3 months ago
image

is this normal? after just running the docker compose up setup command, I ran docker compose up, then the setup container Exited

antoineco commented 3 months ago

Thanks! Now I have all the information I need.

Your Elasticsearch switched to read-only because the disk space is insufficient:

{"@timestamp":"2024-06-13T06:51:40.672Z", "log.level": "WARN", "message":"flood stage disk watermark [95%] exceeded

Please see my comment at https://github.com/deviantony/docker-elk/issues/863#issuecomment-1567966149 for the resolution.

tl;dr:

curl -X PUT -u elastic:changeme -H 'Content-Type: application/json' \
    'http://localhost:9200/_cluster/settings?pretty' \
    -d '{"persistent":{"cluster.routing.allocation.disk.threshold_enabled":false}}'

Ref.

wilsonakbar commented 2 months ago
image

Thank you very much sir