search

deviantony / docker-elk

The Elastic stack (ELK) powered by Docker and Compose.
MIT License
17.15k stars 6.76k forks source link

Failed to authenticate user [kibana_system]: at least one primary shard for the index [.security-7] is unavailable #795

Closed jmvermeulen closed 1 year ago

jmvermeulen commented 1 year ago

Hello,

A fresh docker compose up gives authentication errors. Tried to build first, removed all docker cache etc. Nothing works.

Do you have a suggestion? 

{"@timestamp":"2022-11-30T09:20:43.231Z", "log.level": "INFO", "message":"Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][transport_worker][T#3]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","trace.id":"1e4f9aea87762a8b8edfe27e2d162c75","elasticsearch.cluster.uuid":"twd3RorFQG2-fLtGSjknkw","elasticsearch.node.id":"E_pLHGQWS9azm-2YL56OiA","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster"}
docker-elk-elasticsearch-1  | {"@timestamp":"2022-11-30T09:20:44.238Z", "log.level":"ERROR", "message":"failed to retrieve password hash for reserved user [kibana_system]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][transport_worker][T#3]","log.logger":"org.elasticsearch.xpack.security.authc.esnative.ReservedRealm","trace.id":"1e4f9aea87762a8b8edfe27e2d162c75","elasticsearch.cluster.uuid":"twd3RorFQG2-fLtGSjknkw","elasticsearch.node.id":"E_pLHGQWS9azm-2YL56OiA","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster","error.type":"org.elasticsearch.action.UnavailableShardsException","error.message":"at least one primary shard for the index [.security-7] is unavailable","error.stack_trace":"org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:138)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:603)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:275)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:135)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:147)\n\tat org.elasticsearch.xcore@8.5.2/org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:234)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:83)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:180)\n\tat org.elasticsearch.xcore@8.5.2/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:158)\n\tat org.elasticsearch.xcore@8.5.2/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:158)\n\tat org.elasticsearch.xcore@8.5.2/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:158)\n\tat org.elasticsearch.xcore@8.5.2/org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:136)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:95)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127)\n\tat org.elasticsearch.security@8.5.2/org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:100)\n\tat org.elasticsearch.server@8.5.2/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:396)\n\tat org.elasticsearch.server@8.5.2/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:530)\n\tat org.elasticsearch.server@8.5.2/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:311)\n\tat org.elasticsearch.server@8.5.2/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:379)\n\tat org.elasticsearch.server@8.5.2/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:460)\n\tat org.elasticsearch.server@8.5.2/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:353)\n\tat org.elasticsearch.transport.netty4@8.5.2/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128)\n\tat org.elasticsearch.transport.netty4@8.5.2/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.77.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.codec@4.1.77.Final/io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.77.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.77.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.77.Final/io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)\n\tat io.netty.codec@4.1.77.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler@4.1.77.Final/io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.77.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)\n\tat io.netty.transport@4.1.77.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat io.netty.common@4.1.77.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)\n\tat io.netty.common@4.1.77.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\n"}

"org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable

antoineco commented 1 year ago

First of all, a friendly reminder that we ask people to fill in the issue template for good reasons. Your issue report is omitting almost all requested logs, and one error in isolation doesn't provide enough context for solving most issues.

If this is indeed an authentication issue, please share the output of docker compose logs setup (example) so we can at least confirm that the kibana_system user was initialized correctly.

If not, the root cause is hidden somewhere else in the logs.

antoineco commented 1 year ago

One wild guess—without seeing the logs—maybe Elasticsearch is throwing the following error (sanitized and pretty printed for readability):

{
  "log.level": "WARN",
  "message": "high disk watermark [90%] exceeded on [7dX6kMvsRvmjXZVi0ugi_w][ecfad7cdffbc][/usr/share/elasticsearch/data] free: 4.6gb[9.9%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete",
  "process.thread.name": "elasticsearch[ecfad7cdffbc][masterService#updateTask][T#1]",
  "log.logger": "org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor"
}

This is an indicator that your disk is dangerously full (> 90%), in which case Elasticsearch's default behaviour is to switch to read-only mode.

If you see such error in your logs, consider setting watermarks for disk usage higher than the default values in your Elasticsearch config:

cluster.routing.allocation.disk.watermark.high: 95%
cluster.routing.allocation.disk.watermark.flood_stage: 97%

Cross-referencing https://github.com/deviantony/docker-elk/issues/729#issuecomment-1170571124

jmvermeulen commented 1 year ago

Indeed it was the disk space..! Thank you.

mnuriyumusak commented 1 year ago

One wild guess—without seeing the logs—maybe Elasticsearch is throwing the following error (sanitized and pretty printed for readability):

{
  "log.level": "WARN",
  "message": "high disk watermark [90%] exceeded on [7dX6kMvsRvmjXZVi0ugi_w][ecfad7cdffbc][/usr/share/elasticsearch/data] free: 4.6gb[9.9%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete",
  "process.thread.name": "elasticsearch[ecfad7cdffbc][masterService#updateTask][T#1]",
  "log.logger": "org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor"
}

This is an indicator that your disk is dangerously full (> 90%), in which case Elasticsearch's default behaviour is to switch to read-only mode.

If you see such error in your logs, consider setting watermarks for disk usage higher than the default values in your Elasticsearch config:

cluster.routing.allocation.disk.watermark.high: 95%
cluster.routing.allocation.disk.watermark.flood_stage: 97%

Cross-referencing #729 (comment)

It actually worked. I'm suprised, what kind of relations is there between authentication user and disk space. Weird.

antoineco commented 1 year ago

@mnuriyumusak the relation is that docker-elk's setup container needs to write into the .security-7 index to enable the kibana_system user (disabled by default), but this index is read-only due to the high disk watermark. Meanwhile Kibana keeps trying to authenticate, so you see these authentication errors on top of the watermark errors.