Kibana server is not ready yet. - after password change

[robinpecha]

Hi, thanks for this elk stack. After docker-compose up, I can see kibana running and asking for password at http://localhost:5601/. User elastic with changeme password didnt work (As you mentioned: ... from Elastic v8.0.0, it is no longer possible to use elastic user in Kibana ...). So I follow your instruction and generate all 3 passwords, then I have changed them in .env file and rerun $ docker-compose up -d logstash kibana. But after this I cannot even open Kibana login screen - it start loading and then throw: Kibana server is not ready yet.

Last message of Kibana in terminal is: kibana_1 | [2022-06-30T05:57:13.430+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]

And on each attempt to login elastic throws: elasticsearch_1 | {"@timestamp":"2022-06-30T06:04:18.139Z", "log.level": "INFO", "message":"Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[d9e2443d7e4a][system_critical_read][T#3]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","trace.id":"4a625a3ec76bc5289c167bd600e9eb0d","elasticsearch.cluster.uuid":"8Kf4OhkvTcCjhsAisWrpKA","elasticsearch.node.id":"h-bcM0oZRxS3CA8CDPCuEA","elasticsearch.node.name":"d9e2443d7e4a","elasticsearch.cluster.name":"docker-cluster"} Did I follow your documentation wrong?

[robinpecha]

Ok, previous issue here was the same. And I can see it works for you without problem. https://github.com/deviantony/docker-elk/issues/729#issuecomment-1161677730

Only difference I found is that Im runnig it all as sudoer. sudo docker-compose up ...

[robinpecha]

Now I change only elastic password, docker-compose up logstash kibana and now I can login.

[antoineco]

User elastic with changeme password didnt work (As you mentioned: ... from Elastic v8.0.0, it is no longer possible to use elastic user in Kibana ...)

That is not what the warning about the elastic user means. The elastic user can't be used by Kibana itself (the backend) to interact with Elasticsearch. But users can log in with the elastic user through the Kibana frontend.

In your case though, it sounds a lot like you're running a version of Compose below 1.26, which doesn't handle quotes around passwords properly in the .env file.

In the future, please include all the requested information from the issue template in your issue description. There is a reason why we ask for versions of Docker components and logs: it helps us find the root cause of your issue much quicker.

[Mydayyy]

Greetings,

is there a chance that the passwords you generated first were not long enough to meet the requirements? Iirc they need to be atleast 6 characters.

In case you still have the full logfiles, there should be an error printed about the password requirements if it did not meet them.

[antoineco]

Closing due to inactivity, feel free to reopen.

[irrapaj]

I had the exact same problem. It turned out I had way too many images and my system was close to 95% full after the whole stack was brought up.

Removing the containers and reverting the .env file to use the previous 7.17 version would work. The reason I believe it worked is because the image for 7.17.5 is only 610MB as compared to the image for 8.3.2 which is 1.24GB

docker-elk_setup                                latest               97aa0e62b3a8   5 hours ago      1.24GB
docker-elk_elasticsearch                        latest               8ae7ba76ca7f   2 weeks ago      1.24GB
docker.elastic.co/elasticsearch/elasticsearch   8.3.2                e76e95312216   2 weeks ago      1.24GB

docker.elastic.co/elasticsearch/elasticsearch   7.17.5               11df7a62573d   4 weeks ago      610MB

I pruned a lot of dangling images, started the stack from scratch and it worked out of the box like it should have initially. So the fault was mine for not cleaning up my images.

For future reference if someone encounters the same issue, make sure to have enough free disk space on your system.

[robinpecha]

Im sorry for delay, moved to another project, will inspect when I will be back on this. But what realy helps in my case was your suggestion about compose. I have probably on that vm installed docker and docker-compose and there was wrong version. Later i remove docker-compose and create simlink from docker compes plugin to docker-compose and then it works. If I remember it right.