elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.75k stars 24.68k forks source link

Failed to retrieve password hash for reserved user kibana_system #110744

Open mireiambm opened 2 months ago

mireiambm commented 2 months ago

Elasticsearch Version

8.14.2

Installed Plugins

No response

Java Version

11.0.23

OS Version

ubuntu 22.04

Problem Description

From one day to the next, my Elasticsearch stopped working without me doing anything, and this error appears: [2024-07-11T09:22:37,457][ERROR][o.e.x.s.a.e.ReservedRealm] [comexicloud-prod-1] failed to retrieve password hash for reserved user [kibana_system]

Steps to Reproduce

What I did was change the password for kibana_system, update it in the kibana.yml file, restart Elasticsearch, and it still doesn't work. The cluster status appears correct, and the password verification for kibana_system also checks out fine. I don't know what else I can do.

Logs (if relevant)

ubuntu@ip-10-100-100-11:~$ sudo cat /var/log/elasticsearch/comexicloud-prod.log | grep ERROR [2024-07-11T08:09:57,935][ERROR][o.e.x.s.a.e.ReservedRealm] [comexicloud-prod-1] failed to retrieve password hash for reserved user [kibana_system] [2024-07-11T08:09:58,674][ERROR][o.e.x.s.a.e.ReservedRealm] [comexicloud-prod-1] failed to retrieve password hash for reserved user [kibana_system] [2024-07-11T08:09:58,889][ERROR][o.e.x.s.a.e.ReservedRealm] [comexicloud-prod-1] failed to retrieve password hash for reserved user [kibana_system] [2024-07-11T09:22:35,918][ERROR][o.e.x.s.a.e.ReservedRealm] [comexicloud-prod-1] failed to retrieve password hash for reserved user [kibana_system] [2024-07-11T09:22:36,418][ERROR][o.e.x.s.a.e.ReservedRealm] [comexicloud-prod-1] failed to retrieve password hash for reserved user [kibana_system] [2024-07-11T09:22:36,918][ERROR][o.e.x.s.a.e.ReservedRealm] [comexicloud-prod-1] failed to retrieve password hash for reserved user [kibana_system] [2024-07-11T09:22:36,968][ERROR][o.e.x.s.a.e.ReservedRealm] [comexicloud-prod-1] failed to retrieve password hash for reserved user [kibana_system] [2024-07-11T09:22:37,457][ERROR][o.e.x.s.a.e.ReservedRealm] [comexicloud-prod-1] failed to retrieve password hash for reserved user [kibana_system]

mireiambm commented 2 months ago

They know what might be happening; everything seems fine.

Enter host password for user 'elastic':
{"cluster_name":"comexicloud-prod","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":179,"active_shards":179,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}u

index       shard prirep state   docs   store dataset ip            node
.security-7 0     p      STARTED  190 489.8kb 489.8kb 10.100.100.11 comexicloud-prod-1

I have changed and configured the new password for kibana_system, the cluster is green, all shards are started. What else can I look at?