LME changed password for kebana_system - Elasticsearch is not responding

[LeitLei]

Describe the bug

LME changed password for kebana_system during update. Update fails with "Elasticsearch is not responding"

To Reproduce

I want to update from 1.0.0 to 1.3.2 So I followed your instructions here.

After creating the services there is the message "Waiting for Elasticsearch to be ready" followed by "Elasticsearch is not responding after 25 attempts - existing". With sudo docker ps -a I could see that the kibana container is still on "health: starting". So I checked the logs and found:

Unable to retrieve version information from Elasticsearch nodes. security_exception
        Root causes:
                security_exception: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
[2024-01-30T10:44:06.267+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell

I noticed, that the password for kibana_system in docker-compose-stack-live.yml is now different as pre update. Also the password for logstash_writer in logstash.edited.conf.

After that, I followed your instructions here (but _elasticsearch-reset-password -v -u kibana_system -i --url https://localhost:9200_ instead elastic) and update the password as listed in docker-compose-stack-live.yml.

After a restart, all works fine.

But why is LME changing my passwords? I found only this which is similar but not the same in your Changelog. Is the installation finish after changing my password? Or is there anything more to do (except your points listed in your upgrade manual)?

Please complete the following information

Desktop:

• OS: not significant
• Browser: not significant
• Software version: not significant

Server:

• OS: Ubuntu 20.04.06 LTS
• Software Versions:
  • ELK (pre update): 8.7.1
  • Docker: 20.10.23, build 7155243
• The output of these commands:

  free -h
  total        used        free      shared  buff/cache   available
  Mem:           11Gi       7,6Gi       150Mi       4,0Mi       3,8Gi       3,5Gi
  Swap:         4,0Gi       0,0Ki       4,0Gi
  df -h 
  Filesystem                         Size  Used Avail Use% Mounted on
  udev                               5,8G     0  5,8G   0% /dev
  tmpfs                              1,2G  1,2M  1,2G   1% /run
  /dev/mapper/ubuntu--vg-ubuntu--lv  117G   87G   26G  78% /
  tmpfs                              5,9G     0  5,9G   0% /dev/shm
  tmpfs                              5,0M     0  5,0M   0% /run/lock
  tmpfs                              5,9G     0  5,9G   0% /sys/fs/cgroup
  tmpfs                              5,9G     0  5,9G   0% /run/shm
  /dev/loop0                          56M   56M     0 100% /snap/core18/2812
  /dev/loop1                          56M   56M     0 100% /snap/core18/2796
  /dev/loop3                          41M   41M     0 100% /snap/snapd/20671
  /dev/loop4                          92M   92M     0 100% /snap/lxd/23991
  /dev/loop2                          92M   92M     0 100% /snap/lxd/24061
  /dev/loop5                          64M   64M     0 100% /snap/core20/2105
  /dev/loop7                          64M   64M     0 100% /snap/core20/2015
  /dev/loop6                          41M   41M     0 100% /snap/snapd/20290
  /dev/sda2                          974M  231M  677M  26% /boot
  /dev/sda1                          511M  6,1M  505M   2% /boot/efi
  overlay                            117G   87G   26G  78% /var/lib/docker/overlay2/f8704fcb62d15c757d831ed3bcd5f0ae5d9f176466de4cd0a5992ca390ba836e/merged
  overlay                            117G   87G   26G  78% /var/lib/docker/overlay2/c485f9ae57f08ff8d5f64f941f7a7b335801d3bb1a088fb2f9da63ed91281c28/merged
  overlay                            117G   87G   26G  78% /var/lib/docker/overlay2/82ce5636f2695979c551c8a464d20aa6e290e46a617b02c826032e4e90225f30/merged
  tmpfs                              5,9G   12K  5,9G   1% /var/lib/docker/containers/d0afc3888e957af4c96a3ee840a031511e3f1e8380619a40c8a6777ff57315fb/mounts/secrets
  tmpfs                              5,9G   12K  5,9G   1% /var/lib/docker/containers/14fbca8a6cdf95110efddc92c8caedf7f3946aa80697c5304ea688b8a20c64b3/mounts/secrets
  tmpfs                              5,9G   20K  5,9G   1% /var/lib/docker/containers/205952da8b1974bf7737397e629f1a5309f14eed02a29caccea9eac20abaca50/mounts/secrets
  tmpfs                              1,2G     0  1,2G   0% /run/user/1000`
  uname -a 
  Linux log02 5.15.0-1054-azure #62~20.04.1-Ubuntu SMP Wed Jan 17 12:22:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
  lsb_release -a
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 20.04.6 LTS
  Release:        20.04
  Codename:       focal

• Relevant container logs:

  see description

Expected behavior

LME should not change the password during update or should tell the user, what he have to do after the automatic password change. The user needs information, if the Update was successful after changed the password with your code listed in 'troubleshooting'.

Screenshots

Additional context

[llwaterhouse]

Hello, thank you for submitting your ticket.

In order to best respond to your issue, please resubmit this using the Bug template and provide all of the requested information.

Thank you.

[LeitLei]

Hello, thank you for submitting your ticket.

In order to best respond to your issue, please resubmit this using the Bug template and provide all of the requested information.

Thank you.

@llwaterhouse : Updated!

[LeitLei]

@llwaterhouse : Please reopen it. Thanks a lot!

[llwaterhouse]

I have reopened it. Please resubmit with the template filled out so we can best answer. Thank you.

[LeitLei]

Hi @llwaterhouse,

can you tell me meanwhile, if the Update is successful finished after I changed the password with your code listed in 'troubleshooting'?

[llwaterhouse]

Can you log into kibana and are the panels displaying data?

[LeitLei]

@llwaterhouse :

yes, all works fine.

But I also have to set _elasticsearch-reset-password -v -u logstashwriter -i --url https://localhost:9200/