Cannot reset the password of the logstash_internal user

[AAAbou]

cannot rest the password of the logstach_insternal user and is causing connection issues

[2022-06-20T11:03:23,692][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
[2022-06-20T11:03:28,699][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
[2022-06-20T11:03:33,705][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
[2022-06-20T11:03:38,713][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
[2022-06-20T11:03:43,720][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}

[antoineco]

@AAAbou what is the output of docker-compose up setup?

[AAAbou]

@antoineco

$ docker-compose up setup
Creating network "docker-elk-test_elk" with driver "bridge"
Creating volume "docker-elk-test_setup" with default driver
Creating volume "docker-elk-test_elasticsearch" with default driver
Building setup
Sending build context to Docker daemon  11.78kB
Step 1/7 : ARG ELASTIC_VERSION
Step 2/7 : FROM docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}
 ---> 59306705ed62
Step 3/7 : USER root
 ---> Using cache
 ---> 6853629bb04c
Step 4/7 : COPY . /
 ---> Using cache
 ---> ed03701ba563
Step 5/7 : RUN set -eux;        mkdir /state;   chown elasticsearch /state;     chmod +x /entrypoint.sh
 ---> Using cache
 ---> 30301188d80f
Step 6/7 : USER elasticsearch:root
 ---> Using cache
 ---> f18126828831
Step 7/7 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> 524502d4bf3f
Successfully built 524502d4bf3f
Successfully tagged docker-elk-test_setup:latest
WARNING: Image for service setup was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating docker-elk-test_setup_1 ... done
Attaching to docker-elk-test_setup_1
setup_1          | -------- Mon Jun 20 12:53:35 UTC 2022 --------
setup_1          | [+] Waiting for availability of Elasticsearch

[antoineco]

Elasticsearch is not up yet:

setup_1 | [+] Waiting for availability of Elasticsearch

Give it a few minutes.

It the setup does not complete, your host might have issues running Elasticsearch. You may find out the reason using docker-compose logs elasticsearch.

[AAAbou]

Ahhh, i Only ran the setup part, didnt include the rest; Generally, i ran the entire thing before and elastic search and Kibana worked fine, changed their passwords and reached their UI. What i noticed was that i was able to change the passwords for the 'built-in' users but not the custom one 'logstash_internal'

[AAAbou]

This below is the output for the entire run,

$ docker-compose up
WARNING: Found orphan containers (docker-elk_apm-server_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Starting docker-elk_elasticsearch_1 ... done
Starting docker-elk_setup_1         ... done
Starting docker-elk_kibana_1        ... done
Recreating docker-elk_logstash_1    ... done
Attaching to docker-elk_setup_1, docker-elk_elasticsearch_1, docker-elk_kibana_1, docker-elk_logstash_1
logstash_1       | Using bundled JDK: /usr/share/logstash/jdk
setup_1          | -------- Mon Jun 20 13:04:26 UTC 2022 --------
setup_1          | [+] Waiting for availability of Elasticsearch
logstash_1       | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
elasticsearch_1  | {"@timestamp":"2022-06-20T13:04:44.058Z", "log.level": "INFO", "message":"version[8.2.3], pid[7], build[default/docker/9905bfb62a3f0b044948376b4f607f70a8a151b4/2022-06-08T22:21:36.455508792Z], OS[Linux/4.18.0-372.9.1.el8.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/18.0.1.1/18.0.1.1+2-6]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"096dea45070b","elasticsearch.cluster.name":"docker-cluster"}
elasticsearch_1  | {"@timestamp":"2022-06-20T13:04:44.074Z", "log.level": "INFO", "message":"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"096dea45070b","elasticsearch.cluster.name":"docker-cluster"}

Edit: ElasticSearch Logs

{"@timestamp":"2022-06-20T13:05:28.088Z", "log.level": "INFO",  "current.health":"GREEN","message":"Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.fleet-policies-7][0]]]).","previous.health":"YELLOW","reason":"shards started [[.fleet-policies-7][0]]"

[antoineco]

@AAAbou that's why I asked about the setup service, because the logstash_internal user is created using that service.

What I'm missing is the entire output of docker-compose up setup, not just the first line. After 5 min, if it hasn't managed to connect to Elasticsearch, it will print a message, but you have to be patient and wait for the error to occur. Based on that error I can maybe think about a reason why your stack isn't working properly.

[zakrian07]

@antoineco thanks for helping me everything working like a charm I have just a question I want to put my own generated password because I have 300 hosts and where I need to run agents for ship logs so I want to keep my own generated password. is this possible? right now the command generating password

[AAAbou]

Running on already running elk:

$ docker-compose up setup
WARNING: Found orphan containers (docker-elk_apm-server_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Starting docker-elk_setup_1 ... done
Attaching to docker-elk_setup_1
setup_1          | -------- Tue Jun 21 10:57:35 UTC 2022 --------
setup_1          | [+] Waiting for availability of Elasticsearch
setup_1          | 
setup_1          | 
docker-elk_setup_1 exited with code 1

Running on new elk:

$ docker-compose up setup
Creating network "docker-elk-test_elk" with driver "bridge"
Creating volume "docker-elk-test_setup" with default driver
Creating volume "docker-elk-test_elasticsearch" with default driver
Building setup
Sending build context to Docker daemon  11.78kB
Step 1/7 : ARG ELASTIC_VERSION
Step 2/7 : FROM docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}
 ---> 59306705ed62
Step 3/7 : USER root
 ---> Using cache
 ---> 88bcbd3cae3a
Step 4/7 : COPY . /
 ---> Using cache
 ---> fc4b52e6cfb1
Step 5/7 : RUN set -eux;        mkdir /state;   chown elasticsearch /state;     chmod +x /entrypoint.sh
 ---> Using cache
 ---> 184bdb0da1b4
Step 6/7 : USER elasticsearch:root
 ---> Using cache
 ---> 2ba01c24718a
Step 7/7 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> 7e3fa4b52c59
Successfully built 7e3fa4b52c59
Successfully tagged docker-elk-test_setup:latest
WARNING: Image for service setup was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating docker-elk-test_setup_1 ... done
Attaching to docker-elk-test_setup_1
setup_1          | -------- Tue Jun 21 11:17:40 UTC 2022 --------
setup_1          | [+] Waiting for availability of Elasticsearch
setup_1          | 
setup_1          | 
docker-elk-test_setup_1 exited with code 1

[antoineco]

@AAAbou interesting. It seems like the setup container is unable to communicate with Elasticsearch.

Is Elasticsearch already running?

If not, could you please try this instead?

docker-compose up -d elasticsearch
docker-compose up setup

[AAAbou]

@antoineco You're right...its already running and I can reach it fine via the port and container says its fine. Both Kibana and Elastic are working. Logstash is working but can't reach the elasticsearch (which explains the '401' error).

docker-compose up elasticsearch -d
ERROR: No such service: -d

docker-compose up setup
Starting docker-elk_setup_1 ... done
Attaching to docker-elk_setup_1
setup_1          | -------- Tue Jun 21 13:02:58 UTC 2022 --------
setup_1          | [+] Waiting for availability of Elasticsearch
setup_1          | 
setup_1          | 
docker-elk_setup_1 exited with code 1

[2022-06-21T13:11:03,919][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error
 {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}

[antoineco]

Sorry, the -d must be right after up, not after elasticsearch #oops

OK that's encouraging, at least Logstash is able to communicate with Elasticsearch, only the setup container isn't. Now we need to figure out why.

I just pushed a change to the main branch that causes the setup container to print an error message with some context in case of failure to connect to Elasticsearch. Would you mind trying with that revision and sharing the output again?

Don't forget to rebuild the setup container with docker-compose build first.

[AAAbou]

Sorry for the late reply.

> Executing task: docker logs --tail 1000 -f dffd3057e4bfb061266917ddbdc403d80ad97da8eae35a826ec3fd231b777516 <

-------- Tue Jun 21 18:42:48 UTC 2022 --------
[+] Waiting for availability of Elasticsearch. This can take several minutes.
   ⠍ Connection to Elasticsearch failed. Exit code: 52

Terminal will be reused by tasks, press any key to close it.

[antoineco]

This error code indicates that the server didn't respond. It often occurs if there is a firewall between the two containers.

Do you have any firewall in place that might block communications between containers, even if they are attached to the same network?

[AAAbou]

all containers are inside the same docker created network; if there was an issue concerning that, then Kibana wouldn't be able to reach out to elastic and use the inbuilt sample data.

[antoineco]

That is true. Likewise for Logstash.

I'm really running out of ideas unfortunately. If all containers are in the same network and there is no firewall between them, I can't think about a reason why the setup container specifically can't communicate with Elasticsearch. I've never encountered such thing before.

[AAAbou]

No problem, maybe a clean install will help, I'm just going to create a new machine and just reinstall everything. Nonetheless, thank you :).

[AAAbou]

hmm

[2022-06-22T08:16:45,316][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}

[AAAbou]

I manually created the logstash internal user on kibana and updated the password in the env file, then did another docker-compose up (401 is gone and authenticates apparently but now a new one occurred)

[2022-06-22T08:19:17,217][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-06-22T08:19:18,406][INFO ][org.reflections.Reflections] Reflections took 151 ms to scan 1 urls, producing 120 keys and 395 values 
[2022-06-22T08:19:19,502][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-06-22T08:19:19,552][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]}
[2022-06-22T08:19:19,981][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_internal:xxxxxx@elasticsearch:9200/]}}
[2022-06-22T08:19:21,943][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/"}
[2022-06-22T08:19:21,973][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.2.3) {:es_version=>8}
[2022-06-22T08:19:21,975][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-06-22T08:19:22,081][INFO ][logstash.outputs.elasticsearch][main] Config is compliant with data streams. `data_stream => auto` resolved to `true`
[2022-06-22T08:19:22,125][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-06-22T08:19:22,164][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-06-22T08:19:22,313][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x25da5abc run>"}
[2022-06-22T08:19:22,469][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"ecs-logstash"}

but had another error after....

[2022-06-22T08:19:22,164][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-06-22T08:19:22,313][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x25da5abc run>"}
[2022-06-22T08:19:22,469][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"ecs-logstash"}
[2022-06-22T08:19:22,775][ERROR][logstash.outputs.elasticsearch][main] Failed to install template {:message=>"Got response code '403' contacting Elasticsearch at URL 'http://elasticsearch:9200/_index_template/ecs-logstash'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:398:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:310:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:408:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:85:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch.rb:494:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch.rb:318:in `finish_register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch.rb:283:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:149:in `block in after_successful_connection'"]}

Which made subsequent issues

[2022-06-22T08:22:15,340][INFO ][org.logstash.beats.BeatsHandler][main][27dfbb7f38dfc5bffbd7ce11deb78daf39a131fcbb4eceab2c6a1d7100069751] [local: 192.168.0.2:5044, remote: 10.xx.xx.xx:12030] Handling exception: io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69 (caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69)
[2022-06-22T08:22:15,344][WARN ][io.netty.channel.DefaultChannelPipeline][main][27dfbb7f38dfc5bffbd7ce11deb78daf39a131fcbb4eceab2c6a1d7100069751] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

[antoineco]

The setup service also creates a role called logstash_writer, and grants it to the logstash_internal user.

If you take the manual route, you also have to perform those steps for Logstash to work.

[AAAbou]

Thank You!

[antoineco]

This command should allow you to set up everything correctly:

env LOGSTASH_INTERNAL_PASSWORD=<password> ELASTICSEARCH_HOST=localhost setup/entrypoint.sh

[antoineco]

@AAAbou do you still need any assistance with this issue? Did you eventually succeed in creating the user and role?