stuck with docker compose up process and unable to access Kibana

capmerah commented 2 years ago

Describe the bug stuck with docker compose up process and unable to access Kibana

To Reproduce after sudo docker-compose up

Operating System (please complete the following information):

OS (Linux 4.4.180+ x86_64"):
Version of Docker (20.10.3, build 55f0773):
Version of Docker-Compose (1.28.5):

Elasticsearch, Logstash, Kibana (please complete the following information):

Version of ELK (all latest version)

**Service logs {"@timestamp":"2022-07-16T15:58:01.090Z", "log.level": "INFO", "message":"using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/md2)]], net usable_space [233.1gb], net total_space [3.4tb], types [btrfs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-docker-cluster"} es01 | {"@timestamp":"2022-07-16T15:58:01.093Z", "log.level": "INFO", "message":"heap size [512mb], compressed ordinary object pointers [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-docker-cluster"} es01 | {"@timestamp":"2022-07-16T15:58:01.338Z", "log.level": "INFO", "message":"node name [es01], node ID [ak8K5T76SfidrbZnn07TFw], cluster name [es-docker-cluster], roles [data_cold, data, remote_cluster_client, master, data_warm, data_content, transform, data_hot, ml, data_frozen, ingest]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-docker-cluster"} logstash | [INFO ] 2022-07-16 15:58:03.390 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Failed to perform request {:message=>"Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)", :exception=>Manticore::SocketException, :cause=>org.apache.http.conn.HttpHostConnectException: Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)} logstash | [WARN ] 2022-07-16 15:58:03.394 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@es01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://es01:9200/][Manticore::SocketException] Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)"} logstash | [INFO ] 2022-07-16 15:58:08.405 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Failed to perform request {:message=>"Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)", :exception=>Manticore::SocketException, :cause=>org.apache.http.conn.HttpHostConnectException: Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)} logstash | [WARN ] 2022-07-16 15:58:08.408 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@es01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://es01:9200/][Manticore::SocketException] Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)"} logstash | [INFO ] 2022-07-16 15:58:13.423 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Failed to perform request {:message=>"Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)", :exception=>Manticore::SocketException, :cause=>org.apache.http.conn.HttpHostConnectException: Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)} logstash | [WARN ] 2022-07-16 15:58:13.427 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@es01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://es01:9200/][Manticore::SocketException] Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)"} logstash | [ERROR] 2022-07-16 15:58:18.390 [monitoring-license-manager] licensereader - Unable to retrieve license information from license server {:message=>"No Available connections"} logstash | [INFO ] 2022-07-16 15:58:18.436 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Failed to perform request {:message=>"Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)", :exception=>Manticore::SocketException, :cause=>org.apache.http.conn.HttpHostConnectException: Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)} logstash | [WARN ] 2022-07-16 15:58:18.437 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@es01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://es01:9200/][Manticore::SocketException] Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)"} logstash | [INFO ] 2022-07-16 15:58:20.997 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] licensereader - Failed to perform request {:message=>"Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)", :exception=>Manticore::SocketException, :cause=>org.apache.http.conn.HttpHostConnectException: Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)} logstash | [WARN ] 2022-07-16 15:58:21.014 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] licensereader - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash_system:xxxxxx@es01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://es01:9200/][Manticore::SocketException] Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)"} es02 | {"@timestamp":"2022-07-16T15:58:23.065Z", "log.level": "WARN", "message":"unable to install syscall filter: ", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.JNANatives","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"es-docker-cluster","error.type":"java.lang.UnsupportedOperationException","error.message":"seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed","error.stack_trace":"java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and emCallFilter(Natives.java:102)\n\tat org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:112)\n\tat org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:183)\n\tat org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358)\n\tat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)\n\tat org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)\n\tat org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81)\n\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)\n\tat org.elasticsearch.cli.Command.main(Command.java:77)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)\n"}logstash | [INFO ] 2022-07-16 15:58:38.507 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Failed to perform request {:message=>"Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)", :exception=>Manticore::SocketException, :cause=>org.apache.http.conn.HttpHostConnectException: Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)}logstash | [WARN ] 2022-07-16 15:58:38.518 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:213] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@es01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://es01:9200/][Manticore::SocketException] Connect to es01:9200 [es01/172.22.0.3] failed: Connection refused (Connection refused)"}es01 | {"@timestamp":"2022-07-16T15:58:39.886Z", "log.level": "INFO", "message":"creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=512mb}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.transport.netty4.NettyAllocator","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-docker-cluster"}es03 | {"@timestamp":"2022-07-16T15:58:40.018Z", "log.level": "INFO", "message":"version[8.2.2], pid[10], build[default/docker/9876968ef3c745186b94fdabd4483e01499224ef/2022-05-25T15:47:06.259735307Z], OS[Linux/4.4.180+/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/18.0.1.1/18.0.1.1+2-6]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"es-docker-cluster"}es03 | {"@timestamp":"2022-07-16T15:58:40.028Z", "log.level": "INFO", "message":"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"es-docker-cluster"}es03 | {"@timestamp":"2022-07-16T15:58:40.030Z", "log.level": "INFO", "message":"JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-12626546595507795926, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"es-docker-cluster"}es01 | {"@timestamp":"2022-07-16T15:58:40.067Z", "log.level": "INFO", "message":"using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-docker-cluster"}es01 | {"@timestamp":"2022-07-16T15:58:40.246Z", "log.level": "INFO", "message":"using discovery type [multi-node] and seed hosts providers [settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.discovery.DiscoveryModule","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-docker-cluster"}

a3ilson commented 2 years ago

Appears that you need to configure the security (eg step 4) from the readme.

capmerah commented 2 years ago

Appears that you need to configure the security (eg step 4) from the readme.

I did, I replace ELASTIC-PASSWORD, KIBANA-PASSWORD, LOGSTASH-PASSWORD with my own password and used the script.

a3ilson commented 2 years ago

Did you amend the passwords then run the script?

Try running just the script, if you haven’t already, from a fresh unzip of the main.zip.

capmerah commented 2 years ago

Did you amend the passwords then run the script?

Yes I did.

Try running just the script, if you haven’t already, from a fresh unzip of the main.zip.

I tried running the script. Same error is still exists.

I am currently running synology ds220+ with 18gb ram(16+2) so I can only use 7z x instead of unzip, synology is still linux and it still can run docker though. But another problem with synology is that I cannot run apt-get. Let me know if using synology is not possible so that I can power up a linux VM inside synology and run an implementation from a VM. Thanks.

a3ilson commented 2 years ago

I’m currently away but I’ll treat this next week and see if i can determine the issue. However, i do but have the same hardware which may be the culprit.

capmerah commented 2 years ago

I’m currently away but I’ll treat this next week and see if i can determine the issue. However, i do but have the same hardware which may be the culprit.

Thanks for input. For now I'll power up my librenms(temp) on my docker from synology based on https://github.com/librenms/docker which is up and running. I still prefer grafana though so I'll wait for your next step on how to troubleshoot. If still no dice then VM it is.

alovillagon commented 2 years ago

Same issue here, all running

"Docker ps" show all process are running

Cant access from the LAN to the port 5601... but when I use "curl ip:port" from the server, can access it

alovillagon commented 2 years ago

SOLVED:

Replace all values where "127.0.0.1" to "0.0.0.0" in "docker-compose.yml"

Neccie commented 2 years ago

I’m currently away but I’ll treat this next week and see if i can determine the issue. However, i do but have the same hardware which may be the culprit.

Thanks for input. For now I'll power up my librenms(temp) on my docker from synology based on https://github.com/librenms/docker which is up and running. I still prefer grafana though so I'll wait for your next step on how to troubleshoot. If still no dice then VM it is.

I have ran in the same issue i think, running with docker-compose and my kibana got stuck on starting. I cannot confim but i think it had to do with the passwords, i used complex ones and that seemed to fail. Tried simple passwords (as in no special characters) and it suddenly worked (or it was something else). Worth a try and confirm perhaps?

a3ilson commented 2 years ago

@Neccie are you also running in a synology? This odd running on my current setup via Ubuntu.

Neccie commented 2 years ago

@Neccie are you also running in a synology? This odd running on my current setup via Ubuntu.

Ubuntu 22.04 LTS on a existing docker installation. I'm still not sure what i did but double check .env passwords and docker-compose.yml. If i now purge everything and use my working 2 config files I get everything to start up as it should.

a3ilson commented 2 years ago

Are your working config files different than the repo? Or was this a glitch/fluke?

Neccie commented 2 years ago

Actually tried it in a new VM to confirm, might be me but it goes wrong for me when i use special characters in the .env file for the passwords. For instance if i use "1234567890" it fails cause it expects a string, so "ab1234567890cd" works fine but "ab12345$%^&67890cd" will fail again which was my initial problem i guess. With or without qoutes doesn't matter.

a3ilson commented 1 year ago

@Neccie - the password limitations (character limitations) are related to Elastic

pfelk / docker

stuck with docker compose up process and unable to access Kibana #43