ElasticSearch doesn't run on macOS host with sebp/elk:683

Hello!

I'm the author of Distributed Systems with Node.js. This is a printed book that guides the reader through the installation and use of many different services. For a section on logging I guide the user through using an ELK cluster by running the following command:

Example 4-2. Running ELK within Docker

$ sudo sysctl -w vm.max_map_count=262144 # Linux Only
$ docker run -p 5601:5601 -p 9200:9200 \
  -p 5044:5044 -p 7777:7777/udp \
  -v $PWD/misc/elk/udp.conf:/etc/logstash/conf.d/99-input-udp.conf \
  -e MAX_MAP_COUNT=262144 \
  -it --name distnode-elk sebp/elk:683

The contents of misc/elk/udp.conf are available at available here: https://github.com/tlhunter/distributed-node/blob/master/misc/elk/udp.conf

This command was previously tested on two separate macOS laptops and they both worked great, back when sebp/elk:683 was the current version. However, now both one of my readers and one of my technical reviewers are getting errors when running this command. They're both using a non-M1 MacBook pro. A copy of the STDOUT/STEDERR from the containers have been attached with things like timestamps and PID values removed to make them easy to diff.

The gist of the errors is that ElasticSearch doesn't seem to be running, and when they run curl http://localhost:9200/_cluster/health on the host they get an error of curl: (52) Empty reply from server. Here's some of the notable entries where Kibana can't reach ElasticSearch:

Unable to revive connection: http://localhost:9200/
No living connections

My Linux machine still runs the container fine.

Do you know what the problem might be? It seems like something changed over time that caused this to stop working. Perhaps ES phones home and Elastic changed some behavior?

Command Output:

Hi Thomas,

Honoured to have my image featured in your book 😄, let me try to help you out.

I’ll preface this by saying that I don’t have access to a Mac so I won’t be able to replicate the issue – if any watchers of this repo see this and do have a Mac, then your input is more than welcome.

Looking at the issue and the logs, nothing springs to mind (ES is starting up properly until the licence bit, then no longer seems to be responding), although I should mention that a few unresolved issues have been raised with strange behaviours on macOS, and that very recent issue #349 appears to display the same behaviour using the latest image. The idea that ES is phoning home and getting rejected for some reason sounds appealing to explain why something that used to work has stopped working, but on the other hand it’s still working perfectly on other hosts, e.g. your Linux, my Windows host (just tested with 6.8.3 a minute ago).

So might I suggest:

Confirming that ES has indeed stopped running by attempting to connect from your browser to http://localhost:9200/
Attempting to start ES manually from within the container by going through the instructions at https://elk-docker.readthedocs.io/#es-not-starting – hoping that the start-up output will give a hint as to what is going on

Here's the output that my reader received when they attempted to run the ES daemon manually:

rickhedin@Ricks-MacBook-Pro distributed-node % docker run -it sebp/elk:683 bash

root@d88eb48b5835:/# /opt/logstash/bin/logstash --path.data /tmp/logstash/data \

>     -e 'input { stdin { } } output { elasticsearch { hosts => ["localhost"] } }'

Sending Logstash logs to /opt/logstash/logs which is now configured via log4j2.properties

[2021-04-18T21:45:11,891][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/tmp/logstash/data/queue"}

[2021-04-18T21:45:11,905][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/tmp/logstash/data/dead_letter_queue"}

[2021-04-18T21:45:12,736][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

[2021-04-18T21:45:12,758][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.8.3"}

[2021-04-18T21:45:12,813][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"5609c084-2a97-4532-91b3-29161f7d8e04", :path=>"/tmp/logstash/data/uuid"}

[2021-04-18T21:45:20,329][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}

[2021-04-18T21:45:21,149][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}

[2021-04-18T21:45:21,493][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

[2021-04-18T21:45:21,543][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}

[2021-04-18T21:45:21,678][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x1a9c3218 run>"}

The stdin plugin is now waiting for input:

[2021-04-18T21:45:21,763][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

[2021-04-18T21:45:22,231][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

[2021-04-18T21:45:26,556][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

[2021-04-18T21:45:31,569][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

[2021-04-18T21:45:36,584][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

[2021-04-18T21:45:41,593][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

[2021-04-18T21:45:47,572][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

[2021-04-18T21:45:52,587][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

[2021-04-18T21:45:57,597][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

[2021-04-18T21:46:02,608][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

[2021-04-18T21:46:07,658][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

[2021-04-18T21:46:12,677][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

[2021-04-18T21:46:18,658][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

[2021-04-18T21:46:23,673][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

^C[2021-04-18T21:46:26,861][WARN ][logstash.runner          ] SIGINT received. Shutting down.

[2021-04-18T21:46:27,710][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x1a9c3218 run>"}

[2021-04-18T21:46:27,716][INFO ][logstash.runner          ] Logstash shut down.

root@d88eb48b5835:/# 

root@d88eb48b5835:/# 

root@d88eb48b5835:/# exit

exit

rickhedin@Ricks-MacBook-Pro distributed-node % 

rickhedin@Ricks-MacBook-Pro distributed-node % 

rickhedin@Ricks-MacBook-Pro distributed-node %

Actually, I think the reader ran the wrong command, I'll follow up.

This is the real output:

rickhedin@Ricks-MacBook-Pro distributed-node % docker run -it sebp/elk:683 bash

root@b4cb24d18911:/# ES_PATH_CONF=/etc/elasticsearch gosu elasticsearch /opt/elasticsearch/bin/elasticsearch -Epath.logs=/var/log/elasticsearch -Epath.data=/var/lib/elasticsearch

[2021-04-19T03:23:32,910][INFO ][o.e.e.NodeEnvironment    ] [8gO1KUf] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/vda1)]], net usable_space [51.9gb], net total_space [58.4gb], types [ext4]

[2021-04-19T03:23:32,917][INFO ][o.e.e.NodeEnvironment    ] [8gO1KUf] heap size [990.7mb], compressed ordinary object pointers [true]

[2021-04-19T03:23:32,920][INFO ][o.e.n.Node               ] [8gO1KUf] node name derived from node ID [8gO1KUfkTv6G3htNJAgW1w]; set [node.name] to override

[2021-04-19T03:23:32,921][INFO ][o.e.n.Node               ] [8gO1KUf] version[6.8.3], pid[17], build[default/tar/0c48c0e/2019-08-29T19:05:24.312154Z], OS[Linux/5.10.25-linuxkit/amd64], JVM[Private Build/OpenJDK 64-Bit Server VM/1.8.0_222/25.222-b10]

[2021-04-19T03:23:32,921][INFO ][o.e.n.Node               ] [8gO1KUf] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-2136078822882508995, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar]

[2021-04-19T03:23:35,992][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [aggs-matrix-stats]

[2021-04-19T03:23:35,993][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [analysis-common]

[2021-04-19T03:23:35,993][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [ingest-common]

[2021-04-19T03:23:35,993][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [ingest-geoip]

[2021-04-19T03:23:35,994][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [ingest-user-agent]

[2021-04-19T03:23:35,994][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [lang-expression]

[2021-04-19T03:23:35,994][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [lang-mustache]

[2021-04-19T03:23:35,995][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [lang-painless]

[2021-04-19T03:23:35,995][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [mapper-extras]

[2021-04-19T03:23:35,996][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [parent-join]

[2021-04-19T03:23:35,996][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [percolator]

[2021-04-19T03:23:35,996][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [rank-eval]

[2021-04-19T03:23:35,997][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [reindex]

[2021-04-19T03:23:35,998][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [repository-url]

[2021-04-19T03:23:35,998][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [transport-netty4]

[2021-04-19T03:23:35,998][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [tribe]

[2021-04-19T03:23:35,999][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [x-pack-ccr]

[2021-04-19T03:23:35,999][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [x-pack-core]

[2021-04-19T03:23:36,000][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [x-pack-deprecation]

[2021-04-19T03:23:36,000][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [x-pack-graph]

[2021-04-19T03:23:36,001][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [x-pack-ilm]

[2021-04-19T03:23:36,001][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [x-pack-logstash]

[2021-04-19T03:23:36,002][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [x-pack-ml]

[2021-04-19T03:23:36,002][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [x-pack-monitoring]

[2021-04-19T03:23:36,003][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [x-pack-rollup]

[2021-04-19T03:23:36,003][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [x-pack-security]

[2021-04-19T03:23:36,004][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [x-pack-sql]

[2021-04-19T03:23:36,005][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [x-pack-upgrade]

[2021-04-19T03:23:36,006][INFO ][o.e.p.PluginsService     ] [8gO1KUf] loaded module [x-pack-watcher]

[2021-04-19T03:23:36,007][INFO ][o.e.p.PluginsService     ] [8gO1KUf] no plugins loaded

[2021-04-19T03:23:44,169][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [8gO1KUf] [controller/95] [Main.cc@109] controller (64 bit): Version 6.8.3 (Build 7ace96ffff9215) Copyright (c) 2019 Elasticsearch BV

[2021-04-19T03:23:45,403][DEBUG][o.e.a.ActionModule       ] [8gO1KUf] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security

[2021-04-19T03:23:45,886][INFO ][o.e.d.DiscoveryModule    ] [8gO1KUf] using discovery type [zen] and host providers [settings]

[2021-04-19T03:23:47,828][INFO ][o.e.n.Node               ] [8gO1KUf] initialized

[2021-04-19T03:23:47,829][INFO ][o.e.n.Node               ] [8gO1KUf] starting ...

[2021-04-19T03:23:48,185][INFO ][o.e.t.TransportService   ] [8gO1KUf] publish_address {172.17.0.2:9300}, bound_addresses {0.0.0.0:9300}

[2021-04-19T03:23:48,217][INFO ][o.e.b.BootstrapChecks    ] [8gO1KUf] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2021-04-19T03:23:51,335][INFO ][o.e.c.s.MasterService    ] [8gO1KUf] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {8gO1KUf}{8gO1KUfkTv6G3htNJAgW1w}{AznK8uHtSaC9Cesx7wj68w}{172.17.0.2}{172.17.0.2:9300}{ml.machine_memory=2084249600, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

[2021-04-19T03:23:51,348][INFO ][o.e.c.s.ClusterApplierService] [8gO1KUf] new_master {8gO1KUf}{8gO1KUfkTv6G3htNJAgW1w}{AznK8uHtSaC9Cesx7wj68w}{172.17.0.2}{172.17.0.2:9300}{ml.machine_memory=2084249600, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {8gO1KUf}{8gO1KUfkTv6G3htNJAgW1w}{AznK8uHtSaC9Cesx7wj68w}{172.17.0.2}{172.17.0.2:9300}{ml.machine_memory=2084249600, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])

[2021-04-19T03:23:51,459][INFO ][o.e.h.n.Netty4HttpServerTransport] [8gO1KUf] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}

[2021-04-19T03:23:51,459][INFO ][o.e.n.Node               ] [8gO1KUf] started

[2021-04-19T03:23:51,479][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [8gO1KUf] Failed to clear cache for realms [[]]

[2021-04-19T03:23:51,570][INFO ][o.e.g.GatewayService     ] [8gO1KUf] recovered [0] indices into cluster_state

[2021-04-19T03:23:51,869][INFO ][o.e.c.m.MetaDataIndexTemplateService] [8gO1KUf] adding template [.watches] for index patterns [.watches*]

[2021-04-19T03:23:51,902][INFO ][o.e.c.m.MetaDataIndexTemplateService] [8gO1KUf] adding template [.triggered_watches] for index patterns [.triggered_watches*]

[2021-04-19T03:23:51,965][INFO ][o.e.c.m.MetaDataIndexTemplateService] [8gO1KUf] adding template [.watch-history-9] for index patterns [.watcher-history-9*]

[2021-04-19T03:23:51,994][INFO ][o.e.c.m.MetaDataIndexTemplateService] [8gO1KUf] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]

[2021-04-19T03:23:52,047][INFO ][o.e.c.m.MetaDataIndexTemplateService] [8gO1KUf] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]

[2021-04-19T03:23:52,077][INFO ][o.e.c.m.MetaDataIndexTemplateService] [8gO1KUf] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]

[2021-04-19T03:23:52,107][INFO ][o.e.c.m.MetaDataIndexTemplateService] [8gO1KUf] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]

[2021-04-19T03:23:52,127][INFO ][o.e.c.m.MetaDataIndexTemplateService] [8gO1KUf] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]

[2021-04-19T03:23:52,291][INFO ][o.e.l.LicenseService     ] [8gO1KUf] license [ea2a907d-14e8-469a-8a3f-0fe40fb652ba] mode [basic] - valid

^C[2021-04-19T03:26:38,889][INFO ][o.e.x.m.p.NativeController] [8gO1KUf] Native controller process has stopped - no new native processes can be started

[2021-04-19T03:26:38,894][INFO ][o.e.n.Node               ] [8gO1KUf] stopping ...

[2021-04-19T03:26:38,911][INFO ][o.e.x.w.WatcherService   ] [8gO1KUf] stopping watch service, reason [shutdown initiated]

[2021-04-19T03:26:39,098][INFO ][o.e.n.Node               ] [8gO1KUf] stopped

[2021-04-19T03:26:39,098][INFO ][o.e.n.Node               ] [8gO1KUf] closing ...

[2021-04-19T03:26:39,131][INFO ][o.e.n.Node               ] [8gO1KUf] closed

root@b4cb24d18911:/# # I pressed ctrl-c. 

root@b4cb24d18911:/# 

root@b4cb24d18911:/# 

root@b4cb24d18911:/# exit

exit

rickhedin@Ricks-MacBook-Pro distributed-node % date

Sun Apr 18 23:26:57 EDT 2021

rickhedin@Ricks-MacBook-Pro distributed-node % 

rickhedin@Ricks-MacBook-Pro distributed-node % 

rickhedin@Ricks-MacBook-Pro distributed-node % 

rickhedin@Ricks-MacBook-Pro distributed-node %

Thanks. The logs seem to show Elasticseach starting up fine (adjust like the logs in the original message) before your reader ^C’d. Could I ask if Elasticsearch was responding properly after starting up? (See first point of my previous comment.)

There’s a the possibility that Elasticsearch ran out of memory and simply died when the other services started. To check for that could you start a container as usual, then docker exec into the image when you get the error messages from Kibana, and check if the process is running e.g. using ps?

@spujadas this is from my reader:

I increased the memory (from 2.75 Gb?) to 4.0 Gb. That appears to have been the solution.

So it sounds like the solution is to raise the default memory limit of Docker Desktop. Might affect Windows too? Not sure if the default memory allocated on macOS has changed in the last several months, or maybe a new version of macOS changed memory management, or what.

Not sure if ES spits out a special exit status if it doesn't get enough memory allocated. If so, it might be nice to have the containers print a special error message about memory before terminating.

So it sounds like the solution is to raise the default memory limit of Docker Desktop.

Yes, this is true across all platforms. In the documentation I recommend allocating at least 4GB (up from 2GB a few years ago).

Not sure if ES spits out a special exit status if it doesn't get enough memory allocated.

Unfortunately it doesn’t. ES is fine with the available memory when it starts up, but then gets killed by the kernel when the container runs out of memory after starting the other services. So ES doesn’t know it’s being killed, let alone alone logs it, which is why troubleshooting memory issues can be tricky with this image.

@spujadas this is from my reader:

I increased the memory (from 2.75 Gb?) to 4.0 Gb. That appears to have been the solution.

So it sounds like the solution is to raise the default memory limit of Docker Desktop. Might affect Windows too? Not sure if the default memory allocated on macOS has changed in the last several months, or maybe a new version of macOS changed memory management, or what.

Can confirm this works.

Perhaps a note in the README would suffice?

Or even better, a warning in the docker logs?

Perhaps a note in the README would suffice?

It’s documented in the prerequisites section of the main documentation. I’ve added a note for Docker Desktop for Windows.

Or even better, a warning in the docker logs?

No, there are too many variables at play to determine when a warning should be issued.

spujadas / elk-docker

ElasticSearch doesn't run on macOS host with sebp/elk:683 #350

Command Output: