Closed baptisteem closed 3 months ago
Thanks for the detailed issue report!
It's really strange indeed, especially knowing that our CI pipeline also runs on Ubuntu 22.04, in Azure (Standard_D4ads_v5, eastus).
up setup
again solves the issue?Thanks for that quick answer, strange indeed 😬
- Does the server have enough resources?
I think so yes, I would expect the container not to be able to start if that were the case. There is a warning in the logs that the max virtual memory is too low, but I don't think it's related as I'm able to ping the elastic container from the server and get an answer
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144];
- Could it be that Elasticsearch simply took a while to start due to 1. and running up setup again solves the issue?
Sadly no, I tried to re-run everything a couple of time and the setup reacts the same even when elastic is started
edit: the issue seems to be for the setup container to reach the elastic one and I can't find a reason why 🤔
Maybe the firewall is interfering with the docker-elk bridge. If that's the case, all containers will fail to reach Elasticsearch, no just the setup container. This should be visible in Kibana's logs if you up kibana
.
Yes you're right, Kibana can't reach the elastic container as well
kibana-1 | [2024-06-08T20:24:59.579+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. Request timed out
Possibly related:
Thanks for the pointer, I had a look and it seems to be a bit different. What you link is about Docker between too permissive and over-writing the rules from the firewall, whereas I'm having the opposite issue 😓
I couldn't spend more time debugging the issue and I installed elastic directly on the server without using Docker. Thanks a lot for your help anyway 🙏
@baptisteem the problem I observed was more about firewalld/ufw getting in the way of the iptables rules written by Docker.
In docker-elk, we have a dedicated Docker network bridge that containers are being attached to: https://github.com/deviantony/docker-elk/blob/9a51242878031d0187ac88c9947fa12254608fa2/docker-compose.yml#L109-L111
By default, Docker enables containers within the bridge to communicate freely between each other (but isolates them from containers which aren't attached to the bridge). However, your firewall management software might not be aware of this, and instead enforce a block/drop between any sources/destinations which were not explicitly allowed.
As an aside, the reason why you were able to communicate with Elasticsearch from the host is because the port mapping 9200:9200
essentially creates a direct access from the host network namespace to the container, which usually bypasses container-to-container firewall rules.
Problem description
Trying to setup the stack on an Azur server running on Ubuntu and it gets stuck waiting for Elasticsearch to answer
Command ran
docker compose up setup
Extra information
curl -X GET "http://0.0.0.0:9200" -u elastic:changeme
Stack configuration
Docker setup
Container logs
Setup container
Elasticsearch container