[BUG] Elasticsearch hostname in Logstash config not defined

[novaksam]

BEFORE CREATING THE ISSUE, CHECK THE FOLLOWING GUIDES:

• [x] FAQ
• [ ] Troubleshooting
• [ ] Search current/closed issues for similar questions, and utilize github/google search to see if an answer exists for the error I'm encountering.

If the above did not answer your question, proceed with creating an issue below:

Describe the bug

[] I was assisting someone to setup LME, and it appears that the hostname 'elasticsearch' https://github.com/cisagov/LME/blob/cb66dbbc9ee7a39253a70ff37ec5baa38fc97933/Chapter%203%20Files/logstash.conf#L39 https://github.com/cisagov/LME/blob/cb66dbbc9ee7a39253a70ff37ec5baa38fc97933/Chapter%203%20Files/logstash.conf#L51 https://github.com/cisagov/LME/blob/cb66dbbc9ee7a39253a70ff37ec5baa38fc97933/Chapter%203%20Files/logstash.conf#L62 is not defined anywhere. I have years of experience running ELK on traditional servers, so there may be a disconnect with my current understanding of containerization, but I believe adding an alias to the docker compose file https://github.com/cisagov/LME/blob/cb66dbbc9ee7a39253a70ff37ec5baa38fc97933/Chapter%203%20Files/docker-compose-stack.yml#L42

    networks:
      esnet:
        aliases:
          - elasticsearch

might resolve the issue. This obviously doesn't scale, but I'm sure some sort of options could be added to the deployment shell script for text replacements.

To Reproduce

Deploy LME step 3. I don't have access to the logs or immediate configs, as it was someone else's configuration.

Expected behavior

Logstash is able to find the elasticsearch hostname that is defined in the output sections of the pipeline.

Screenshots OPTIONAL

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

[aarz-snl]

I may not be following here --

but maybe you're referring to the docker compose services?

In the docker compose live yml youll see services listed. One of them is called 'elasticsearch' this allows you to communicate in the private docker network using https://elasticsearch. This will not work from OUTSIDE the docker network. ie. from your client machine to https://elasticsearch. These services have background communications happening that they talk to each other with these domain names

Communication to the linux machine must happen with its domain name and port.

So https://linuxIPaddress or https://linux.machine.domain.name

If you had a docker service running exposed with port say 123 you would have to use

https://linux.machine.domain.name:123

because logstash runs as a container in the docker swarm it can communicate to elasticsearch using https://elasticsearch:port

From within the docker network other services can communicate with logstash using https://logstash... this is based on what you name them in the docker compose file under services... as long as they're all grouped under the same network which we call 'esnet' . I hope that makes sense.

[novaksam]

@aarz-snl This appears to be a lack of familiarity with Docker and compose! The person i was working with made some networking adjustments and they are up and running, so this is a non-issue :)