Not able to expose elasticsearch port 9200 externally(outside docker) and connect

ashishmgupta commented 3 years ago

Describe the problem

I'm trying to send logs from the Office 365 using the o365 filebeat to the ElasticSearch. Filebeat gets the data from o365 but not able to send to ElasticSearch and shows below error Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get "http://localhost:9200": EOF I added 9200:9200 to the docker config files so It could be exposed externally outside docker. helk-kibana-analysis-alert-basic.yml helk-kibana-analysis-basic.yml
helk-kibana-notebook-analysis-alert-basic.yml
helk-kibana-notebook-analysis-basic.yml

and then composed docker for each file. docker-compose -f docker/<config> up --build -d

After composing the docker using one file, I would test the filebeat using filebeat -e

and It would give me the same error Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get "http://localhost:9200": EOF

Attaching the config files here. helk-kibana-analysis-alert-basic.yml.txt helk-kibana-analysis-basic.yml.txt helk-kibana-notebook-analysis-alert-basic.yml.txt helk-kibana-notebook-analysis-basic.yml.txt

Provide the output of the following commands

Get operating system and version for linux (except Mac) use:
cat /etc/os-release
for Mac/OSX use:
sw_vers
Get disk space, memory, processor cores, and docker storage
echo -e "\nDocker Space:" && df -h /var/lib/docker; echo -e "\nMemory:" && free -g; echo -e "\nCores:" && getconf _NPROCESSORS_ONLN
Get output of the HELK docker containers:
docker ps --filter "name=helk"

Place all output, from the above commands, here

NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Docker Space:
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        49G   24G   24G  50% /

Memory:
              total        used        free      shared  buff/cache   available
Mem:              9           8           0           0           1           0
Swap:             1           1           0

Cores:
4

CONTAINER ID   IMAGE                                                 COMMAND                  CREATED          STATUS                          PORTS                                                                                                                                                                                                                          NAMES
6faf14986e88   otrf/helk-nginx:0.3.0                                 "/opt/helk/scripts/n…"   12 minutes ago   Up 12 minutes                   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                                                                                                                                                       helk-nginx
c862b1efcfdc   otrf/helk-spark-worker:2.4.5                          "./spark-worker-entr…"   13 minutes ago   Up 13 minutes                                                                                                                                                                                                                                                  helk-spark-worker
7519521961bb   docker_helk-jupyter                                   "/opt/jupyter/script…"   13 minutes ago   Up 13 minutes                   8000/tcp, 8888/tcp                                                                                                                                                                                                             helk-jupyter
873c054ac1b3   otrf/helk-spark-master:2.4.5                          "./spark-master-entr…"   13 minutes ago   Up 13 minutes                   7077/tcp, 0.0.0.0:8080->8080/tcp                                                                                                                                                                                               helk-spark-master
272cfe509229   otrf/helk-kafka-broker:2.4.0                          "./kafka-entrypoint.…"   45 minutes ago   Restarting (1) 47 seconds ago                                                                                                                                                                                                                                  helk-kafka-broker
560a76acd0ed   otrf/helk-logstash:7.6.2.1                            "/usr/share/logstash…"   45 minutes ago   Up 45 minutes                   0.0.0.0:3515->3515/tcp, 0.0.0.0:5044->5044/tcp, 0.0.0.0:5514->5514/tcp, 0.0.0.0:5514->5514/udp, 0.0.0.0:8515-8516->8515-8516/tcp, 0.0.0.0:8531->8531/tcp, 0.0.0.0:9200->9200/tcp, 0.0.0.0:8515-8516->8515-8516/udp, 9600/tcp   helk-logstash
ccbfa5fb9275   confluentinc/cp-ksql-cli:5.1.3                        "/bin/sh"                2 weeks ago      Up 45 minutes                                                                                                                                                                                                                                                  helk-ksql-cli
5f2a5a34b2dd   confluentinc/cp-ksql-server:5.1.3                     "/etc/confluent/dock…"   2 weeks ago      Up 10 hours                     0.0.0.0:8088->8088/tcp                                                                                                                                                                                                         helk-ksql-server
2941692e1a9a   otrf/helk-elastalert:latest                           "./elastalert-entryp…"   2 weeks ago      Up 10 hours                                                                                                                                                                                                                                                    helk-elastalert
858d52e2c774   otrf/helk-zookeeper:2.4.0                             "./zookeeper-entrypo…"   2 weeks ago      Up 10 hours                     2181/tcp, 2888/tcp, 3888/tcp                                                                                                                                                                                                   helk-zookeeper
58f1e061c42e   docker.elastic.co/kibana/kibana:7.6.2                 "/usr/share/kibana/s…"   2 weeks ago      Up 10 hours                     5601/tcp                                                                                                                                                                                                                       helk-kibana
d8d753680a0d   docker.elastic.co/elasticsearch/elasticsearch:7.6.2   "/usr/share/elastics…"   2 weeks ago      Up 10 hours                     9200/tcp, 9300/tcp                                                                                                                                                                                                             helk-elasticsearch

What version of HELK are you using

run the command from within the HELK repo run git log -1 --oneline

Place the output here

What version of Winlogbeat are you using if you are using Windows/WEF logs

b40f92f (HEAD -> master, origin/master, origin/HEAD) Update kibana.md

What steps did you take trying to fix the issue

How could we replicate the issue

Any additionally code or log context you would like to provide

Place the output here

Any additional context or input you have

pictures, comments, etc.

neu5ron commented 3 years ago

you can use nginx to "expose" port 9200 to forward to elasticsearch

ashishmgupta commented 3 years ago

Thank you for replying Nate. Do you have any guidance/notes on how to do this?

From: Nate Guagenti @.> Sent: Wednesday, April 28, 2021 2:24 AM To: Cyb3rWard0g/HELK @.> Cc: Ashish Gupta @.>; Author @.> Subject: Re: [Cyb3rWard0g/HELK] Not able to expose elasticsearch port 9200 externally(outside docker) and connect (#556)

you can use nginx to "expose" port 9200 to forward to elasticsearch

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Cyb3rWard0g/HELK/issues/556#issuecomment-828179734, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAH5JU4K2FOL5ANI2KI7KMLTK6SWZANCNFSM43Q5JZAA.

Cyb3rWard0g commented 3 years ago

I just checked one of your configs, and I see you modifying the docker config file and adding port 9200 to it, but you added it to the logstash service and not Elasticsearch. can you verify that please @ashishmgupta ? thank you!

ashishmgupta commented 3 years ago

Thank you. What would be the name of the elasticsearch file so I can make the change there?

Thanks and Regards, Ashish Gupta

From: Roberto Rodriguez @.> Sent: Sunday, May 9, 2021 12:14:37 AM To: Cyb3rWard0g/HELK @.> Cc: Ashish Gupta @.>; Mention @.> Subject: Re: [Cyb3rWard0g/HELK] Not able to expose elasticsearch port 9200 externally(outside docker) and connect (#556)

I just checked one of your configs, and I see you modifying the docker config file and adding port 9200 to it, but you added it to the logstash service and not Elasticsearch. can you verify that please @ashishmguptahttps://github.com/ashishmgupta ? thank you!

[image]https://user-images.githubusercontent.com/9653181/117560403-77010480-b05b-11eb-9d35-e75ced6a5ad7.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/Cyb3rWard0g/HELK/issues/556#issuecomment-835668955, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAH5JU3YJH4SYRDC6TJKRSLTMYD23ANCNFSM43Q5JZAA.

ashishmgupta commented 3 years ago

Ah. I realized it would be same file but elasticsearch section and add a port section to it with 9200. Will give that a shot.

Thanks and Regards, Ashish Gupta

From: Ashish Gupta @.> Sent: Sunday, May 9, 2021 8:26:26 AM To: Cyb3rWard0g/HELK @.>; Cyb3rWard0g/HELK @.> Cc: Mention @.> Subject: Re: [Cyb3rWard0g/HELK] Not able to expose elasticsearch port 9200 externally(outside docker) and connect (#556)

Thank you. What would be the name of the elasticsearch file so I can make the change there?

Thanks and Regards, Ashish Gupta

From: Roberto Rodriguez @.> Sent: Sunday, May 9, 2021 12:14:37 AM To: Cyb3rWard0g/HELK @.> Cc: Ashish Gupta @.>; Mention @.> Subject: Re: [Cyb3rWard0g/HELK] Not able to expose elasticsearch port 9200 externally(outside docker) and connect (#556)

I just checked one of your configs, and I see you modifying the docker config file and adding port 9200 to it, but you added it to the logstash service and not Elasticsearch. can you verify that please @ashishmguptahttps://github.com/ashishmgupta ? thank you!

[image]https://user-images.githubusercontent.com/9653181/117560403-77010480-b05b-11eb-9d35-e75ced6a5ad7.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/Cyb3rWard0g/HELK/issues/556#issuecomment-835668955, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAH5JU3YJH4SYRDC6TJKRSLTMYD23ANCNFSM43Q5JZAA.

Cyb3rWard0g / HELK