search

idaholab / Malcolm

Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files), Zeek logs and Suricata alerts.
https://idaholab.github.io/Malcolm/
Other
356 stars 59 forks source link

Opensearch Error - ENOTFOUND #207

Closed e101sg closed 1 year ago

e101sg commented 1 year ago

Dear Community,

Malcolm installation on top of the Ubuntu 22.04 as VM was working well. It was installed on Feb 2023. Since last week, i am getting this strange error as seen on screen shot. Issue: Can not access the Arkime, Opensearch Dashboard. Seems this issue related to Kibana/Elastic search inherited into OpenSearch ??!! Restarted the Malcolm and VM but issue not solved. Any thoughts highly welcome.

On another note: Wish the Malcolm community should have a Slack or Discord channel to share the knowledge/idea.... Thanks a lot.

Screenshot from 2023-06-05 17-18-18

mmguero commented 1 year ago

The tell-tale line in this screenshot is the malcolm_opensearch_1 exited with code 1, which means basically that opensearch either crashed, or, more likely, was killed, which is almost 100% of the time due to resources issues. Malcolm (because of OpenSearch and Logstash) is a memory hog.

So let's check how much memory you've got total, and how much is allowed for OpenSearch and Logstash:

For example, on my workstation right now:

$ head -n 1 /proc/meminfo 
MemTotal:       65778428 kB

$ grep JAVA_OPTS config/*.env
config/logstash.env:LS_JAVA_OPTS=-server -Xms6g -Xmx6g -Xss1536k -XX:-HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom -Dlog4j.formatMsgNoLookups=true
config/opensearch.env:OPENSEARCH_JAVA_OPTS=-server -Xms30g -Xmx30g -Xss256k -XX:-HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom -Dlog4j.formatMsgNoLookups=true

So you can see my system has 64GiB of memory and I'm giving 30 of that to OpenSearch and 6 to LogStash.

Malcolm will run acceptably well for small analysis tasks on a system with 16GiB of memory (in which case, if I remember right, the install.py --configure script will suggest 10 for OpenSearch and 2.5 or so for Logstash). I have run it on systems with less than 16GiB (like 12 or maybe even 10) but it was pretty slow and I would occasionally get opensearch crashes.

e101sg commented 1 year ago

Appreciate your reply,Seth. Interestingly the my VM is 64GB. In docker-compose.yml . Opensearch has 30 GB and LS has 6GB. I have increased both of it 36GB and 8 GB in .yml file and restarted the VM. No success. As a last resort, running the ./scripts/configure and allocate higher memory to Opensearch and Logstash; Will it help. Thanks a lot.

Malcolm_memory_1 Malcolm_memory_2

mmguero commented 1 year ago

This is actually a bit counterintuitive, but why don't we try going to other way. Set 24G for OpenSearch and 3g for Logstash and see what happens.

e101sg commented 1 year ago

Reduced the memory to 24G/ 3G and freshly run ./scripts/install.py Still no peace. Almost entire day tried. Same issue connect ECONNREFUSED 172.18.0.8:9200 and /getaddrinfo ENOTFOUND opensearch. Not sure the connection to 172.18.0.8 is a problem. Anyway, i am still using it in University lab. Wish to install latest Malcolm freshly.... Is the deleting the Malcolm folder in /home/ubuntu/Malcolm enough or i need to take extra caution.
Cheers & Thanks :)

mmguero commented 1 year ago

How long does it run before getting the crash on opensearch usually?

Right after it crashes, could you do a docker-compose logs opensearch and share the output here?

As far as clearing malcolm out, yeah deleting the folder should be fine.

We should be able to figure it out and get it working. Another option (since this is a VM anyway so you've probably got some control over what you're running) is that you could grab the Malcolm installer ISO and just install that in a VM and see if it behaves differently (see the end-to-end guide).

e101sg commented 1 year ago

  1. Usually the ECONNREFUSED 172.18.0.8:9200 appears first and around 4th or 5th Minute /getaddrinfo ENOTFOUND opensearch appears. Also, when i remove ( yesterday,just tried.. is it make any difference) the Ubuntu VM's swap memory The errors appears even faster than the usual time.

  2. Docker-compose Logs Yesterday, i have noticed the same error on Docker Desktop when Malcolm's Opensearch or dashboard container (can not recall the correct name) crashed. It points to /usr/share/openshare/config where i added just empty file opensearch.yml Not useful. Later, i have noticed, the Opensearch.Dockerfile mentions about Opensearch- security plugins in line 42,43, 44. as seen in below. Not sure.. is it both related ? How it related? Guess, it gives a hint.

On the another note: We close this issue. Will try fresh Malcolm install on top of VM. (prefer this than ISO install). Thanks a lot. Cheers! docker_logs_opensearch

docker_logs_opensearch_2

mmguero commented 1 year ago

As far as the warning about those plugins being disabled, this is normal.

Also I don't understand what you mean about creating an empty file at /usr/share/openshare/config/opensearch.yml. All of that exists in the docker container, those are not files you need to create on your host.

Beyond that, I'm not sure what to tell you, these are not problems I'm seeing anywhere else. I'd suggest following the instructions using docker community edition (not docker desktop) using Ubuntu LTS as in this guideline or with the ISO install guideline.