Closed e101sg closed 1 year ago
The tell-tale line in this screenshot is the malcolm_opensearch_1 exited with code 1
, which means basically that opensearch either crashed, or, more likely, was killed, which is almost 100% of the time due to resources issues. Malcolm (because of OpenSearch and Logstash) is a memory hog.
So let's check how much memory you've got total, and how much is allowed for OpenSearch and Logstash:
For example, on my workstation right now:
$ head -n 1 /proc/meminfo
MemTotal: 65778428 kB
$ grep JAVA_OPTS config/*.env
config/logstash.env:LS_JAVA_OPTS=-server -Xms6g -Xmx6g -Xss1536k -XX:-HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom -Dlog4j.formatMsgNoLookups=true
config/opensearch.env:OPENSEARCH_JAVA_OPTS=-server -Xms30g -Xmx30g -Xss256k -XX:-HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom -Dlog4j.formatMsgNoLookups=true
So you can see my system has 64GiB of memory and I'm giving 30 of that to OpenSearch and 6 to LogStash.
Malcolm will run acceptably well for small analysis tasks on a system with 16GiB of memory (in which case, if I remember right, the install.py --configure
script will suggest 10 for OpenSearch and 2.5 or so for Logstash). I have run it on systems with less than 16GiB (like 12 or maybe even 10) but it was pretty slow and I would occasionally get opensearch crashes.
Appreciate your reply,Seth. Interestingly the my VM is 64GB. In docker-compose.yml
. Opensearch has 30 GB and LS has 6GB. I have increased both of it 36GB and 8 GB in .yml file and restarted the VM. No success. As a last resort, running the ./scripts/configure
and allocate higher memory to Opensearch and Logstash; Will it help.
Thanks a lot.
This is actually a bit counterintuitive, but why don't we try going to other way. Set 24G for OpenSearch and 3g for Logstash and see what happens.
Reduced the memory to 24G/ 3G and freshly run ./scripts/install.py
Still no peace. Almost entire day tried. Same issue connect ECONNREFUSED 172.18.0.8:9200
and /getaddrinfo ENOTFOUND opensearch.
Not sure the connection to 172.18.0.8 is a problem.
Anyway, i am still using it in University lab. Wish to install latest Malcolm freshly.... Is the deleting the Malcolm folder in /home/ubuntu/Malcolm
enough or i need to take extra caution.
Cheers & Thanks :)
How long does it run before getting the crash on opensearch usually?
Right after it crashes, could you do a docker-compose logs opensearch
and share the output here?
As far as clearing malcolm out, yeah deleting the folder should be fine.
We should be able to figure it out and get it working. Another option (since this is a VM anyway so you've probably got some control over what you're running) is that you could grab the Malcolm installer ISO and just install that in a VM and see if it behaves differently (see the end-to-end guide).
Usually the ECONNREFUSED 172.18.0.8:9200
appears first and around 4th or 5th Minute /getaddrinfo ENOTFOUND opensearch
appears. Also, when i remove ( yesterday,just tried.. is it make any difference) the Ubuntu VM's swap memory The errors appears even faster than the usual time.
Docker-compose Logs
Yesterday, i have noticed the same error on Docker Desktop when Malcolm's Opensearch or dashboard container (can not recall the correct name) crashed. It points to /usr/share/openshare/config
where i added just empty file opensearch.yml
Not useful. Later, i have noticed, the Opensearch.Dockerfile
mentions about Opensearch- security plugins in line 42,43, 44.
as seen in below. Not sure.. is it both related ? How it related? Guess, it gives a hint.
On the another note: We close this issue. Will try fresh Malcolm install on top of VM. (prefer this than ISO install). Thanks a lot. Cheers!
As far as the warning about those plugins being disabled, this is normal.
Also I don't understand what you mean about creating an empty file at /usr/share/openshare/config/opensearch.yml
. All of that exists in the docker container, those are not files you need to create on your host.
Beyond that, I'm not sure what to tell you, these are not problems I'm seeing anywhere else. I'd suggest following the instructions using docker community edition (not docker desktop) using Ubuntu LTS as in this guideline or with the ISO install guideline.
Dear Community,
Malcolm installation on top of the Ubuntu 22.04 as VM was working well. It was installed on Feb 2023. Since last week, i am getting this strange error as seen on screen shot. Issue: Can not access the Arkime, Opensearch Dashboard. Seems this issue related to Kibana/Elastic search inherited into OpenSearch ??!! Restarted the Malcolm and VM but issue not solved. Any thoughts highly welcome.
On another note: Wish the Malcolm community should have a Slack or Discord channel to share the knowledge/idea.... Thanks a lot.