search

idaholab / Malcolm

Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files), Zeek logs and Suricata alerts.
https://idaholab.github.io/Malcolm/
Other
359 stars 59 forks source link

The memory usage of Malcolm is over 30GB, even set only 10G for Opensearch. #543

Closed Aster-Lin closed 2 months ago

Aster-Lin commented 2 months ago

Describe the bug (edited) Something went wrong in starting Malcolm by ./scripts/start at opensearch. I found the reason that because the memory is out using. However, I have set the JVM to 10G for OpenSearch, and the total memory use of Malcolm is still over 30G.

To Reproduce Steps to reproduce the behavior:

  1. Start ./scripts/start

Expected behavior NA

Screenshots and/or Logs 

...
malcolm-pcap-monitor-1        | 2024-08-19 04:54:28 WARNING: GET http://opensearch:9200/_cluster/health/arkime_files?wait_for_status=yellow [status:N/A request:9.619s]
malcolm-pcap-monitor-1        | http.client.RemoteDisconnected: Remote end closed connection without response
malcolm-opensearch-1          | bash: line 20:    47 Killed                  "/usr/local/bin/service_check_passthrough.sh" -s opensearch /usr/share/opensearch/opensearch-docker-entrypoint.sh

Malcolm Version:

How are you running Malcolm?

Additional context What is the reason that Opensearch is killed? How to set and control the other functions memory use?

mmguero commented 2 months ago

The only functions you've got much control over are opensearch and logstash. You could try setting Logstash to 2500m. I have run Malcolm successfully in a VM with 16GB RAM total, giving 10g to OpenSearch and 2500m to Logstash.

Depending on your linux distribution, there may be settings you could tweak for the OOM, but I won't be much help there for that, you'd just have to experiment. Also, during installation, it may have prompted you to add some things to /etc/sysctl.conf, did you say yes to those things? And did you reboot the computer after the installation?

This is not a Malcolm bug, so I'm going to close this issue, but we can continue to discuss it here if you need to.

Aster-Lin commented 2 months ago

Hi @mmguero, Thanks for your kind help. And I found when I turn zeek-live-capture to false, the usage can be down to 15~16GB. So the main reason is that the zeek-live-capture function costs too much memory. Therefore, I decide not to use the live-capture function and set zeek_rotated_pcap to true, pcap_rotate_minutes=1, to get a quick off-live analysis result from local PCAP files and cost down the usage of the memory at the same time.

But the strange thing is I set arkime_rotate_pcap=false, but the analysis data by arkime are still shown on the Arkime dashboard. Is it correct or was there something else I missed to turn off?

mmguero commented 2 months ago

Probably what's happening is the flag ARKIME_AUTO_ANALYZE_PCAP_FILES is taking precedence. If you want to turn off the arkime analysis altogether set both ARKIME_AUTO_ANALYZE_PCAP_FILES and ARKIME_ROTATED_PCAP to false. But I think you are right, I think that's actually not supposed to process the rotated PCAP based on that first variable you set. I'll look into that in a future release.