Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files), Zeek logs and Suricata alerts.
Added ARM64/AArch64 support. Malcolm can now run natively on ARM64 hardware. The ./scripts/configure script should detect the architecture and automatically adjust the image: names in the docker-compose.yml files in Docker deployments, or this can be changed manually by appending -arm64 to the tag for Malcolm's Docker images, e.g., ghcr.io/idaholab/malcolm/zeek:24.05.0-arm64. (idaholab/Malcolm#369)
Support for new environment variables added to Hedgehog Linux's control_vars.conf for tuning capture-related settings for Arkime on the sensor, bringing them into parity with those that were available in the arkime-live container in Malcolm. (idaholab/Malcolm#476)
Tweaked some of the default resource-related live capture settings for Suricata and Arkime.
Reworked the environment variables used for tuning Zeek live capture resource and performance on both Malcolm and Hedgehog Linux. An in-depth discussion of these tuning parameters can be found in the documentation. (idaholab/Malcolm#475)
Allow setting the spiDataMaxIndexes variable for Arkime's config.ini file via the ARKIME_SPI_DATA_MAX_INDICES environment variable. (idaholab/Malcolm#471)
Allow custom tags to be specified at the point of log file ingestion (i.e., FileBeat) on Malcolm and Hedgehog Linux. This makes it easier to specify custom tags used to group network traffic by sensor. (idaholab/Malcolm#463)
Handle invalid URLs made to the Malcolm web-based UIs better (with a custom 404/502 page). (idaholab/Malcolm#461)
Switched to official .deb packages for Arkime rather than building from source, reducing build times significantly. (Thanks @awick.)
Also, going forward Malcolm will track the latest Suricata release (from the Debian Stable Backports APT repository) rather than what's in the Debian Stable APT repository. (idaholab/Malcolm#462)
requests Python library to v2.32.0 for CVE-2024-35195
flask-cors Python library on Hedgehog Linux to v4.0.1 for CVE-2024-1681
Jinja Python library on Hedgehog Linux to v3.1.4 for CVE-2024-34064
Werkzeug Python library on Hedgehog linux to v3.0.3 for CVE-2024-34069
Bug fixes
The code that cleans up already-processed Zeek and Suricata logs after a defined period of time was out of date for the current FileBeat registry behavior and would potentially leave log files around longer than they needed to be. This has been remedied. (idaholab/Malcolm#479)
Fixed issue where the BPF capture filter was not passed to Zeek correctly. (idaholab/Malcolm#474)
The process which queries threat intelligence feeds and generates the corresponding Zeek intel files will no longer relpace existing intel definitions unless it succeeds in pulling definitions from at least one of the specified feeds. (idaholab/Malcolm#472)
Fixed calculation of memory and CPU resources used in ./scripts/status for Kubernetes deployment. (idaholab/Malcolm#467)
Added ARKIME_SPI_DATA_MAX_INDICES to arkime.env with a default value of 7, which manifests as spiDataMaxIndexes in Arkime's config.ini. If you are changing the Arkime index period from daily to weekly, hourly, etc., you may wish to adjust this value. (idaholab/Malcolm#471)
Added EXTRA_TAGS to upload-common.env for specifying custom tags to be associated with logs forwarded to Logstash by FileBeat. (idaholab/Malcolm#463)
A number of new and modified environment variables are available and can be added to zeek-live.env for tuning the resource utilization and performance of Zeek's live capture, see the documentation for details. (idaholab/Malcolm#475)
Hedgehog Linux
A number of new and modified environment variables are available for control_vars.conf for tuning the resource utilization and performance of Zeek's live capture, see the documentation for details. (idaholab/Malcolm#475)
Added support for new environment variables added to Hedgehog Linux's control_vars.conf for tuning capture-related settings for Arkime on the sensor, bringing them into parity with those that were available in the arkime-live container in Malcolm. (idaholab/Malcolm#476)
Malcolm v24.05.0 contains new features, improvements, bug fixes and component version updates.
https://github.com/cisagov/Malcolm/compare/v24.04.0...v24.05.0
./scripts/configurescript should detect the architecture and automatically adjust theimage:names in thedocker-compose.ymlfiles in Docker deployments, or this can be changed manually by appending-arm64to the tag for Malcolm's Docker images, e.g.,ghcr.io/idaholab/malcolm/zeek:24.05.0-arm64. (idaholab/Malcolm#369)control_vars.conffor tuning capture-related settings for Arkime on the sensor, bringing them into parity with those that were available in thearkime-livecontainer in Malcolm. (idaholab/Malcolm#476)config.inifile via theARKIME_SPI_DATA_MAX_INDICESenvironment variable. (idaholab/Malcolm#471)requestsPython library to v2.32.0 for CVE-2024-35195flask-corsPython library on Hedgehog Linux to v4.0.1 for CVE-2024-1681JinjaPython library on Hedgehog Linux to v3.1.4 for CVE-2024-34064WerkzeugPython library on Hedgehog linux to v3.0.3 for CVE-2024-34069./scripts/statusfor Kubernetes deployment. (idaholab/Malcolm#467)./config/) for Malcolm and incontrol_vars.conffor Hedgehog LinuxARKIME_SPI_DATA_MAX_INDICEStoarkime.envwith a default value of7, which manifests asspiDataMaxIndexesin Arkime's config.ini. If you are changing the Arkime index period from daily to weekly, hourly, etc., you may wish to adjust this value. (idaholab/Malcolm#471)EXTRA_TAGStoupload-common.envfor specifying custom tags to be associated with logs forwarded to Logstash by FileBeat. (idaholab/Malcolm#463)zeek-live.envfor tuning the resource utilization and performance of Zeek's live capture, see the documentation for details. (idaholab/Malcolm#475)control_vars.conffor tuning the resource utilization and performance of Zeek's live capture, see the documentation for details. (idaholab/Malcolm#475)control_vars.conffor tuning capture-related settings for Arkime on the sensor, bringing them into parity with those that were available in thearkime-livecontainer in Malcolm. (idaholab/Malcolm#476)