Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files), Zeek logs and Suricata alerts.
Because some of the environment variables used for configuring Malcolm have been reorganized in the .env files found in the ./config directory, it is recommended you re-run ./scripts/configure for this release.
Improvements to creation of index templates, dashboards, and other saved objects on startup (idaholab/Malcolm#208) to ensure that saved objects get created correctly upon upgrade (see this comment for more details on this feature).
Populating the NetBox inventory via passively-gathered network traffic metadata now uses network traffic logs for DNS, NTLM, and DHCP to identify assets' host names when possible for use when populating device and VM names (idaholab/Malcolm#415). Autopopulated devices now have their status field set to Active rather than Stage, and uses tags instead to indicated that they were created through autopopulation.
Users can now specify pruning thresholds for carved files so that old files are deleted in order to avoid filling available storage (idaholab/Malcolm#453). See a new section of documentation on Managing disk usage for more information about this and similar settings.
Users can now specify a prefix that will be prepended to dashboards as they are imported into OpenSearch Dashboards or Kibana, allowing users who have dashboards from other sources to differentiate between those and Malcolm's (idaholab/Malcolm#455).
The default anomaly detectors created for the OpenSearch Anomaly Detection plugin are now created with category fields for high cardinality to allow for better breakdown of contributing values to anomalies discovered (idaholab/Malcolm#464).
Include JA4+ plugin in Arkime. See idaholab/Malcolm#419 for status on upcoming full JA4+ support in Malcolm.
Hedgehog Linux sensors can now periodically refresh their Zeek inteligence files. NOTE: due to an oversight, a necessary variable is missing in this release that is required for this to work. Appending the line export INTEL_DIR=/opt/sensor/sensor_ctl/zeek/intel to /opt/sensor/sensor_ctl/control_vars.conf will correct this. This will be corrected in the next Malcolm release.
The documentation for Windows host system configuration was out of date and has been updated for the latest version of Microsoft Windows Subsystem for Linux (idaholab/Malcolm#421).
An issue was fixed in which Malcolm's list of users and their password hashes could become corrupted if the file did not initially end with a newline character (idaholab/Malcolm#426).
The manner in which Zeek intel files are generated has been changed to avoid problems found in Kubernetes deployments when scaling out the number of zeek-live containers (idaholab/Malcolm#456). See this comment for more details.
Removed the version top-level element from docker-compose.yml files as it is now obsolete and caused a warning message that sometimes was not handled correctly.
Fix Malcolm ISO not correctly detecting if it's in a live boot ISO environment or installed mode.
Restart live Zeek instances with zeekctl deploy instead of zeekctl restart.
DASHBOARDS_PREFIX in dashboards-helper.env has been added for idaholab/Malcolm#455 (see above in Features and Enhancements).
LOGSTASH_NETBOX_ENRICHMENT_DATASETS in logstash.env has been changed to include zeek.dhcp, zeek.dns, and zeek.ntlm to support idaholab/Malcolm#415 (see above in Features and Enhancements).
LOGSTASH_ZEEK_IGNORED_LOGS in logstash.env has been changed to remove capture_loss and stats so that those diagnostic Zeek logs can be parsed without the user having to manually change this variable.
ZEEK_CRON has been removed from zeek-live.env and ZEEK_INTEL_REFRESH_CRON_EXPRESSION was removed from zeek.env and moved to the "offline" version of the container in zeek-offline.env for idaholab/Malcolm#456.
EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE, EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT, and EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS were added to zeek.env for idaholab/Malcolm#453. See a new section of documentation on Managing disk usage for more information about these and similar settings.
Malcolm v24.04.0 contains new features, improvements, bug fixes and component version updates.
https://github.com/cisagov/Malcolm/compare/v24.03.1...v24.04.0
Because some of the environment variables used for configuring Malcolm have been reorganized in the
.envfiles found in the./configdirectory, it is recommended you re-run./scripts/configurefor this release.Activerather thanStage, and uses tags instead to indicated that they were created through autopopulation.export INTEL_DIR=/opt/sensor/sensor_ctl/zeek/intelto/opt/sensor/sensor_ctl/control_vars.confwill correct this. This will be corrected in the next Malcolm release.zeek-livecontainers (idaholab/Malcolm#456). See this comment for more details.docker-compose.ymlfiles as it is now obsolete and caused a warning message that sometimes was not handled correctly.zeekctl deployinstead ofzeekctl restart../config/)ARKIME_QUERY_ALL_INDICESinarkime.envcan be set to control thequeryAllIndicessetting in Arkime'sconfig.ini.DASHBOARDS_PREFIXindashboards-helper.envhas been added for idaholab/Malcolm#455 (see above in Features and Enhancements).LOGSTASH_NETBOX_ENRICHMENT_DATASETSinlogstash.envhas been changed to includezeek.dhcp,zeek.dns, andzeek.ntlmto support idaholab/Malcolm#415 (see above in Features and Enhancements).LOGSTASH_ZEEK_IGNORED_LOGSinlogstash.envhas been changed to removecapture_lossandstatsso that those diagnostic Zeek logs can be parsed without the user having to manually change this variable.ZEEK_CRONhas been removed fromzeek-live.envandZEEK_INTEL_REFRESH_CRON_EXPRESSIONwas removed fromzeek.envand moved to the "offline" version of the container inzeek-offline.envfor idaholab/Malcolm#456.EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE,EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT, andEXTRACTED_FILE_PRUNE_INTERVAL_SECONDSwere added tozeek.envfor idaholab/Malcolm#453. See a new section of documentation on Managing disk usage for more information about these and similar settings.