Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files), Zeek logs and Suricata alerts.
Certain features cause different fields to be indexed whether they're enabled or disabled. These might include:
NetBox features (some of these will need to be checked combinatorially)
NETBOX_ENRICHMENT in netbox-common.env
NETBOX_AUTO_POPULATE and NETBOX_AUTO_CREATE_PREFIX and NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER in netbox-common.env (probably just test with them set to true)
Other enrichments
FREQ_LOOKUP in lookup-common.env
LOGSTASH_OUI_LOOKUP in logstash.env
LOGSTASH_SEVERITY_SCORING in logstash.env
LOGSTASH_REVERSE_DNS in logstash.env
LOGSTASH_NETBOX_ENRICHMENT_DATASETS in logstash.env
test this with NETBOX_ENRICHMENT on and test it with the defaults vs all for this value
What I'd like to know is how storage space is affected based on enabling/disabling these things are. So basically set up Malcolm, ingest a bunch of PCAP and then do a du on the opensearch directory with and without these features enabled, and report on the differences.
In addition to measuring disk usage, I'd also like to profile the logstash pipelines with these features enabled disabled. So after ingesting the pcap, take this measurement:
which will give you a breakdown of how much time was spent in each Logstash Filter.
The one I'm most particularly interested in is the NetBox one, with NetBox enabled and auto-populate turned on, test LOGSTASH_NETBOX_ENRICHMENT_DATASETS with the defaults and then test it again with all. I'd like to know how much more time is added and how much more disk space is used if we enrich ALL log types from NetBox.
Certain features cause different fields to be indexed whether they're enabled or disabled. These might include:
NETBOX_ENRICHMENTinnetbox-common.envNETBOX_AUTO_POPULATEandNETBOX_AUTO_CREATE_PREFIXandNETBOX_DEFAULT_AUTOCREATE_MANUFACTURERinnetbox-common.env(probably just test with them set totrue)FREQ_LOOKUPinlookup-common.envLOGSTASH_OUI_LOOKUPinlogstash.envLOGSTASH_SEVERITY_SCORINGinlogstash.envLOGSTASH_REVERSE_DNSinlogstash.envLOGSTASH_NETBOX_ENRICHMENT_DATASETSinlogstash.envNETBOX_ENRICHMENTon and test it with the defaults vsallfor this valueWhat I'd like to know is how storage space is affected based on enabling/disabling these things are. So basically set up Malcolm, ingest a bunch of PCAP and then do a
duon the opensearch directory with and without these features enabled, and report on the differences.In addition to measuring disk usage, I'd also like to profile the logstash pipelines with these features enabled disabled. So after ingesting the pcap, take this measurement:
which will give you a breakdown of how much time was spent in each Logstash Filter.
The one I'm most particularly interested in is the NetBox one, with NetBox enabled and auto-populate turned on, test
LOGSTASH_NETBOX_ENRICHMENT_DATASETSwith the defaults and then test it again withall. I'd like to know how much more time is added and how much more disk space is used if we enrich ALL log types from NetBox.