Malcolm report to itself on capture statistics

[mmguero]

Development Update:

---

Zeek

The following environment variables have been added, with these defaults:

• logstash.env

  # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs)
  LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout

• zeek-live.env

  # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log
  ZEEK_DISABLE_STATS=true

So, if you wanted to enable capture stats and other diagnostic logs for zeek, you could change it to something like:

• logstash.env

  # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs)
  LOGSTASH_ZEEK_IGNORED_LOGS=bsap_ip_unknown,bsap_serial_unknown,ecat_arp_info,loaded_scripts,png,stderr,stdout

• zeek-live.env

  # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log
  ZEEK_DISABLE_STATS=

The zeek diagnostic logs end up in indices like malcolm_beats_zeek_*:

I have not created dashboards for them, but the data is there.

Suricata

The following environment variables have been added, with these defaults:

• suricata-live.env

  # Whether or not enable capture statistics and include them in eve.json
  SURICATA_STATS_ENABLED=false
  SURICATA_STATS_EVE_ENABLED=false
  SURICATA_STATS_INTERVAL=30
  SURICATA_STATS_DECODER_EVENTS=false

The first two variables are used to enable stats and the inclusion of those stats in the EVE json. The interval is the stats logging interval in seconds. The decoder-events value should be able to turn on and off the decoder events in the stats log but it does not appear to be working. I may have to manually handle it in Logstash.

The suricata stats logs end up in indices like malcolm_beats_suricata_*:

Arkime

Arkime's already has this info in the Capture Stats table in the Stats tab. I'm not going to reinvent the wheel, so there's nothing to do here.

Original issue text:

---

For live capture (both with Malcolm doing its own capture and with Hedgehog) Zeek, Suricata, and Arkime-capture all have some statistics they could report. Zeek's is in stats.log, I'd have to track down where Suricata and Arkime's would be best pulled from.

It would be useful to report these to Malcolm for processing through the "other logs" pipeline (which is where things like sensor metrics, nginx access logs, etc., go).

Today we're just dropping the stats log for Zeek, so we'd need to detect that and route it to the other pipeline somehow instead.

We'd probably want to come up with specific capture-stats dashboards for this as well.

[mmguero]

Need to allow setting stats enabled via environment variable for suricata.