Test Zeek 3.0.1, Elastic 6.8.6, and related updates

[dougburks]

List of packages to be tested:

• securityonion-bro - 3.0.1-1ubuntu1securityonion10 (Zeek 3.0.1)
• securityonion-bro-afpacket - 1.3.0-1ubuntu1securityonion17
• securityonion-bro-scripts - 20121004-0ubuntu0securityonion100
• securityonion-elastic - 20190510-1ubuntu1securityonion83
• securityonion-nsmnow-admin-scripts - 20120724-0ubuntu0securityonion225
• securityonion-onionsalt - 20140917-0ubuntu0securityonion28
• securityonion-samples-bro - 20170824-1ubuntu1securityonion4
• securityonion-setup - 20120912-0ubuntu0securityonion325
• securityonion-sostat - 20120722-0ubuntu0securityonion141
• securityonion-tcpudpflow - 001-0ubuntu0securityonion10
• securityonion-web-page - 20141015-0ubuntu0securityonion105

List of Docker images to be tested:

• Elasticsearch 6.8.6 (both OSS and Elastic Features versions)
• Logstash 6.8.6 (both OSS and Elastic Features versions)
• Kibana 6.8.6 (both OSS and Elastic Features versions)
• Freqserver
• Domainstats
• ElastAlert
• Curator

An overview of the testing process can be found in the comments below. For a complete list of Issues to be tested, please see the Testing/Tested columns at https://github.com/Security-Onion-Solutions/security-onion/projects/10

Please record all testing results via comments on this issue or the individual issues at https://github.com/Security-Onion-Solutions/security-onion/projects/10

Thanks in advance for your time effort!

[dougburks]

How To Start Testing

• install the current 16.04 ISO image

• snapshot the VM if possible

• add the test PPA:

  sudo add-apt-repository -y ppa:securityonion/test

• change DOCKERHUB from "securityonionsolutions" to "securityonionsolutionstest" (OSS license):

  sudo sed -i 's|DOCKERHUB="securityonionsolutions"|DOCKERHUB="securityonionsolutionstest"|g' /etc/nsm/elasticdownload.conf

• update:

  sudo soup

[dougburks]

How To Verify Proper Zeek Operation

• first, please note that Bro has been renamed to Zeek but the packages still adhere to the traditional bro naming convention

• as the Zeek packages install, verify that the package installation scripts display a message about checking configuration and adding back any local customizations

• verify that Bro packages were upgraded:

  dpkg -l |grep securityonion-bro

• verify that the new Zeek packages create symlinks as necessary so that the new zeek paths resolve to the traditional bro locations (for example: /opt/zeek is a symlink to /opt/bro, /nsm/zeek is a symlink to /nsm/bro, etc.)

• verify that the new Zeek packages create symlinks as necessary so that well known bro files resolve to the new zeek locations (for example: /opt/bro/etc/broctl.cfg is a symlink to zeekctl.cfg, so it can be accessed via /opt/bro/etc/broctl.cfg or /opt/zeek/etc/zeekctl.cfg)

• if new installation, run through Setup

• verify that the package installation scripts backed up the following with a _pre-3.0.1 extension: /opt/bro/etc/ /opt/bro/share/bro/

• verify that StatusCmdShowAll has been set to 0 in /opt/zeek/etc/zeekctl.cfg:

  grep StatusCmdShowAll /opt/zeek/etc/zeekctl.cfg

• verify that "lb_custom.InterfacePrefix=af_packet::" has been added to /opt/zeek/etc/zeekctl.cfg:

  grep af_packet /opt/zeek/etc/zeekctl.cfg

• Restart Zeek:

  sudo so-zeek-restart

• check status:

  sudo so-status

• check Zeek startup logs for any warnings/errors out of the ordinary:

  cat /nsm/zeek/logs/current/reporter.log
  cat /nsm/zeek/logs/current/stdout.log
  cat /nsm/zeek/logs/current/stderr.log

• replay LOTS of traffic:

  sudo so-test

• verify that files are extracted to /nsm/zeek/extracted:

  ls -alh /nsm/zeek/extracted

• verify that /nsm/zeek/logs/current/conn.log contains the proper sensorname at the end of each log entry:

  cat /nsm/zeek/logs/current/conn.log

• verify that Zeek logs are in the same format as they were pre-upgrade (should be JSON by default).

• verify that the Elastic Stack is parsing and displaying Zeek logs properly (whether JSON or TSV format)

• verify that you can pivot to CapMe for both TCP and UDP traffic

• check sostat output for anything out of the ordinary (specifically, check the pf_ring and Zeek sections for packet loss)

• verify that Zeek ja3 script is loaded and logging:

  grep ja3 /nsm/zeek/logs/current/*

• verify that Zeek hassh script is loaded and logging:

  grep hassh /nsm/zeek/logs/current/*

• verify that /etc/cron.d/bro has been moved to /etc/cron.d/zeek and that it works properly

• verify that everything else works properly with no regressions

• reboot and make sure everything still works properly

Please test in as many different combinations as possible:

• Evaluation Mode (Bro Standalone mode) vs Production Mode (Bro cluster mode)

• single sniffing interface vs multiple sniffing interfaces

• file extraction enabled or disabled

• json-logs enabled or disabled

• traffic without vlan tags vs traffic with vlan tags

• new installation vs upgrade

• Zeek cluster mode - PF_RING (lb_method=pf_ring) vs AF_PACKET (lb_method=custom)

[dougburks]

How To Verify Proper Elastic Operation

Please test in as many different combinations as possible:

• verify that traditional Logstash parsing correctly parses Zeek logs in JSON or TSV format

• verify that Elasticsearch Ingest parsing correctly parses Zeek logs in JSON format

• verify Kibana dashboards visualize those parsed logs correctly

• check elasticsearch, logstash, and kibana logs in /var/log/ for anything out of the ordinary

• so-import-pcap vs sosetup-minimal vs traditional Setup

• Setup GUI vs CLI

• Evaluation Mode vs Production Mode

• standalone vs distributed deployments

• new installation vs upgrade

• Elastic OSS vs Elastic Features license (use so-elastic-features to switch from OSS to Features)

• SSO vs Elastic native auth (use so-elastic-auth to switch to Elastic native auth)

[weslambert]

Tested in various configurations without any issues.

[bryant-treacle]

broctl & zeekctl script in /usr/sbin/ not functioning properly. No text being displayed. I can still launch them from their native directory. It looks like an issue is in the grep -v after we call the function. It work fine when I comment it out.

Everything else seems to work without issues.

[chris-cuevas]

No issues seen in my testing with more than 70,000,000 events per hour.

As per the checklist for testing...

dpkg -l |grep securityonion-bro ii securityonion-bro 3.0.1-1ubuntu1securityonion10 amd64 The Bro Network Security Monitor ii securityonion-bro-afpacket 1.3.0-1ubuntu1securityonion17 all Plugin providing native AF_Packet support for Bro. ii securityonion-bro-scripts 20121004-0ubuntu0securityonion100 all Bro scripts for Security Onion

root@test-host1:~# ls -l /opt/zeek lrwxrwxrwx 1 root root 3 Feb 4 13:06 /opt/zeek -> bro

root@test-host1:~# ls -l /nsm/zeek lrwxrwxrwx 1 root root 3 Feb 4 13:06 /nsm/zeek -> bro

root@test-host1:~# ls -l /opt/bro/etc/broctl.cfg lrwxrwxrwx 1 root root 11 Feb 4 13:06 /opt/bro/etc/broctl.cfg -> zeekctl.cfg

root@test-host1:~# ls -l /opt/bro/ total 0 drwxr-xr-x 2 root root 257 Feb 4 13:06 bin drwxr-xr-x 2 root root 101 Feb 4 13:13 etc drwxr-xr-x 2 root root 60 Sep 17 18:26 etc_pre-2.6.4 drwxr-xr-x 2 root root 60 Feb 4 13:04 etc_pre-3.0.1

root@test-host1:~# ls -l /opt/bro/share/ total 0 lrwxrwxrwx 1 root root 4 Feb 4 13:06 bro -> zeek drwxr-xr-x 4 root root 31 Feb 4 13:06 bro.pre-3.0.1

root@test-host1:~# grep StatusCmdShowAll /opt/zeek/etc/zeekctl.cfg StatusCmdShowAll = 0

root@test-host1:~# grep af_packet /opt/zeek/etc/zeekctl.cfg lb_custom.InterfacePrefix=af_packet::

root@test-host1:~# ls /etc/cron.d/ anacron capme mdadm netsniff-sync nsm-watchdog php salt-update sensor-clean sensor-newday sguil-db-purge so-sensor-backup-config so-server-backup-config squert-ip2c sysstat zeek

I have rebooted the system and things come up smoothly on reboot.

I had no issue with with zeekctl or broctl displaying text but I only ran a zeekctl config so didn't test that extensively.

[dougburks]

Thanks @chris-cuevas !

[dougburks]

Published: https://blog.securityonion.net/2020/02/zeek-301-elastic-686-and-cyberchef-9120.html