search

Security-Onion-Solutions / securityonion

Security Onion is a free and open platform for threat hunting, enterprise security monitoring, and log management. It includes our own interfaces for alerting, dashboards, hunting, PCAP, detections, and case management. It also includes other tools such as osquery, CyberChef, Elasticsearch, Logstash, Kibana, Suricata, and Zeek.
https://securityonion.net
3.28k stars 507 forks source link

so-steno docker container missing until force start, then error. #5071

Closed w4rc0n closed 3 years ago

w4rc0n commented 3 years ago

Fresh install of a sensor on Ubuntu 18.04.5 ISO directly from Canonical, all default settings besides RAID1 for the root partition, and /nsm being mounted to a zfs pool.

The rest of the environment includes one manager node, and one search node.

Installed according to documentation, aka: git clone https://github.com/Security-Onion-Solutions/securityonion cd securityonion sudo bash so-setup-network

Installer did report that an interface was in use, and to make it unmanaged. Which did also according to the documentation: Remove interface from netplan then: sudo netplan apply sudo touch /etc/NetworkManager/conf.d/10-globally-managed-devices.conf sudo service network-manager restart

Installer no longer alerted about the interface. Installation only yields a single error: [ERROR ] 'mine.send': False

From here, all docker containers start and remain healthy with the exception of so-steno.

Manually starting so-steno yields a healthy looking state application:

root@sensor:/opt/so/log/stenographer# so-pcap-start 
=========================================================================
Starting steno...

This could take a while if another Salt job is running. 
Run this command with --force to stop all Salt jobs before proceeding.
=========================================================================
local:
----------
          ID: stenographergroup
    Function: group.present
        Name: stenographer
      Result: True
     Comment: Group stenographer is present and up to date
     Started: 00:21:07.087987
    Duration: 6.672 ms
     Changes:   
----------
          ID: stenographer
    Function: user.present
      Result: True
     Comment: User stenographer is present and up to date
     Started: 00:21:07.095509
    Duration: 33.699 ms
     Changes:   
----------
          ID: stenoconfdir
    Function: file.directory
        Name: /opt/so/conf/steno
      Result: True
     Comment: The directory /opt/so/conf/steno is in the correct state
     Started: 00:21:07.132113
    Duration: 2.256 ms
     Changes:   
----------
          ID: stenoconf
    Function: file.managed
        Name: /opt/so/conf/steno/config
      Result: True
     Comment: File /opt/so/conf/steno/config is in the correct state
     Started: 00:21:07.134566
    Duration: 46.114 ms
     Changes:   
----------
          ID: stenoca
    Function: file.directory
        Name: /opt/so/conf/steno/certs
      Result: True
     Comment: The directory /opt/so/conf/steno/certs is in the correct state
     Started: 00:21:07.181044
    Duration: 2.5 ms
     Changes:   
----------
          ID: pcapdir
    Function: file.directory
        Name: /nsm/pcap
      Result: True
     Comment: The directory /nsm/pcap is in the correct state
     Started: 00:21:07.183745
    Duration: 2.629 ms
     Changes:   
----------
          ID: pcaptmpdir
    Function: file.directory
        Name: /nsm/pcaptmp
      Result: True
     Comment: The directory /nsm/pcaptmp is in the correct state
     Started: 00:21:07.186569
    Duration: 2.331 ms
     Changes:   
----------
          ID: pcapoutdir
    Function: file.directory
        Name: /nsm/pcapout
      Result: True
     Comment: The directory /nsm/pcapout is in the correct state
     Started: 00:21:07.189116
    Duration: 3.265 ms
     Changes:   
----------
          ID: pcapindexdir
    Function: file.directory
        Name: /nsm/pcapindex
      Result: True
     Comment: The directory /nsm/pcapindex is in the correct state
     Started: 00:21:07.192686
    Duration: 3.351 ms
     Changes:   
----------
          ID: stenolog
    Function: file.directory
        Name: /opt/so/log/stenographer
      Result: True
     Comment: The directory /opt/so/log/stenographer is in the correct state
     Started: 00:21:07.196343
    Duration: 2.997 ms
     Changes:   
----------
          ID: so-steno
    Function: docker_container.running
      Result: True
     Comment: Created container 'so-steno'
     Started: 00:21:07.275159
    Duration: 1562.519 ms
     Changes:   
              ----------
              container_id:
                  ----------
                  added:
                      0c4a9867f6cb4073f34f0b40d542f95ab2551d713c13905367a27ae48eb3a97a
              state:
                  ----------
                  new:
                      running
                  old:
                      None
----------
          ID: append_so-steno_so-status.conf
    Function: file.append
        Name: /opt/so/conf/so-status/so-status.conf
      Result: True
     Comment: unless condition is true
     Started: 00:21:08.838279
    Duration: 2789.361 ms
     Changes:   
----------
          ID: delete_so-steno_so-status.disabled
    Function: file.uncomment
        Name: /opt/so/conf/so-status/so-status.conf
      Result: True
     Comment: Pattern already uncommented
     Started: 00:21:11.628133
    Duration: 10.129 ms
     Changes:   

Summary for local
-------------
Succeeded: 13 (changed=1)
Failed:     0
-------------
Total states run:     13
Total run time:    4.468 s

Attempting to run the docker image alone yields:

root@sensor:/nsm/pcap# docker run a3bf7460b9f8
Generating CA state
Generating key/cert for 'client'
Generating key/cert for 'server'
-bash: /var/log/stenographer/stenographer.log: Permission denied

stenographer.log:

root@sensor:/opt/so/log/stenographer# cat stenographer.log 
2021/08/05 00:01:56 Deleted stale output file "/nsm/pcap/.1628121604057373"
2021-08-05T00:01:56.668044Z T:711cf7 [stenotype.cc:567] Starting, page size is 4096
2021-08-05T00:01:56.668169Z T:711cf7 [stenotype.cc:594] Setting up AF_PACKET sockets for packet reading
2021-08-05T00:01:57.231698Z T:711cf7 [stenotype.cc:262] Dropping privileges
2021-08-05T00:01:57.231743Z T:711cf7 [stenotype.cc:267] Dropping priviledges from 941 to GID stenographer
2021-08-05T00:01:57.232081Z T:711cf7 [stenotype.cc:279] Dropping priviledges from 941 to UID stenographer
2021-08-05T00:01:57.238191Z T:eecf77 [stenotype.cc:466] Thread 0 starting to process packets
2021-08-05T00:01:57.238472Z T:eecf77 [aio.cc:190] Opening packet file /tmp/stenographer814677620/PKT0/.1628121717238325: -1
2021-08-05T00:01:57.238494Z T:eecf77 [stenotype.cc:478] CHECK(SUCCEEDED(__check_success_error__)) output.Rotate(file_dirname, micros, flag_preallocate_file_mb << 20): Invalid argument <- open
ABORTABORTABORT
/usr/bin/stenotype() [0x4081f8]
/usr/bin/stenotype() [0x430568]
/lib64/libstdc++.so.6(+0xb5330) [0x7fe0700a7330]
/lib64/libpthread.so.0(+0x7ea5) [0x7fe070748ea5]
/lib64/libc.so.6(clone+0x6d) [0x7fe06f80a9fd]
2021/08/05 00:01:57 Stenotype stopped after 1.015670448s: stenotype wait failed: signal: aborted (core dumped)
2021/08/05 00:01:57 Stenotype ran for too little time, crashing to avoid stenotype crash loop
2021/08/05 00:03:32 Deleted stale output file "/nsm/pcap/.1628121717238325"
2021-08-05T00:03:32.791351Z T:70f067 [stenotype.cc:567] Starting, page size is 4096
2021-08-05T00:03:32.791464Z T:70f067 [stenotype.cc:594] Setting up AF_PACKET sockets for packet reading
2021-08-05T00:03:33.423311Z T:70f067 [stenotype.cc:262] Dropping privileges
2021-08-05T00:03:33.423361Z T:70f067 [stenotype.cc:267] Dropping priviledges from 941 to GID stenographer
2021-08-05T00:03:33.423726Z T:70f067 [stenotype.cc:279] Dropping priviledges from 941 to UID stenographer
2021-08-05T00:03:33.442311Z T:eea2e7 [stenotype.cc:466] Thread 0 starting to process packets
2021-08-05T00:03:33.442926Z T:eea2e7 [aio.cc:190] Opening packet file /tmp/stenographer207587303/PKT0/.1628121813442538: -1
2021-08-05T00:03:33.442950Z T:eea2e7 [stenotype.cc:478] CHECK(SUCCEEDED(__check_success_error__)) output.Rotate(file_dirname, micros, flag_preallocate_file_mb << 20): Invalid argument <- open
ABORTABORTABORT
/usr/bin/stenotype() [0x4081f8]
/usr/bin/stenotype() [0x430568]
/lib64/libstdc++.so.6(+0xb5330) [0x7f956fdde330]
/lib64/libpthread.so.0(+0x7ea5) [0x7f957047fea5]
/lib64/libc.so.6(clone+0x6d) [0x7f956f5419fd]
2021/08/05 00:03:33 Stenotype stopped after 1.12630958s: stenotype wait failed: signal: aborted (core dumped)
2021/08/05 00:03:33 Stenotype ran for too little time, crashing to avoid stenotype crash loop
2021-08-05T00:05:04.920673Z T:5bcdc7 [stenotype.cc:567] Starting, page size is 4096
2021-08-05T00:05:04.920831Z T:5bcdc7 [stenotype.cc:594] Setting up AF_PACKET sockets for packet reading
2021-08-05T00:05:05.558486Z T:5bcdc7 [stenotype.cc:262] Dropping privileges
2021-08-05T00:05:05.558557Z T:5bcdc7 [stenotype.cc:267] Dropping priviledges from 941 to GID stenographer
2021-08-05T00:05:05.558970Z T:5bcdc7 [stenotype.cc:279] Dropping priviledges from 941 to UID stenographer
2021-08-05T00:05:05.559986Z T:d98047 [stenotype.cc:466] Thread 0 starting to process packets
2021-08-05T00:05:05.560413Z T:d98047 [aio.cc:190] Opening packet file /tmp/stenographer709499317/PKT0/.1628121905560171: -1
2021-08-05T00:05:05.560438Z T:d98047 [stenotype.cc:478] CHECK(SUCCEEDED(__check_success_error__)) output.Rotate(file_dirname, micros, flag_preallocate_file_mb << 20): Invalid argument <- open
ABORTABORTABORT
/usr/bin/stenotype() [0x4081f8]
/usr/bin/stenotype() [0x430568]
/lib64/libstdc++.so.6(+0xb5330) [0x7f9a5abb4330]
/lib64/libpthread.so.0(+0x7ea5) [0x7f9a5b255ea5]
/lib64/libc.so.6(clone+0x6d) [0x7f9a5a3179fd]
2021/08/05 00:05:06 Stenotype stopped after 1.084316041s: stenotype wait failed: signal: aborted (core dumped)
2021/08/05 00:05:06 Stenotype ran for too little time, crashing to avoid stenotype crash loop
2021/08/05 00:05:37 Deleted stale output file "/nsm/pcap/.1628121905560171"
2021-08-05T00:05:37.135225Z T:93c867 [stenotype.cc:567] Starting, page size is 4096
2021-08-05T00:05:37.135362Z T:93c867 [stenotype.cc:594] Setting up AF_PACKET sockets for packet reading
2021-08-05T00:05:37.735371Z T:93c867 [stenotype.cc:262] Dropping privileges
2021-08-05T00:05:37.735416Z T:93c867 [stenotype.cc:267] Dropping priviledges from 941 to GID stenographer
2021-08-05T00:05:37.735748Z T:93c867 [stenotype.cc:279] Dropping priviledges from 941 to UID stenographer
2021-08-05T00:05:37.737515Z T:117ae7 [stenotype.cc:466] Thread 0 starting to process packets
2021-08-05T00:05:37.737762Z T:117ae7 [aio.cc:190] Opening packet file /tmp/stenographer030801244/PKT0/.1628121937737595: -1
2021-08-05T00:05:37.737793Z T:117ae7 [stenotype.cc:478] CHECK(SUCCEEDED(__check_success_error__)) output.Rotate(file_dirname, micros, flag_preallocate_file_mb << 20): Invalid argument <- open
ABORTABORTABORT
/usr/bin/stenotype() [0x4081f8]
/usr/bin/stenotype() [0x430568]
/lib64/libstdc++.so.6(+0xb5330) [0x7f5c92b5e330]
/lib64/libpthread.so.0(+0x7ea5) [0x7f5c931ffea5]
/lib64/libc.so.6(clone+0x6d) [0x7f5c922c19fd]
2021/08/05 00:05:38 Stenotype stopped after 1.013770085s: stenotype wait failed: signal: aborted (core dumped)
2021/08/05 00:05:38 Stenotype ran for too little time, crashing to avoid stenotype crash loop
2021/08/05 00:21:08 Deleted stale output file "/nsm/pcap/.1628121937737595"
2021-08-05T00:21:08.563097Z T:9895b7 [stenotype.cc:567] Starting, page size is 4096
2021-08-05T00:21:08.563206Z T:9895b7 [stenotype.cc:594] Setting up AF_PACKET sockets for packet reading
2021-08-05T00:21:09.107823Z T:9895b7 [stenotype.cc:262] Dropping privileges
2021-08-05T00:21:09.107886Z T:9895b7 [stenotype.cc:267] Dropping priviledges from 941 to GID stenographer
2021-08-05T00:21:09.108198Z T:9895b7 [stenotype.cc:279] Dropping priviledges from 941 to UID stenographer
2021-08-05T00:21:09.109004Z T:164837 [stenotype.cc:466] Thread 0 starting to process packets
2021-08-05T00:21:09.109289Z T:164837 [aio.cc:190] Opening packet file /tmp/stenographer312058502/PKT0/.1628122869109161: -1
2021-08-05T00:21:09.109307Z T:164837 [stenotype.cc:478] CHECK(SUCCEEDED(__check_success_error__)) output.Rotate(file_dirname, micros, flag_preallocate_file_mb << 20): Invalid argument <- open
ABORTABORTABORT
/usr/bin/stenotype() [0x4081f8]
/usr/bin/stenotype() [0x430568]
/lib64/libstdc++.so.6(+0xb5330) [0x7f9d97833330]
/lib64/libpthread.so.0(+0x7ea5) [0x7f9d97ed4ea5]
/lib64/libc.so.6(clone+0x6d) [0x7f9d96f969fd]
2021/08/05 00:21:09 Stenotype stopped after 918.029961ms: stenotype wait failed: signal: aborted (core dumped)
2021/08/05 00:21:09 Stenotype ran for too little time, crashing to avoid stenotype crash loop

A full highstate yields no failed states. I can provide that output if needed, but will refrain for now due to the length of said output.

At a loss from how to troubleshoot from here. Let me know what other information I can provide to help get to the bottom of this.

w4rc0n commented 3 years ago

Ran the installer again, this time no errors of any kind in sosetup.log but same issue.

TOoSmOotH commented 3 years ago

Its a known issue with ZFS and steno:

https://github.com/Security-Onion-Solutions/securityonion/discussions/4203

w4rc0n commented 3 years ago

Man, I tried so hard to find any mention of it here. Thank you!