search

clong / DetectionLab

Automate the creation of a lab environment complete with security tooling and logging best practices
MIT License
4.62k stars 984 forks source link

Logger builds are failing to due Bro pre-reqs #313

Closed clong closed 5 years ago

clong commented 5 years ago

Please verify that you are building from an updated Master branch before filing an issue.

Description of the issue:

Logger fails to build

Link to Gist Containing Build Logs:

    logger: You are using pip version 8.1.1, however version 19.1.1 is available.
    logger: You should consider upgrading via the 'pip install --upgrade pip' command.
    logger: Traceback (most recent call last):
    logger:   File "/usr/local/bin/zkg", line 4, in <module>
    logger:     from zeekpkg._util import (
    logger:   File "/usr/local/lib/python2.7/dist-packages/zeekpkg/__init__.py", line 18, in <module>
    logger:     from .manager import *
    logger:   File "/usr/local/lib/python2.7/dist-packages/zeekpkg/manager.py", line 27, in <module>
    logger:     import git
    logger:   File "/usr/local/lib/python2.7/dist-packages/git/__init__.py", line 46, in <module>
    logger:     from git.repo import Repo               # @NoMove @IgnorePep8
    logger:   File "/usr/local/lib/python2.7/dist-packages/git/repo/__init__.py", line 4, in <module>
    logger:     from .base import *
    logger:   File "/usr/local/lib/python2.7/dist-packages/git/repo/base.py", line 7, in <module>
    logger:     from builtins import str
    logger: ImportError: 
    logger: No module named builtins
    logger: Traceback (most recent call last):
    logger:   File "/usr/local/bin/zkg", line 4, in <module>
    logger:     
    logger: from zeekpkg._util import (
    logger:   File "/usr/local/lib/python2.7/dist-packages/zeekpkg/__init__.py", line 18, in <module>
    logger:     
    logger: from .manager import *
    logger:   File "/usr/local/lib/python2.7/dist-packages/zeekpkg/manager.py", line 27, in <module>
    logger:     
    logger: import git
    logger:   File "/usr/local/lib/python2.7/dist-packages/git/__init__.py", line 46, in <module>
    logger:     
    logger: from git.repo import Repo               # @NoMove @IgnorePep8
    logger:   File "/usr/local/lib/python2.7/dist-packages/git/repo/__init__.py", line 4, in <module>
    logger:     
    logger: from .base import *
    logger:   File "/usr/local/lib/python2.7/dist-packages/git/repo/base.py", line 7, in <module>
    logger:     
    logger: from builtins import str
    logger: ImportError
    logger: : 
    logger: No module named builtins
    logger: Traceback (most recent call last):
    logger:   File "/usr/local/bin/zkg", line 4, in <module>
    logger:     from zeekpkg._util import (
    logger:   File "/usr/local/lib/python2.7/dist-packages/zeekpkg/__init__.py", line 18, in <module>
    logger:     from .manager import *
    logger:   File "/usr/local/lib/python2.7/dist-packages/zeekpkg/manager.py", line 27, in <module>
    logger:     import git
    logger:   File "/usr/local/lib/python2.7/dist-packages/git/__init__.py", line 46, in <module>
    logger:     from git.repo import Repo               # @NoMove @IgnorePep8
    logger:   File "/usr/local/lib/python2.7/dist-packages/git/repo/__init__.py", line 4, in <module>
    logger:     from .base import *
    logger:   File "/usr/local/lib/python2.7/dist-packages/git/repo/base.py", line 7, in <module>
    logger:     from builtins import str
    logger: ImportError: No module named builtins
clong commented 5 years ago

This issue caused the most recent CI build to fail: https://circleci.com/gh/clong/DetectionLab/300#artifacts/containers/0

It should be fixed by https://github.com/clong/DetectionLab/commit/a07e9cbaac695c35dfeba90ecd8abe241578724a

4ndr3w6 commented 5 years ago

Hey @clong hope all is well. Just tried building a fresh local logger. The logger is failing to build on this issue again. Thought I would comment here (for history purposes), instead of making a new issue. Below are my logs:

 logger: W: Target Translations (universe/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list:5 and /etc/apt/sources.list:13
    logger:
    logger: W: Target Packages (multiverse/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list:5 and /etc/apt/sources.list:13
    logger: W: Target Packages (multiverse/binary-i386/Packages) is configured multiple times in /etc/apt/sources.list:5 and /etc/apt/sources.list:13
    logger: W: Target Packages (multiverse/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:5 and /etc/apt/sources.list:13
    logger: W: Target Translations (multiverse/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list:5 and /etc/apt/sources.list:13
    logger: W: Target Translations (multiverse/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list:5 and /etc/apt/sources.list:13
    logger: Requirement already satisfied (use --upgrade to upgrade): bro-pkg in /usr/local/lib/python2.7/dist-packages
    logger: Requirement already satisfied (use --upgrade to upgrade): future in /usr/local/lib/python2.7/dist-packages
    logger: Requirement already satisfied (use --upgrade to upgrade): gitpython in /usr/local/lib/python2.7/dist-packages (from bro-pkg)
    logger: Requirement already satisfied (use --upgrade to upgrade): semantic-version in /usr/local/lib/python2.7/dist-packages (from bro-pkg)
    logger: Requirement already satisfied (use --upgrade to upgrade): configparser in /usr/local/lib/python2.7/dist-packages (from bro-pkg)
    logger: Requirement already satisfied (use --upgrade to upgrade): btest in /usr/local/lib/python2.7/dist-packages (from bro-pkg)
    logger: Requirement already satisfied (use --upgrade to upgrade): gitdb2>=2.0.0 in /usr/local/lib/python2.7/dist-packages (from gitpython->bro-pkg)
    logger: Requirement already satisfied (use --upgrade to upgrade): smmap2>=2.0.0 in /usr/local/lib/python2.7/dist-packages (from gitdb2>=2.0.0->gitpython->bro-pkg)
    logger: You are using pip version 8.1.1, however version 19.2.3 is available.
    logger: You should consider upgrading via the 'pip install --upgrade pip' command.
    logger: Traceback (most recent call last):
    logger:   File "/usr/local/bin/zkg", line 4, in <module>
    logger:     from zeekpkg._util import (
    logger:   File "/usr/local/lib/python2.7/dist-packages/zeekpkg/__init__.py", line 18, in <module>
    logger:     from .manager import *
    logger:   File "/usr/local/lib/python2.7/dist-packages/zeekpkg/manager.py", line 28, in <module>
    logger:     import semantic_version as semver
    logger:   File "/usr/local/lib/python2.7/dist-packages/semantic_version/__init__.py", line 6, in <module>
    logger:     from .base import compare, match, validate, SimpleSpec, NpmSpec, Spec, SpecItem, Version
    logger:   File "/usr/local/lib/python2.7/dist-packages/semantic_version/base.py", line 87
    logger:     *,
    logger:      ^
    logger: SyntaxError: invalid syntax
    logger: Traceback (most recent call last):
    logger:   File "/usr/local/bin/zkg", line 4, in <module>
    logger:
    logger: from zeekpkg._util import (
    logger:   File "/usr/local/lib/python2.7/dist-packages/zeekpkg/__init__.py", line 18, in <module>
    logger:
    logger: from .manager import *
    logger:   File "/usr/local/lib/python2.7/dist-packages/zeekpkg/manager.py", line 28, in <module>
    logger:
    logger: import semantic_version as semver
    logger:   File "/usr/local/lib/python2.7/dist-packages/semantic_version/__init__.py", line 6, in <module>
    logger:
    logger: from .base import compare, match, validate, SimpleSpec, NpmSpec, Spec, SpecItem, Version
    logger:   File "
    logger: /usr/local/lib/python2.7/dist-packages/semantic_version/base.py
    logger: ", line
    logger: 87
    logger:
    logger: *,
    logger:
    logger:
    logger: ^
    logger: SyntaxError
    logger: :
    logger: invalid syntax
    logger: Traceback (most recent call last):
    logger:   File "/usr/local/bin/zkg", line 4, in <module>
    logger:     from zeekpkg._util import (
    logger:   File "/usr/local/lib/python2.7/dist-packages/zeekpkg/__init__.py", line 18, in <module>
    logger:     from .manager import *
    logger:   File "/usr/local/lib/python2.7/dist-packages/zeekpkg/manager.py", line 28, in <module>
    logger:     import semantic_version as semver
    logger:   File "/usr/local/lib/python2.7/dist-packages/semantic_version/__init__.py", line 6, in <module>
    logger:     from .base import compare, match, validate, SimpleSpec, NpmSpec, Spec, SpecItem, Version
    logger:   File "/usr/local/lib/python2.7/dist-packages/semantic_version/base.py", line 87
    logger:     *,
    logger:      ^
    logger: SyntaxError: invalid syntax
    logger: Job for bro.service failed because the control process exited with error code. See "systemctl status bro.service" and "journalctl -xe" for details.
    logger: fatal: destination path '/opt/splunk/etc/apps/TA-bro_json' already exists and is not an empty directory.
    logger: Stopping splunkd...
    logger: Shutting down.  Please wait, as this may take a few minutes.
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger:
    logger: Stopping splunk helpers...
    logger: Done.
    logger:
    logger: Splunk> The IT Search Engine.
    logger:
    logger: Checking prerequisites...
    logger:     Checking http port [8000]:
    logger: open
    logger:     Checking mgmt port [8089]:
    logger: open
    logger:     Checking appserver port [127.0.0.1:8065]:
    logger: open
    logger:     Checking kvstore port [8191]:
    logger: open
    logger:     Checking configuration...
    logger: Done.
    logger:     Checking critical directories...    Done
    logger:     Checking indexes...
    logger:         Validated: _audit _internal _introspection _telemetry _thefishbucket bro history json_bro main osquery osquery-status powershell summary suricata sysmon threathunting wineventlog
    logger:     Done
    logger:     Checking filesystem compatibility...  Done
    logger:     Checking conf files for problems...
    logger:     Done
    logger:     Checking default conf files for edits...
    logger:     Validating installed files against hashes from '/opt/splunk/splunk-7.3.1-bd63e13aa157-linux-2.6-x86_64-manifest'
    logger:     All installed files intact.
    logger:     Done
    logger: All preliminary checks passed.
    logger:
    logger: Starting splunk server daemon (splunkd)...
    logger: Done
    logger:
    logger: Waiting for web server at https://127.0.0.1:8000 to be available
    logger: .
    logger: . Done
    logger:
    logger:
    logger: If you get stuck, we're here to help.
    logger: Look for answers here: http://docs.splunk.com
    logger:
    logger: The Splunk web interface is at https://logger:8000
    logger: Bro attempted to start but is not running. Exiting
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.
clong commented 5 years ago

@4ndr3w6 looking into this now

clong commented 5 years ago

@4ndr3w6 I'm not able to reproduce this:

    logger: You are using pip version 8.1.1, however version 19.2.3 is available.
    logger: You should consider upgrading via the 'pip install --upgrade pip' command.
    logger: Refresh package source: zeek
    logger:     No changes
    logger: Refresh installed packages
    logger:     No new outdated packages
    logger: Successfully wrote config file to /root/.zkg/config
    logger: Installing "zeek/salesforce/ja3"
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger: .
    logger:
    logger: Installed "zeek/salesforce/ja3" (master)
    logger: Loaded "zeek/salesforce/ja3"

I'm concerned by this line from your output - it looks like somehow apt sources got defined multiple times.

 Target Translations (universe/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list:5 and /etc/apt/sources.list:13

Are you able to repro this if you completely destroy and recreate your logger host?

4ndr3w6 commented 5 years ago

LOL. Totally weird!! I swear, I did a complete destroy last night about 4 times. All good now. As always, thanks for quick turn around. :)

logger: Running setup.py bdist_wheel for btest: started logger: Running setup.py bdist_wheel for btest: finished with status 'done' logger: Stored in directory: /root/.cache/pip/wheels/69/1a/30/f129430e28f580c38ef102a8e6d236315b1356e3ee630f9453 logger: Successfully built future gitpython btest logger: Installing collected packages: smmap2, gitdb2, gitpython, semantic-version, configparser, btest, bro-pkg, future logger: Successfully installed bro-pkg-2.0.4 btest-0.59 configparser-3.8.1 future-0.17.1 gitdb2-2.0.5 gitpython-3.0.2 semantic-version-2.8.1 smmap2-2.0.5 logger: You are using pip version 8.1.1, however version 19.2.3 is available. logger: You should consider upgrading via the 'pip install --upgrade pip' command. logger: Refresh package source: zeek logger: No changes logger: Refresh installed packages logger: No new outdated packages logger: Successfully wrote config file to /root/.zkg/config logger: Installing "zeek/salesforce/ja3" logger: logger: Installed "zeek/salesforce/ja3" (master) logger: Loaded "zeek/salesforce/ja3" logger: Created symlink from /etc/systemd/system/multi-user.target.wants/bro.service to /lib/systemd/system/bro.service. logger: Cloning into '/opt/splunk/etc/apps/TA-bro_json'... logger: Stopping splunkd... logger: Shutting down. Please wait, as this may take a few minutes. logger: . logger: . logger: logger: Stopping splunk helpers... logger: Done. logger: logger: Splunk> The IT Search Engine. logger: logger: Checking prerequisites... logger: Checking http port [8000]: logger: open logger: Checking mgmt port [8089]: logger: open logger: Checking appserver port [127.0.0.1:8065]: logger: open logger: Checking kvstore port [8191]: logger: open logger: Checking configuration... logger: Done. logger: Checking critical directories... Done logger: Checking indexes... logger: Validated: _audit _internal _introspection _telemetry _thefishbucket bro history json_bro main osquery osquery-status powershell summary suricata sysmon threathunting wineventlog logger: Done logger: Checking filesystem compatibility... Done logger: Checking conf files for problems... logger: Done logger: Checking default conf files for edits... logger: Validating installed files against hashes from '/opt/splunk/splunk-7.3.1-bd63e13aa157-linux-2.6-x86_64-manifest' logger: All installed files intact. logger: Done logger: All preliminary checks passed. logger: logger: Starting splunk server daemon (splunkd)... logger: Done logger: logger: Waiting for web server at https://127.0.0.1:8000 to be available logger: . logger: . Done logger: logger: logger: If you get stuck, we're here to help. logger: Look for answers here: http://docs.splunk.com logger: logger: The Splunk web interface is at https://logger:8000