search

corelight / zeek-community-id

Zeek support for Community ID flow hashing.
https://github.com/corelight/community-id-spec
Other
33 stars 18 forks source link

Install fails atop Homebrew-installed Zeek v4.0.0 on macOS #15

Closed philrz closed 3 years ago

philrz commented 3 years ago

I originally stumbled onto this problem on my Mac laptop running macOS Big Sur (11.2), but as I get into below, I've reproduced it on GitHub Actions runners using macOS 10.15 as well. I figure using Actions makes it easier for others to acquire & run this on "scratch" macOS instances.

Starting from a fresh macOS host, the steps & failure log in a nutshell:

Mac-1616963011858:~ runner$ brew install zeek
==> Downloading https://homebrew.bintray.com/bottles/caf-0.18.0.catalina.bottle
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/0af809980707fd565374
######################################################################## 100.0%
==> Downloading https://homebrew.bintray.com/bottles/geoip-1.6.12.catalina.bott
######################################################################## 100.0%
==> Downloading https://homebrew.bintray.com/bottles/zeek-4.0.0.catalina.bottle
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/1113a1c88e878f05d170
######################################################################## 100.0%
==> Installing dependencies for zeek: caf and geoip
==> Installing zeek dependency: caf
==> Pouring caf-0.18.0.catalina.bottle.tar.gz
🍺  /usr/local/Cellar/caf/0.18.0: 528 files, 4.9MB
==> Installing zeek dependency: geoip
==> Pouring geoip-1.6.12.catalina.bottle.2.tar.gz
🍺  /usr/local/Cellar/geoip/1.6.12: 18 files, 639.9KB
==> Installing zeek
==> Pouring zeek-4.0.0.catalina.bottle.tar.gz
🍺  /usr/local/Cellar/zeek/4.0.0: 1,482 files, 21.3MB

Mac-1616963011858:~ runner$ sudo pip3 install zkg
WARNING: The directory '/Users/runner/Library/Caches/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting zkg
  Downloading zkg-2.7.1-py2.py3-none-any.whl (48 kB)
     |████████████████████████████████| 48 kB 16.2 MB/s 
Collecting gitpython
  Downloading GitPython-3.1.14-py3-none-any.whl (159 kB)
     |████████████████████████████████| 159 kB 30.9 MB/s 
Collecting semantic-version
  Downloading semantic_version-2.8.5-py2.py3-none-any.whl (15 kB)
Collecting btest
  Downloading btest-0.67.tar.gz (88 kB)
     |████████████████████████████████| 88 kB 42.6 MB/s 
Collecting gitdb<5,>=4.0.1
  Downloading gitdb-4.0.7-py3-none-any.whl (63 kB)
     |████████████████████████████████| 63 kB 34.7 MB/s 
Collecting smmap<5,>=3.0.1
  Downloading smmap-4.0.0-py2.py3-none-any.whl (24 kB)
Building wheels for collected packages: btest
  Building wheel for btest (setup.py) ... done
  Created wheel for btest: filename=btest-0.67-py3-none-any.whl size=35497 sha256=d0f00a00550d0b1f0c14c1efa4a8e8ff4877aa9e09ee8f86cf5b1cfc05bda156
  Stored in directory: /private/tmp/pip-ephem-wheel-cache-ocjlrm79/wheels/18/4c/2a/2d6b3acca99d9c921b3acf8cf53cc1784004d5ade916e12e60
Successfully built btest
Installing collected packages: smmap, gitdb, semantic-version, gitpython, btest, zkg
Successfully installed btest-0.67 gitdb-4.0.7 gitpython-3.1.14 semantic-version-2.8.5 smmap-4.0.0 zkg-2.7.1

Mac-1616963011858:~ runner$ sudo zkg autoconfig
Successfully wrote config file to /Users/runner/.zkg/config

Mac-1616963011858:~ runner$ sudo zkg install --force https://github.com/corelight/zeek-community-id
Running unit tests for "https://github.com/corelight/zeek-community-id"
error: failed to run tests for https://github.com/corelight/zeek-community-id: package build_command failed, see log in /Users/runner/.zkg/logs/zeek-community-id-build.log
Installing "https://github.com/corelight/zeek-community-id".....
Failed installing "https://github.com/corelight/zeek-community-id": package build_command failed, see log in /Users/runner/.zkg/logs/zeek-community-id-build.log
error: incomplete installation, the follow packages failed to be installed:
  https://github.com/corelight/zeek-community-id (3.2.0)

Mac-1616963011858:~ runner$ cat /Users/runner/.zkg/logs/zeek-community-id-build.log
=== STDERR ===
In file included from /Users/runner/.zkg/clones/package/zeek-community-id/src/Plugin.cc:2:
In file included from /Users/runner/.zkg/clones/package/zeek-community-id/src/Plugin.h:3:
In file included from /usr/local/include/zeek/plugin/Plugin.h:11:
In file included from /usr/local/include/zeek/logging/WriterBackend.h:7:
In file included from /usr/local/include/zeek/threading/MsgThread.h:5:
In file included from /usr/local/include/zeek/threading/Queue.h:10:
In file included from /usr/local/include/zeek/Reporter.h:12:
In file included from /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/unordered_set:364:
In file included from /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__hash_table:18:
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:317:9: error: no member named 'signbit' in the global namespace
using ::signbit;
      ~~^
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:318:9: error: no member named 'fpclassify' in the global namespace
using ::fpclassify;
      ~~^
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:319:9: error: no member named 'isfinite' in the global namespace; did you mean 'finite'?
using ::isfinite;
      ~~^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/math.h:749:12: note: 'finite' declared here
extern int finite(double)
           ^
In file included from /Users/runner/.zkg/clones/package/zeek-community-id/src/Plugin.cc:2:
In file included from /Users/runner/.zkg/clones/package/zeek-community-id/src/Plugin.h:3:
In file included from /usr/local/include/zeek/plugin/Plugin.h:11:
In file included from /usr/local/include/zeek/logging/WriterBackend.h:7:
In file included from /usr/local/include/zeek/threading/MsgThread.h:5:
In file included from /usr/local/include/zeek/threading/Queue.h:10:
In file included from /usr/local/include/zeek/Reporter.h:12:
In file included from /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/unordered_set:364:
In file included from /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__hash_table:18:
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:320:9: error: no member named 'isinf' in the global namespace
using ::isinf;
      ~~^
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:321:9: error: no member named 'isnan' in the global namespace
using ::isnan;
      ~~^
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:322:9: error: no member named 'isnormal' in the global namespace
using ::isnormal;
      ~~^
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:323:7: error: no member named 'isgreater' in the global namespace; did you mean '::std::greater'?
using ::isgreater;
      ^~
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:738:29: note: '::std::greater' declared here
struct _LIBCPP_TEMPLATE_VIS greater : binary_function<_Tp, _Tp, bool>
                            ^
In file included from /Users/runner/.zkg/clones/package/zeek-community-id/src/Plugin.cc:2:
In file included from /Users/runner/.zkg/clones/package/zeek-community-id/src/Plugin.h:3:
In file included from /usr/local/include/zeek/plugin/Plugin.h:11:
In file included from /usr/local/include/zeek/logging/WriterBackend.h:7:
In file included from /usr/local/include/zeek/threading/MsgThread.h:5:
In file included from /usr/local/include/zeek/threading/Queue.h:10:
In file included from /usr/local/include/zeek/Reporter.h:12:
In file included from /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/unordered_set:364:
In file included from /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__hash_table:18:
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:324:7: error: no member named 'isgreaterequal' in the global namespace; did you mean '::std::greater_equal'?
using ::isgreaterequal;
      ^~
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:767:29: note: '::std::greater_equal' declared here
struct _LIBCPP_TEMPLATE_VIS greater_equal : binary_function<_Tp, _Tp, bool>
                            ^
In file included from /Users/runner/.zkg/clones/package/zeek-community-id/src/Plugin.cc:2:
In file included from /Users/runner/.zkg/clones/package/zeek-community-id/src/Plugin.h:3:
In file included from /usr/local/include/zeek/plugin/Plugin.h:11:
In file included from /usr/local/include/zeek/logging/WriterBackend.h:7:
In file included from /usr/local/include/zeek/threading/MsgThread.h:5:
In file included from /usr/local/include/zeek/threading/Queue.h:10:
In file included from /usr/local/include/zeek/Reporter.h:12:
In file included from /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/unordered_set:364:
In file included from /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__hash_table:18:
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:325:9: error: no member named 'isless' in the global namespace
using ::isless;
      ~~^
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:326:9: error: no member named 'islessequal' in the global namespace
using ::islessequal;
      ~~^
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:327:9: error: no member named 'islessgreater' in the global namespace
using ::islessgreater;
      ~~^
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:328:9: error: no member named 'isunordered' in the global namespace
using ::isunordered;
      ~~^
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:329:9: error: no member named 'isunordered' in the global namespace
using ::isunordered;
      ~~^
13 errors generated.
make[2]: *** [CMakeFiles/Corelight-CommunityID.darwin-x86_64.dir/src/Plugin.cc.o] Error 1
make[1]: *** [CMakeFiles/Corelight-CommunityID.darwin-x86_64.dir/all] Error 2
make: *** [all] Error 2
=== STDOUT ===
Build Directory        : build
Zeek Source Directory  : 
-- The C compiler identification is AppleClang 12.0.0.12000032
-- The CXX compiler identification is AppleClang 12.0.0.12000032
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenSSL: /usr/local/opt/openssl/lib/libcrypto.dylib (found version "1.1.1j")  
-- Performing Test cxx17_already_works
-- Performing Test cxx17_already_works - Failed
-- Performing Test cxx17_works
-- Performing Test cxx17_works - Success
-- Found BinPAC: /usr/local/include/binpac  
-- Found CAF: 0.18.0  found components: core io openssl 
-- Found Broker: /usr/local/lib/libbroker.dylib  
-- Zeek executable      : /usr/local/Cellar/zeek/4.0.0/bin/zeek
-- Zeek source          : 
-- Zeek build           : 
-- Zeek install prefix  : /usr/local/Cellar/zeek/4.0.0
-- Zeek plugin directory: /usr/local/Cellar/zeek/4.0.0/lib/zeek/plugins
-- Zeek debug mode      : 
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/runner/.zkg/clones/package/zeek-community-id/build
Scanning dependencies of target generate_outputs
[  0%] Built target generate_outputs
Scanning dependencies of target copy-scripts-Corelight_CommunityID
[  0%] Built target copy-scripts-Corelight_CommunityID
Scanning dependencies of target bif-plugin-Corelight_CommunityID-communityid.bif
[ 11%] [BIFCL] Processing src/communityid.bif
[ 11%] Built target bif-plugin-Corelight_CommunityID-communityid.bif
Scanning dependencies of target bro-plugin-Corelight_CommunityID
[ 22%] Creating __bro_plugin__ for Corelight::CommunityID
[ 22%] Built target bro-plugin-Corelight_CommunityID
Scanning dependencies of target bif-init-Corelight_CommunityID
[ 22%] Built target bif-init-Corelight_CommunityID
Scanning dependencies of target Corelight-CommunityID.darwin-x86_64
[ 33%] Building CXX object CMakeFiles/Corelight-CommunityID.darwin-x86_64.dir/src/Plugin.cc.o

For GitHub Actions Workflow that reproduces this problem, see this repo:

https://github.com/philrz/zeek-macos-brew-cid-fail

The Actions Workflow file that repeats the steps above to show the failure is at:

https://github.com/philrz/zeek-macos-brew-cid-fail/blob/main/.github/workflows/runtest.yaml

I've confirmed via a separate set of runs that I do not have this problem if I compile Zeek v4.0.0 from source and use its built-in zkg. A repo that runs those steps successfully as a GitHub Actions workflow, for comparison:

https://github.com/philrz/zeek-macos-compiled-cid-success
https://github.com/philrz/zeek-macos-compiled-cid-success/blob/main/.github/workflows/runtest.yaml

ckreibich commented 3 years ago

Thanks a bunch Phil, lots of useful detail here! From the error messages this looks like the problem described here, i.e. a platform-internal ordering problem in the resolution of math.h. That would mean it's certainly not a Community ID problem and arguably also not a Zeek one. But ... this makes it pretty intriguing that your Zeek-bundled zkg doesn't have this problem... I would have assumed this to make zero difference as far as running the package's build_command is concerned. So there's something interesting going on here. I'll dig in and follow up.

ckreibich commented 3 years ago

I tweaked the actions so they dump the generated CMakeCache.txt files out of the package build directory zkg produces in its internal state. There's a pretty suspicious difference in the include folders produced via zeek-config --include_dir. For the succeeding build:

BRO_CONFIG_INCLUDE_DIR:PATH=/usr/local/zeek/include:/usr/local/zeek/include/zeek:/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include:/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include:/usr/local/opt/openssl/include::

For the brew-based, failing one:

BRO_CONFIG_INCLUDE_DIR:PATH=/usr/local/Cellar/zeek/4.0.0/include:/usr/local/Cellar/zeek/4.0.0/include/zeek:/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include:/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include:/usr/local/opt/openssl@1.1/include::

I.e., different SDKs, which matches the diagnosis in the link above.

More tomorrow...

ckreibich commented 3 years ago

zeek/zeek#1368 looks related too, though the focus there was CirrusCI.

ckreibich commented 3 years ago

It looks like the root discrepancy between the brew-based install and the full build is indeed that in the Homebrew install zeek-config --include_dir lists /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include for some system headers Zeek requires, whereas in the local build it ends up being /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include.

A crude workaround if you want to keep using the Homebrew workflow right now is therefore to swap out these paths in zeek-config via something like this:

diff --git a/.github/workflows/runtest.yaml b/.github/workflows/runtest.yaml
index a98eb24..4557e0e 100644
--- a/.github/workflows/runtest.yaml
+++ b/.github/workflows/runtest.yaml
@@ -8,7 +8,9 @@ jobs:
     runs-on: macos-10.15
     steps:
     - name: Install Zeek via brew
-      run: brew install zeek
+      run: |
+        brew install zeek
+        sed -i '' "s|$(xcrun --show-sdk-path)|$(xcodebuild -sdk macosx -version | grep '^Path:' | cut -d' ' -f2)|" $(brew --cellar)/zeek/4.0.0/bin/zeek-config
     - name: Install/setup Zeek Package Manager
       run: |
         sudo pip3 install zkg

I cobbled together a few commands here that seem better than literally spelling out paths subject to change, but it's clearly a hack. I suspect the true fix will be to our MacDependencyPaths CMake module over in Zeek, for which I'll now create a ticket.

This is the second case in the past weeks where it would have been handy to alter the paths that zeek-config reports after installation, for external reasons — the other was a DESTDIR-style FreeBSD override for a Zeek install staged in an alternative location.

philrz commented 3 years ago

I'm pleased to report that with the guidance from @ckreibich and patch suggestions from @bbannier, I've managed to get https://github.com/Homebrew/homebrew-core/pull/74932 merged to address the original symptom described at this issue. Now that the Homebrew formula is updated, here's verification steps on a scratch macOS running on GitHub Actions that shows it's now working:

$ brew update
$ brew install zeek
$ sudo pip3 install zkg
$ sudo zkg autoconfig
$ sudo zkg install --force https://github.com/corelight/zeek-community-id
$ wget https://archive.wrccdc.org/pcaps/2018/wrccdc.2018-03-23.010014000000000.pcap.gz
$ gunzip wrccdc.2018-03-23.010014000000000.pcap.gz 
$ zeek -C -r wrccdc.2018-03-23.010014000000000.pcap --exec "@load packages" local
$ cat conn.log | zeek-cut community_id | head -1
1:Ok4Im3EoUdvUUsMuVAIMNfYoKfg=

Since I was the one who originally opened this issue, I'll go ahead and close it now. Thanks to everyone for their help! It's good knowing I can point users at Homebrew-installed binary Zeek and know they can use the Community ID package. It's become very useful in the Brim experience since it's essential for joining Zeek and Suricata data.

ckreibich commented 3 years ago

Thank you for all your work here, Phil! :+1: