search

SeisoLLC / zeek-kafka

A Zeek log writer plugin that publishes to Kafka.
Apache License 2.0
46 stars 15 forks source link

Installation failure in all ways #84

Closed prestonmcgowan closed 7 months ago

prestonmcgowan commented 8 months ago

Summary of the issue

I have tried installing on my bare metal RHEL 8 with zkg and with make; sudo make install and all results end in failure.

Environment

[preston@skid librdkafka-1.4.4]$ examples/rdkafka_example Usage: examples/rdkafka_example -C|-P|-L -t [-p ] [-b <host1:port1,host2:port2,..>]

librdkafka version 1.4.4 (0x010404ff)


# Installation attempts

## Happy Path with zkg .. test fail

$ zkg install seisollc/zeek-kafka --version v1.2.0 The following packages will be INSTALLED: zeek/seisollc/zeek-kafka (v1.2.0)

Verify the following REQUIRED external dependencies: (Ensure their installation on all relevant systems before proceeding): from zeek/seisollc/zeek-kafka (v1.2.0): librdkafka ~1.4.2

Proceed? [Y/n] y "zeek/seisollc/zeek-kafka" requires a "LIBRDKAFKA_ROOT" value (Path to librdkafka installation tree root): LIBRDKAFKA_ROOT: /usr/local Saved answers to config file: /opt/zeek/etc/zkg/config Running unit tests for "zeek/seisollc/zeek-kafka" error: failed to run tests for zeek/seisollc/zeek-kafka: test_command failed with exit code 1 Proceed to install anyway? [N/y] n Abort.


## Rebuild and try .. test fail

$ cd zeek-kafka-1.2.0/ [preston@skid zeek-kafka-1.2.0]$ make test make -C tests make[1]: Entering directory '/home/preston/homeLab/zeek/zeek-kafka/zeek-kafka-1.2.0/tests' [ 0%] kafka.l2s-l2e-no-overlap ... failed [ 7%] kafka.l2s-set-l2e-set ... failed [ 14%] kafka.l2s-set-l2e-unset ... failed [ 21%] kafka.l2s-unset-l2e-set ... failed [ 28%] kafka.l2s-unset-l2e-unset ... failed [ 35%] kafka.resolved-topic-config ... failed [ 42%] kafka.resolved-topic-default ... failed [ 50%] kafka.resolved-topic-override-and-config ... failed [ 57%] kafka.resolved-topic-override-only ... failed [ 64%] kafka.send-all-active-logs-l2e-set ... failed [ 71%] kafka.send-all-active-logs-l2e-unset ... failed [ 78%] kafka.send-all-active-logs-l2s-set-l2e-set ... failed [ 85%] kafka.send-all-active-logs-l2s-set-l2e-unset ... failed [ 92%] kafka.show-plugin ... failed 14 of 14 tests failed make[1]: [Makefile:19: test] Error 1 make[1]: Leaving directory '/home/preston/homeLab/zeek/zeek-kafka/zeek-kafka-1.2.0/tests' make: [Makefile:52: test] Error 2


# YOLO

## Force the install; hell to the test results.

$ sudo make install $ zeek -N error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/lib/zeek/plugins/SEISO_KAFKA//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/lib/zeek/plugins/SEISO_KAFKA//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11chartraitsIcESaIcEEESA fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors


## Remove the install and zeek works again

$ rm -Rf /opt/zeek/lib/zeek/plugins/SEISO_KAFKA/ $ zeek -N Zeek::AF_Packet - Packet acquisition via AF_Packet (built-in) Zeek::ARP - ARP packet analyzer (built-in) Zeek::AsciiReader - ASCII input reader (built-in) Zeek::AsciiWriter - ASCII log writer (built-in) Zeek::AYIYA - AYIYA packet analyzer (built-in)



# Assistance Request

* I believe I have followed the instructions in the README. 
* I have verified I have installed librdkafka 1.4.4, and I tried 1.4.2 also. 
* On slack, the zeek-kafka 1.2.0 was mentioned as supporting zeek 6.0.

What else could I be doing wrong?
ottobackwards commented 8 months ago

Hi, what version of the developer tool ( gcc etc ) do you have installed? have you activated the most recent versions of those tools?

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/developing_c_and_cpp_applications_in_rhel_8/additional-toolsets-for-development_developing-applications

You need to confirm that you have a C++ 17 version of the gcc toolset installed, and you should run the enable script in your shell before building.

prestonmcgowan commented 8 months ago

GCC:

$ gcc --version
gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20)

$ rpm -qa | grep gcc
gcc-c++-8.5.0-20.el8.x86_64
libgcc-8.5.0-20.el8.x86_64
gcc-8.5.0-20.el8.x86_64

CMake

cmake-3.26.5-1.el8_9.x86_64

I will look for the enable script.

prestonmcgowan commented 8 months ago

Installed toolset 13

$ rpm -qa | grep gcc-toolset
gcc-toolset-13-dwz-0.14-0.el8.x86_64
gcc-toolset-13-runtime-13.0-2.el8.x86_64
gcc-toolset-13-libquadmath-devel-13.1.1-4.3.el8.x86_64
gcc-toolset-13-gcc-13.1.1-4.3.el8.x86_64
gcc-toolset-13-annobin-docs-12.20-1.el8.noarch
gcc-toolset-13-annobin-plugin-gcc-12.20-1.el8.x86_64
gcc-toolset-13-binutils-2.40-14.el8.x86_64
gcc-toolset-13-gdb-12.1-3.el8.x86_64
gcc-toolset-13-gcc-gfortran-13.1.1-4.3.el8.x86_64
gcc-toolset-13-gcc-c++-13.1.1-4.3.el8.x86_64
gcc-toolset-13-libstdc++-devel-13.1.1-4.3.el8.x86_64
gcc-toolset-13-binutils-gold-2.40-14.el8.x86_64
gcc-toolset-13-13.0-2.el8.x86_64

The scl enable commands look like they need to be run by the linker, so I don't think I run anything myself directly. I performed a make clean and reran ./configure on the zeek-kafka-1.2.0 codebase. The test still fail when I run make test.

Did I miss a step in the build process? Should I have been able to use the zkg install? I am only building from source since the zkg install threw errors.

ottobackwards commented 8 months ago

You will probably need something like: source /opt/rh/gcc-toolset-13/enable before you try to build. Please try that.

prestonmcgowan commented 8 months ago

make test is still failing after I run the following:

make clean
source /opt/rh/gcc-toolset-13/enable
./configure
make
make test
ottobackwards commented 8 months ago

where did you get the rpms?

prestonmcgowan commented 8 months ago 
$ sudo yum info gcc-toolset-13-13.0-2.el8.x86_64

Installed Packages
Name         : gcc-toolset-13
Version      : 13.0
Release      : 2.el8
Architecture : x86_64
Size         : 1.7 k
Source       : gcc-toolset-13-13.0-2.el8.src.rpm
Repository   : @System
From repo    : rhel-8-for-x86_64-appstream-rpms
Summary      : Package that installs gcc-toolset-13
License      : GPLv2+
Description  : This is the main package for gcc-toolset-13 Software Collection.
$ sudo yum info zeek-6.0-6.0.3-1.1.x86_64
Updating Subscription Management repositories.
Last metadata expiration check: 0:01:45 ago on Wed 27 Mar 2024 10:10:23 AM EDT.
Installed Packages
Name         : zeek-6.0
Version      : 6.0.3
Release      : 1.1
Architecture : x86_64
Size         : 0.0
Source       : zeek-6.0-6.0.3-1.1.src.rpm
Repository   : @System
From repo    : security_zeek
Summary      : Zeek is a powerful framework for network analysis and security monitoring
URL          : http://zeek.org
License      : BSD-3-Clause
Description  : Zeek is a powerful network analysis framework that is much different from the
             : typical IDS you may know.  While focusing on network security monitoring, Zeek
             : provides a comprehensive platform for more general network traffic analysis as
             : well. Well grounded in more than 15 years of research, Zeek has successfully
             : bridged the traditional gap between academia and operations since its
             : inception. Today, it is relied upon operationally in particular by many
             : scientific environments for securing their cyberinfrastructure. Zeek's user
             : community includes major universities, research labs, supercomputing centers,
             : and open-science communities.
ottobackwards commented 8 months ago

I am interested in the Zeek rpms. the official Zeek builds are not working on centos right now, so I cannot install since I do not have the rpms. I am trying to use the UBI for 8.9, as I don't have a rhel subscription :/

prestonmcgowan commented 8 months ago

Rhel allows for a developer license.

prestonmcgowan commented 7 months ago

Should I switch my Zeek install to 5.x?

ottobackwards commented 7 months ago

I was able to get this to work with almalinux 8.9 in docker. I'm not sure what is going on with your env. but here are my commands and dockerfile, maybe they can point you in the right direction

[root@e126c399dfcc ~]# source /opt/rh/gcc-toolset-13/enable && /opt/zeek/bin/zkg install seisollc/zeek-kafka --version v1.2.0
The following packages will be INSTALLED:
  zeek/seisollc/zeek-kafka (v1.2.0)

Verify the following REQUIRED external dependencies:
(Ensure their installation on all relevant systems before proceeding):
  from zeek/seisollc/zeek-kafka (v1.2.0):
    librdkafka ~1.4.2

Proceed? [Y/n] y
"zeek/seisollc/zeek-kafka" requires a "LIBRDKAFKA_ROOT" value (Path to librdkafka installation tree root):
LIBRDKAFKA_ROOT: /usr/local/lib
Saved answers to config file: /opt/zeek/etc/zkg/config
Running unit tests for "zeek/seisollc/zeek-kafka"
Installing "zeek/seisollc/zeek-kafka".........................
Installed "zeek/seisollc/zeek-kafka" (v1.2.0)
Loaded "zeek/seisollc/zeek-kafka"
[root@e126c399dfcc ~]# /opt/zeek/bin/zeek -N
Zeek::AF_Packet - Packet acquisition via AF_Packet (built-in)
Zeek::ARP - ARP packet analyzer (built-in)
Zeek::AsciiReader - ASCII input reader (built-in)
Zeek::AsciiWriter - ASCII log writer (built-in)
Zeek::AYIYA - AYIYA packet analyzer (built-in)
Zeek::BenchmarkReader - Benchmark input reader (built-in)
Zeek::BinaryReader - Binary input reader (built-in)
Zeek::BitTorrent - BitTorrent Analyzer (built-in)
Zeek::ConfigReader - Configuration file input reader (built-in)
Zeek::ConnSize - Connection size analyzer (built-in)
Zeek::DCE_RPC - DCE-RPC analyzer (built-in)
Zeek::DHCP - DHCP analyzer (built-in)
Zeek::DNP3 - DNP3 UDP/TCP analyzers (built-in)
Zeek::DNS - DNS analyzer (built-in)
Zeek::Ethernet - Ethernet packet analyzer (built-in)
Zeek::FDDI - FDDI packet analyzer (built-in)
Zeek::File - Generic file analyzer (built-in)
Zeek::FileDataEvent - Delivers file content (built-in)
Zeek::FileEntropy - Entropy test file content (built-in)
Zeek::FileExtract - Extract file content (built-in)
Zeek::FileHash - Hash file content (built-in)
Zeek::FTP - FTP analyzer (built-in)
Zeek::Geneve - Geneve packet analyzer (built-in)
Zeek::Gnutella - Gnutella analyzer (built-in)
Zeek::GRE - GRE packet analyzer (built-in)
Zeek::GSSAPI - GSSAPI analyzer (built-in)
Zeek::GTPv1 - GTPv1 analyzer (built-in)
Zeek::HTTP - HTTP analyzer (built-in)
Zeek::ICMP - Packet analyzer for ICMP (built-in)
Zeek::Ident - Ident analyzer (built-in)
Zeek::IEEE802_11 - IEEE 802.11 packet analyzer (built-in)
Zeek::IEEE802_11_Radio - IEEE 802.11 Radiotap packet analyzer (built-in)
Zeek::IMAP - IMAP analyzer (StartTLS only) (built-in)
Zeek::IP - Packet analyzer for IP fallback (v4 or v6) (built-in)
Zeek::IPTunnel - IPTunnel packet analyzer (built-in)
Zeek::IRC - IRC analyzer (built-in)
Zeek::KRB - Kerberos analyzer (built-in)
Zeek::LinuxSLL - Linux cooked capture (SLL) packet analyzer (built-in)
Zeek::LinuxSLL2 - Linux cooked capture version 2 (SLL2) packet analyzer (built-in)
Zeek::LLC - LLC packet analyzer (built-in)
Zeek::Login - Telnet/Rsh/Rlogin analyzers (built-in)
Zeek::MIME - MIME parsing (built-in)
Zeek::Modbus - Modbus analyzer (built-in)
Zeek::MPLS - MPLS packet analyzer (built-in)
Zeek::MQTT - Message Queuing Telemetry Transport v3.1.1 Protocol analyzer (built-in)
Zeek::MySQL - MySQL analyzer (built-in)
Zeek::NCP - NCP analyzer (built-in)
Zeek::NetBIOS - NetBIOS analyzer support (built-in)
Zeek::NFLog - NFLog packet analyzer (built-in)
Zeek::NoneWriter - None log writer (primarily for debugging) (built-in)
Zeek::NOVELL_802_3 - Novell 802.3 variantx packet analyzer (built-in)
Zeek::NTLM - NTLM analyzer (built-in)
Zeek::NTP - NTP analyzer (built-in)
Zeek::Null - Null packet analyzer (built-in)
Zeek::PBB - PBB packet analyzer (built-in)
Zeek::Pcap - Packet acquisition via libpcap (built-in)
Zeek::PE - Portable Executable analyzer (built-in)
Zeek::PIA - Analyzers implementing Dynamic Protocol (built-in)
Zeek::POP3 - POP3 analyzer (built-in)
Zeek::PPPoE - PPPoE packet analyzer (built-in)
Zeek::PPPSerial - PPPSerial packet analyzer (built-in)
Zeek::RADIUS - RADIUS analyzer (built-in)
Zeek::RawReader - Raw input reader (built-in)
Zeek::RDP - RDP analyzer (built-in)
Zeek::RFB - Parser for rfb (VNC) analyzer (built-in)
Zeek::Root - Root packet analyzer (built-in)
Zeek::RPC - Analyzers for RPC-based protocols (built-in)
Zeek::SIP - SIP analyzer UDP-only (built-in)
Zeek::Skip - Skip packet analyzer (built-in)
Zeek::SMB - SMB analyzer (built-in)
Zeek::SMTP - SMTP analyzer (built-in)
Zeek::SNAP - SNAP packet analyzer (built-in)
Zeek::SNMP - SNMP analyzer (built-in)
Zeek::SOCKS - SOCKS analyzer (built-in)
Zeek::Spicy - Support for Spicy parsers (*.hlto) (built-in)
Zeek::SQLiteReader - SQLite input reader (built-in)
Zeek::SQLiteWriter - SQLite log writer (built-in)
Zeek::SSH - Secure Shell analyzer (built-in)
Zeek::SSL - SSL/TLS and DTLS analyzers (built-in)
Zeek::TCP - TCP analyzer (built-in)
Zeek::TCP_PKT - Packet analyzer for TCP (built-in)
Zeek::Teredo - Teredo packet analyzer (built-in)
Zeek::UDP - Packet analyzer for UDP (built-in)
Zeek::VLAN - VLAN packet analyzer (built-in)
Zeek::VNTag - VNTag packet analyzer (built-in)
Zeek::VXLAN - VXLAN packet analyzer (built-in)
Zeek::X509 - X509 and OCSP analyzer (built-in)
Zeek::XMPP - XMPP analyzer (StartTLS only) (built-in)
Zeek::ZIP - Generic ZIP support analyzer (built-in)
Seiso::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
[root@e126c399dfcc ~]#

Using the following Dockerfile:

FROM almalinux:8.9

RUN dnf clean all
RUN dnf install -y 'dnf-command(config-manager)'
RUN dnf config-manager --set-enabled powertools
RUN dnf install -y epel-release
RUN dnf config-manager --set-enabled epel
RUN dnf update -y 
RUN dnf install -y gcc-toolset-13 vim curl cmake
RUN dnf install -y https://forensics.cert.org/centos/cert/8/x86_64/zeek-6.0.2-1.el8.x86_64.rpm \
    https://forensics.cert.org/centos/cert/8/x86_64/zeek-client-6.0.2-1.el8.x86_64.rpm  \
    https://forensics.cert.org/centos/cert/8/x86_64/zeek-core-6.0.2-1.el8.x86_64.rpm \
    https://forensics.cert.org/centos/cert/8/x86_64/zeek-devel-6.0.2-1.el8.x86_64.rpm \
    https://forensics.cert.org/centos/cert/8/x86_64/zeek-spicy-devel-6.0.2-1.el8.x86_64.rpm \
    https://forensics.cert.org/centos/cert/8/x86_64/zeek-zkg-6.0.2-1.el8.x86_64.rpm \
    https://forensics.cert.org/centos/cert/8/x86_64/zeek-btest-6.0.2-1.el8.x86_64.rpm \
    https://forensics.cert.org/centos/cert/8/x86_64/zeek-btest-data-6.0.2-1.el8.x86_64.rpm \
  https://forensics.cert.org/centos/cert/8/x86_64/libbroker-devel-6.0.2-1.el8.x86_64.rpm \
  https://forensics.cert.org/centos/cert/8/x86_64/zeekctl-6.0.2-1.el8.x86_64.rpm
WORKDIR /root
RUN curl -L https://github.com/edenhill/librdkafka/archive/v1.4.4.tar.gz | tar xvz
RUN source /opt/rh/gcc-toolset-13/enable && cd librdkafka-1.4.4/ && \
  ./configure && \
  make && \
  make install
prestonmcgowan commented 7 months ago

I was missing and installed:

https://forensics.cert.org/centos/cert/8/x86_64/zeek-btest-6.0.2-1.el8.x86_64.rpm
https://forensics.cert.org/centos/cert/8/x86_64/zeekctl-6.0.2-1.el8.x86_64.rpm

I reran the librdkafka build process as above.

I am still getting the following:

[root@skid librdkafka-1.4.4]# source /opt/rh/gcc-toolset-13/enable && /opt/zeek/bin/zkg install seisollc/zeek-kafka --version v1.2.0
The following packages will be INSTALLED:
  zeek/seisollc/zeek-kafka (v1.2.0)

Verify the following REQUIRED external dependencies:
(Ensure their installation on all relevant systems before proceeding):
  from zeek/seisollc/zeek-kafka (v1.2.0):
    librdkafka ~1.4.2

Proceed? [Y/n] y
"zeek/seisollc/zeek-kafka" requires a "LIBRDKAFKA_ROOT" value (Path to librdkafka installation tree root):
LIBRDKAFKA_ROOT: /usr/local/lib
Saved answers to config file: /opt/zeek/etc/zkg/config
Running unit tests for "zeek/seisollc/zeek-kafka"
error: failed to run tests for zeek/seisollc/zeek-kafka: test_command failed with exit code 1
Proceed to install anyway? [N/y] n
Abort.

Thank you for all of your help thus far. I am wondering if perhaps I should just run zeek in a container instead of on baremetal.

ottobackwards commented 7 months ago

Can you find the test log somewhere under /var/lib/zkg I think?

Also, make sure you install the packages from your repos that match the version, CERT was just what I could find, I don't think you should mix them with RHEL

prestonmcgowan commented 7 months ago

I have been looking for logs, you gave me a good place to start looking!

[root@skid zeek-kafka]# pwd
/opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka
[root@skid zeek-kafka]# cat zkg.test_command.stderr
kafka.l2s-l2e-no-overlap ... failed
  % 'zeek ../../../scripts/Seiso/Kafka/ /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/tests/.tmp/kafka.l2s-l2e-no-overlap/l2s-l2e-no-overlap.zeek > output' failed unexpectedly (exit code 1)
  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

kafka.l2s-set-l2e-set ... failed
  % 'zeek ../../../scripts/Seiso/Kafka/ /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/tests/.tmp/kafka.l2s-set-l2e-set/l2s-set-l2e-set.zeek > output' failed unexpectedly (exit code 1)
  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

kafka.l2s-set-l2e-unset ... failed
  % 'zeek ../../../scripts/Seiso/Kafka/ /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/tests/.tmp/kafka.l2s-set-l2e-unset/l2s-set-l2e-unset.zeek > output' failed unexpectedly (exit code 1)
  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

kafka.l2s-unset-l2e-set ... failed
  % 'zeek ../../../scripts/Seiso/Kafka/ /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/tests/.tmp/kafka.l2s-unset-l2e-set/l2s-unset-l2e-set.zeek > output' failed unexpectedly (exit code 1)
  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

kafka.l2s-unset-l2e-unset ... failed
  % 'zeek ../../../scripts/Seiso/Kafka/ /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/tests/.tmp/kafka.l2s-unset-l2e-unset/l2s-unset-l2e-unset.zeek > output' failed unexpectedly (exit code 1)
  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

kafka.resolved-topic-config ... failed
  % 'zeek -r ../../../tests/pcaps/exercise-traffic.pcap ../../../scripts/Seiso/Kafka/ /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/tests/.tmp/kafka.resolved-topic-config/resolved-topic-config.zeek > output' failed unexpectedly (exit code 1)
  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

kafka.resolved-topic-default ... failed
  % 'zeek -r ../../../tests/pcaps/exercise-traffic.pcap ../../../scripts/Seiso/Kafka/ /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/tests/.tmp/kafka.resolved-topic-default/resolved-topic-default.zeek > output' failed unexpectedly (exit code 1)
  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

kafka.resolved-topic-override-and-config ... failed
  % 'btest-diff output' failed unexpectedly (exit code 1)
  % cat .diag
  == File ===============================
  == Diff ===============================
  --- /dev/fd/63    2024-04-10 15:42:08.455656673 +0000
  +++ /dev/fd/62    2024-04-10 15:42:08.455656673 +0000
  @@ -1,2 +0,0 @@
  -Kafka topic set to configuration-table-topic
  -Kafka topic set to const-variable-topic
  =======================================

  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

kafka.resolved-topic-override-only ... failed
  % 'zeek -r ../../../tests/pcaps/exercise-traffic.pcap ../../../scripts/Seiso/Kafka/ /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/tests/.tmp/kafka.resolved-topic-override-only/resolved-topic-override-only.zeek > output' failed unexpectedly (exit code 1)
  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

kafka.send-all-active-logs-l2e-set ... failed
  % 'zeek ../../../scripts/Seiso/Kafka/ /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/tests/.tmp/kafka.send-all-active-logs-l2e-set/send-all-active-logs-l2e-set.zeek > output' failed unexpectedly (exit code 1)
  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

kafka.send-all-active-logs-l2e-unset ... failed
  % 'zeek ../../../scripts/Seiso/Kafka/ /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/tests/.tmp/kafka.send-all-active-logs-l2e-unset/send-all-active-logs-l2e-unset.zeek > output' failed unexpectedly (exit code 1)
  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

kafka.send-all-active-logs-l2s-set-l2e-set ... failed
  % 'zeek ../../../scripts/Seiso/Kafka/ /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/tests/.tmp/kafka.send-all-active-logs-l2s-set-l2e-set/send-all-active-logs-l2s-set-l2e-set.zeek > output' failed unexpectedly (exit code 1)
  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

kafka.send-all-active-logs-l2s-set-l2e-unset ... failed
  % 'zeek ../../../scripts/Seiso/Kafka/ /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/tests/.tmp/kafka.send-all-active-logs-l2s-set-l2e-unset/send-all-active-logs-l2s-set-l2e-unset.zeek > output' failed unexpectedly (exit code 1)
  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

kafka.show-plugin ... failed
  % 'btest-diff output' failed unexpectedly (exit code 1)
  % cat .diag
  == File ===============================
  == Diff ===============================
  --- /dev/fd/63    2024-04-10 15:42:09.065658421 +0000
  +++ /dev/fd/62    2024-04-10 15:42:09.065658421 +0000
  @@ -1,12 +0,0 @@
  -Seiso::Kafka - Writes logs to Kafka (dynamic)
  -    [Writer] KafkaWriter (Log::WRITER_KAFKAWRITER)
  -    [Constant] Kafka::kafka_conf
  -    [Constant] Kafka::additional_message_values
  -    [Constant] Kafka::topic_name
  -    [Constant] Kafka::max_wait_on_shutdown
  -    [Constant] Kafka::tag_json
  -    [Constant] Kafka::json_timestamps
  -    [Constant] Kafka::debug
  -    [Constant] Kafka::mock
  -    [Event] kafka_topic_resolved_event
  -
  =======================================

  % cat .stderr
  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN4zeek6plugin6Plugin12HookLoadFileENS1_8LoadTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_
  fatal error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

14 of 14 tests failed

The file looks like it exists.

[root@skid zeek-kafka]# ls -al /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so
-rwxr-xr-x. 1 root root 85272 Apr 10 11:42 /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so
ottobackwards commented 7 months ago

Thanks So what is the "security_zeek" repo you are using?

One possible issue for this is that you have Zeek built with clang or ?, and you are building with zkg and gcc 13 toolset. So kind of an incompatibility between your Zeek build toolchain and the current toolchain. Maybe you can try installing another package and seeing if it fails the same way.

prestonmcgowan commented 7 months ago 

[security_zeek]
name=The Zeek Network Security Monitor. (CentOS_7)
type=rpm-md
baseurl=https://download.opensuse.org/repositories/security:/zeek/CentOS_7/
gpgcheck=1
gpgkey=https://download.opensuse.org/repositories/security:/zeek/CentOS_7/repodata/repomd.xml.key
enabled=1

The above repo was from Zeek documentation: https://zeek.org/get-zeek/

I selected CentOS from: https://software.opensuse.org//download.html?project=security%3Azeek&package=zeek

ottobackwards commented 7 months ago

OK, So, those are centos 7 rpms ( built with centos 7 tools and lib versions ). I would suggest that you do one of the following, as best works for you:

prestonmcgowan commented 7 months ago

TLDR; ottobackwards was correct my repo install was causing all the issues.

For the next poor soul that misses the obvious...

Remove the rpms with yum remove zeek-6.0 Add the repo with the correct rpms. The one Ottobackwards uses above was acceptable for my use cases.

# cat /etc/yum.repos.d/forensics_cert_org.repo
[forensics_cert_org]
name=Forensics Cert
type=rpm-md
baseurl=https://forensics.cert.org/centos/cert/8/x86_64/
gpgcheck=1
gpgkey=https://forensics.cert.org/forensics.asc
enabled=1

Install the zeek rpms, yum install zeek.x86_64 Configure zeek, https://docs.zeek.org/en/master/quickstart.html Install the Kafka plugin, source /opt/rh/gcc-toolset-13/enable && /opt/zeek/bin/zkg install seisollc/zeek-kafka --version v1.2.0 Verify /opt/zeek/bin/zeek -N

Lastly and most importantly, thank ottobackwards for his efforts and patience

ottobackwards commented 7 months ago

Glad it worked out for you. The situation with the SuSe forge wrt centos stream is a real problem. Good luck with everything!