search

SeisoLLC / zeek-kafka

A Zeek log writer plugin that publishes to Kafka.
Apache License 2.0
45 stars 16 forks source link

Add support for Ubuntu (Fixes #19) #21

Closed JonZeolla closed 3 years ago

JonZeolla commented 3 years ago

Summary of the contribution

Adds support for ubuntu:20.04

Testing

make e2e

Checklist

ottobackwards commented 3 years ago 
Complete!
Error: No matching repo to modify: powertools.
ERROR: Service 'zeek' failed to build : The command '/bin/sh -c dnf install -y 'dnf-command(config-manager)'  && dnf config-manager --set-enabled powertools' returned a non-zero code: 1
make: *** [e2e] Error 1
JonZeolla commented 3 years ago

centos succeeds locally but is failing in the pipeline due to a disk space issue. ubuntu is still an issue

ckreibich commented 3 years ago

Hi folks, as mentioned on the Zeek Slack I gave this a go with Ubuntu 20.10, which has packages for librdkafka 1.4.2. Here's a Dockerfile to get going:

FROM ubuntu:20.10

ENV DEBIAN_FRONTEND noninteractive

RUN apt update

# Install common dependencies 
RUN apt install -y bison bzip2 cmake curl flex g++ gcc git gpg libmaxminddb-dev \
    libpcap-dev libssl-dev make nullmailer python3 python3-dev python3-smmap \
    python3-gitdb python3-git python3-semantic-version swig tree zlib1g-dev

# Set up repo to install Zeek from SUSE OBS, as per
# https://software.opensuse.org//download.html?project=security%3Azeek&package=zeek
RUN echo 'deb http://download.opensuse.org/repositories/security:/zeek/xUbuntu_20.10/ /' | tee /etc/apt/sources.list.d/security:zeek.list
RUN curl -fsSL https://download.opensuse.org/repositories/security:zeek/xUbuntu_20.10/Release.key | gpg --dearmor | tee /etc/apt/trusted.gpg.d/security_zeek.gpg > /dev/null

RUN apt update
RUN apt install -y zeek
RUN echo 'export PATH=$PATH:/opt/zeek/bin' >/etc/profile.d/zeek.sh

# Install rdkafka
RUN apt install -y librdkafka-dev

ENTRYPOINT bash -l

When you run this and then say LIBRDKAFKA_ROOT=/usr zkg install seisollc/zeek-kafka --version main, you'll see it bails in the plugin build, with these errors:

  error in /opt/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: /opt/zeek/var/lib/zkg/testing/zeek-kafka/clones/zeek-kafka/build//lib/SEISO-KAFKA.linux-x86_64.so: undefined symbol: _ZN7RdKafka5Topic12PARTITION_UAE

Looks like it dislikes the use of RdKafka::Topic::PARTITION_UA. Not sure what's going on here since this doesn't happen when installing librdkafka 1.4.2 from https://github.com/edenhill/librdkafka/archive/refs/tags/v1.4.2.tar.gz.

JonZeolla commented 3 years ago

Still hitting out of disk errors, need to adjust the Zeek install process to use the open suse package...

ckreibich commented 3 years ago

One more comment — is LIBRDKAFKA_ROOT supposed to be the root of the installation tree, or the library folder? The default says /usr/local/lib, but it looks like the cmake checks look for things like $LIBRDKAFA_ROOT/include/librdkafka/rdkafkacpp.h ... so /usr/local might be a better default.

JonZeolla commented 3 years ago

Nice catch @ckreibich

JonZeolla commented 3 years ago

Ubuntu is now working locally.

JonZeolla commented 3 years ago

I'm not sure what to do about the disk issues yet. It doesn't seem like the opensuse approach is compatible with our preferences on version-pinning packages - I'm only seeing the latest releases on there.

JonZeolla commented 3 years ago

@ottobackwards had to remove the docker build caching but now e2e tests fit in GitHub actions 🤷‍♂️

ottobackwards commented 3 years ago

Well.... I really don't see us building often enough to get heart burn about the build times short of them timing out. So, No worries from me.

ottobackwards commented 3 years ago

@JonZeolla conflicts need straightening out

JonZeolla commented 3 years ago

@ottobackwards rebased on main, should be good to go now.