search

SeisoLLC / zeek-kafka

A Zeek log writer plugin that publishes to Kafka.
Apache License 2.0
45 stars 16 forks source link

Zeek-Kafka not work , error in tests #44

Closed Canon88 closed 2 years ago

Canon88 commented 2 years ago

Summary of the issue

... The plugin installation failed and the test could not be passed.

Expected behavior

... Pass the test and complete the installation.

Steps to reproduce

... $ curl -L https://github.com/edenhill/librdkafka/archive/v1.4.2.tar.gz | tar xvz $ cd librdkafka-1.4.2/ $ ./configure --enable-sasl $ make $ sudo make install

$ zkg install seisollc/zeek-kafka

Logs, errors, etc.

... $ zkg install seisollc/zeek-kafka The following packages will be INSTALLED: zeek/seisollc/zeek-kafka (v1.0.0)

Verify the following REQUIRED external dependencies: (Ensure their installation on all relevant systems before proceeding): from zeek/seisollc/zeek-kafka (v1.0.0): librdkafka ~1.4.2-RC1

Proceed? [Y/n] y "zeek/seisollc/zeek-kafka" requires a "LIBRDKAFKA_ROOT" value (Path to librdkafka installation tree root): LIBRDKAFKA_ROOT: /usr/local Saved answers to config file: /usr/local/zeek/etc/zkg/config Running unit tests for "zeek/seisollc/zeek-kafka" error: failed to run tests for zeek/seisollc/zeek-kafka: test_command failed with exit code 1 Proceed to install anyway? [N/y] n Abort.

$ zeek -N Seiso::Kafka error in /usr/local/zeek/share/zeek/base/init-bare.zeek, line 1: plugin Seiso::Kafka is not available fatal error in /usr/local/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

Your environment

Canon88 commented 2 years ago

I see that the last updated support version is 4.0.5, is this because of the Zeek version?

ottobackwards commented 2 years ago

I have tested up to zeek 4.2.2 against main. I think we need to do a release, else you can try to install main instead of 1.0.0 in the mean time

JonZeolla commented 2 years ago

@Canon88 can you test against the latest version of zeek-kafka? We just did a v1.1.0-rc1 prerelease

Canon88 commented 2 years ago

@Canon88 can you test against the latest version of zeek-kafka? We just did a v1.1.0-rc1 prerelease No problem, on the way!

Canon88 commented 2 years ago

@Canon88 can you test against the latest version of zeek-kafka? We just did a v1.1.0-rc1 prerelease

Can't specify the version to install, am I wrong somewhere? Please correct me?

$ zkg install seisollc/zeek-kafka --version 1.1.0-rc1
error: invalid package "seisollc/zeek-kafka": no such commit, branch, or version tag: "1.1.0-rc1"

The installation of 1.1.0-rc1 worked, but when I use the command to check the version, it is version 0.3.0. 

$ zkg install seisollc/zeek-kafka
The following packages will be INSTALLED:
  zeek/seisollc/zeek-kafka (v1.1.0-rc1)

Verify the following REQUIRED external dependencies:
(Ensure their installation on all relevant systems before proceeding):
  from zeek/seisollc/zeek-kafka (v1.1.0-rc1):
    librdkafka ~1.4.2

Proceed? [Y/n]
"zeek/seisollc/zeek-kafka" requires a "LIBRDKAFKA_ROOT" value (Path to librdkafka installation tree root):
LIBRDKAFKA_ROOT: /usr/local
Saved answers to config file: /usr/local/zeek/etc/zkg/config
Running unit tests for "zeek/seisollc/zeek-kafka"
Installing "zeek/seisollc/zeek-kafka"......................
Installed "zeek/seisollc/zeek-kafka" (v1.1.0-rc1)
Loaded "zeek/seisollc/zeek-kafka"

$ zeek -N Seiso::Kafka
Seiso::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
Canon88 commented 2 years ago

Can't specify the version to install, am I wrong somewhere? Please correct me?

$ zkg install seisollc/zeek-kafka --version 1.1.0-rc1
error: invalid package "seisollc/zeek-kafka": no such commit, branch, or version tag: "1.1.0-rc1"
Canon88 commented 2 years ago

feedback

I set kafka.zeek, but it's not work. Kafka server address(192.168.199.98) is not valid, why connect 127.0.0.1:9092?

kafka.zeek

@load packages/zeek-kafka

redef Kafka::send_all_active_logs = T;

redef Kafka::tag_json = T;

#redef Kafka::logs_to_exclude = set(Conn::LOG);

redef Kafka::topic_name = "zeek";

redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "192.168.199.98:9092"
);

error.log

%3|1656089601.790|FAIL|rdkafka#producer-4| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.790|ERROR|rdkafka#producer-4| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.790|FAIL|rdkafka#producer-2| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.790|ERROR|rdkafka#producer-2| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.790|FAIL|rdkafka#producer-5| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.790|ERROR|rdkafka#producer-5| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.790|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.791|ERROR|rdkafka#producer-1| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.792|FAIL|rdkafka#producer-3| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.792|ERROR|rdkafka#producer-3| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.793|FAIL|rdkafka#producer-6| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.793|ERROR|rdkafka#producer-6| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
ottobackwards commented 2 years ago

you should not be changing kafka.zeek you should be setting things in the local.zeek

Canon88 commented 2 years ago

you should not be changing kafka.zeek you should be setting things in the local.zeek

yes, this my config. but i don't know why not work. 

$ more kafka.zeek

@load packages/zeek-kafka

redef Kafka::send_all_active_logs = T;

redef Kafka::tag_json = T;

#redef Kafka::logs_to_exclude = set(Conn::LOG);

redef Kafka::topic_name = "zeek";

redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "192.168.199.98:9092"
);

$ more local.zeek

@load kafka
Canon88 commented 2 years ago

I think I know what the problem is. This error is coming from my Kafka cluster. I use a Kafka cluster built by Docker. I tried to test it in a production environment and it worked fine.

It works fine in Zeek 4.2.2.