Closed cybersecurity99 closed 2 years ago
JonZeolla
commented
2 years ago @cybersecurity99 Hmm I ran our end to end tests on Ubuntu 20.04 with Zeek 4.0.5 and it seems to be working. Are you seeing anything substantially different between your environment and what we are doing in Docker?
cybersecurity99
commented
2 years ago @cybersecurity99 Hmm I ran our end to end tests on Ubuntu 20.04 with Zeek 4.0.5 and it seems to be working. Are you seeing anything substantially different between your environment and what we are doing in Docker?
38
@JonZeolla I am not using Docker , I have tried this in Multiple VM . I am facing this in all attempts also if I install after failed test I get this when checking installation
./zeek -N Seiso::Kafka
error in /usr/local/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /usr/local/zeek/lib/zeek/plugins/packages/zeek-kafka//lib/SEISO-KAFKA.linux-x86_64.so: /usr/local/lib/librdkafka++.so.1: undefined symbol: rd_kafka_error_string
fatal error in /usr/local/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errorsIs this any configuration file or logs which I can check to see real reason for error
JonZeolla
commented
2 years ago @cybersecurity99 are you using the right version of librdkafka? We only currently support 1.4.2 or 1.4.4
cybersecurity99
commented
2 years ago @cybersecurity99 are you using the right version of librdkafka? We only currently support 1.4.2 or 1.4.4
@JonZeolla , yes I guess I downloaded from apt install librdkafka-dev. Also I tried to delete zeek-kafka packages and reinstall it It installed it without any error . But it doesn't show it ran 10 test cases successfully or failed it just showed running test cases and then installed . It is sending data to kafka But at the same time it is writing logs in zeek/logs/current folder how to stop that or is it permanent ?
JonZeolla
commented
2 years ago @cybersecurity99 at a quick glance that may be installing the wrong version of librdkafka. May be looking at the wrong thing via my phone.
https://ubuntu.pkgs.org/20.04/ubuntu-universe-arm64/librdkafka-dev_1.2.1-1ubuntu1_arm64.deb.html
You could use a modified version of https://github.com/JonZeolla/Development/blob/76d4bc84e222544baec9e8269d71f3cfa55c0869/python/librdkafka_version_checker.py to check the version you have on disk, or more simply you could just use apt/dpkg.
I'm not sure why this package would create different logs on disk than you already had... @ottobackwards any idea on that one
cybersecurity99
commented
2 years ago @cybersecurity99 at a quick glance that may be installing the wrong version of librdkafka. May be looking at the wrong thing via my phone.
https://ubuntu.pkgs.org/20.04/ubuntu-universe-arm64/librdkafka-dev_1.2.1-1ubuntu1_arm64.deb.html
You could use a modified version of https://github.com/JonZeolla/Development/blob/76d4bc84e222544baec9e8269d71f3cfa55c0869/python/librdkafka_version_checker.py to check the version you have on disk, or more simply you could just use apt/dpkg.
I'm not sure why this package would create different logs on disk than you already had... @ottobackwards any idea on that one
Hi @JonZeolla I tried that python script to check but got some error , is there any pre requisite to run that script
from: can't read /var/mail/ctypes
./librdkafka_version_checker.py: line 2: minver: command not found
./librdkafka_version_checker.py: line 3: syntax error near unexpected token `('
./librdkafka_version_checker.py: line 3: `dll = cdll.LoadLibrary("/usr/local/lib/librdkafka.so")'
Also you pointed out that 1.4.2 and 1.4.4 is supported so , we don't need to download librdkafka from apt ? Just that tar file and steps mentioned in 1st point are enough to run. Because I have done apt install librdkafka , its shows it to be 1.8.2 . So should I just remove them and use
$ curl -L https://github.com/edenhill/librdkafka/archive/v1.4.2.tar.gz | tar xvz
$ cd librdkafka-1.4.2/
$ ./configure --enable-sasl
$ make
$ sudo make installAfter doing so and reinstalling zeek-kafka I found that data was sent to kafka and only 1 log was generated in /logs/current/ folder that is .crash-diag.out which contained following info
root@zeek-VirtualBox:/usr/local/zeek/logs/current# cat .crash-diag.out
Zeek 5.0.0-dev.280
Linux 5.13.0-40-generic
Zeek plugins:
Seiso::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
==== No reporter.log
==== No stderr.log
==== No stdout.log
==== No .cmdline
==== No .env_vars
==== No .status
==== No prof.log
==== No packet_filter.log
==== No loaded_scripts.logWhat it shows ?
Also how to do we differentiate data in zeek kafka topic like http.log , conn.log ,etc
Edit When I tried by commenting zeek-kafka in local.zeek , there were no logs in current folder also Thanks
JonZeolla
commented
2 years ago @cybersecurity99 wait you're running Zeek 5? That's definitely not supported yet. Also, correct librdkafka 1.8.2 isn't supported
If updating your versions fixes things please close this issue, otherwise let me know and we can continue to troubleshoot.
cybersecurity99
commented
2 years ago @cybersecurity99 wait you're running Zeek 5? That's definitely not supported yet. Also, correct librdkafka 1.8.2 isn't supported
If updating your versions fixes things please close this issue, otherwise let me know and we can continue to troubleshoot.
@JonZeolla I took this in consideration and created a fresh machine with zeek v4.0.5 , and librdkafka 1.4.2 as given in commands
Still same issue it is sending data to kafka But at the same time writing logs in /zeek/logs/current Can you check this does it write for you also in that folder ?
JonZeolla
commented
2 years ago @cybersecurity99 just to be clear, it is sending logs to Kafka like you want, but it is also storing them on disk and you want to disable that? This project adds the ability to send logs to Kafka, it doesn't replace any local log writing, but you can disable that separately.
I suggest you check out https://docs.zeek.org/en/master/frameworks/logging.html#terminology and go from there
cybersecurity99
commented
2 years ago @cybersecurity99 just to be clear, it is sending logs to Kafka like you want, but it is also storing them on disk and you want to disable that? This project adds the ability to send logs to Kafka, it doesn't replace any local log writing, but you can disable that separately.
I suggest you check out https://docs.zeek.org/en/master/frameworks/logging.html#terminology and go from there
@JonZeolla okay got that part.
Just wanna know if send all logs together. How do we differentiate between http.log,conn.log ,etc at kafka end . Any command to print specific log type using topic "zeek"
JonZeolla
commented
2 years ago @cybersecurity99 I think you're looking for something like tag_json and then post processing the logs after receiving them from Kafka. You could also send different logs to different topics
Summary of the issue
... I tried to install zeek-kafka but it failed
Expected behavior
... It should get installed and work
Steps to reproduce
... installed librdkafka and libsasl2-dev
Then $ curl -L https://github.com/edenhill/librdkafka/archive/v1.4.2.tar.gz | tar xvz $ cd librdkafka-1.4.2/ $ ./configure --enable-sasl $ make $ sudo make install Where applicable, consider providing a patch that uses the end to end testing environment.
Logs, errors, etc.
... root@admin:/usr/local/zeek/bin# ./zkg install seisollc/zeek-kafka The following packages will be INSTALLED: zeek/seisollc/zeek-kafka (v1.0.0)
Verify the following REQUIRED external dependencies: (Ensure their installation on all relevant systems before proceeding): from zeek/seisollc/zeek-kafka (v1.0.0): librdkafka ~1.4.2-RC1
Proceed? [Y/n] y "zeek/seisollc/zeek-kafka" requires a "LIBRDKAFKA_ROOT" value (Path to librdkafka installation tree root): LIBRDKAFKA_ROOT: /usr/local Saved answers to config file: /usr/local/zeek/etc/zkg/config Running unit tests for "zeek/seisollc/zeek-kafka" error: failed to run tests for zeek/seisollc/zeek-kafka: test_command failed with exit code 1 Proceed to install anyway? [N/y]
Your environment