search

zeek / zeek-af_packet-plugin

Plugin providing native AF_Packet support for Zeek.
Other
33 stars 30 forks source link

Fails to automatically detect headers on Debian #13

Closed mkgvt closed 4 years ago

mkgvt commented 5 years ago

Installing using "bro-pkg install bro-af_packet-plugin" on bro v2.6.4 on Debian 10 installed by adding the repository and installing manually as per the instructions. (I also saw the same problem when installing on v.2.6.1. Finally gave up, compiled and installed by hand. Eventually it would be nice to have bro-pkg work.)

$ dpkg -l '*bro*' Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=================-============-============-====================================================== ii bro 2.6.4-0 amd64 Bro is a powerful framework for network analysis and s ii bro-core 2.6.4-0 amd64 Bro is a powerful framework for network analysis and s ii bro-pkg 1.5.2-1 all Bro Package Manager ii broctl 2.6.4-0 amd64 Bro's interactive shell for operating Bro installation

$ uname -a Linux zeekmgr 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20) x86_64 GNU/Linux $ dpkg -l linux-headers-4.19.0-6-amd64 Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-============================-=================-============-===================================== ii linux-headers-4.19.0-6-amd64 4.19.67-2+deb10u1 amd64 Header files for Linux 4.19.0-6-amd64

$ cat /root/.bro-pkg/logs/bro-af_packet-plugin-build.log === STDERR === CMake Error at CMakeLists.txt:6 (include): include could not find load file:

BroPlugin

CMake Warning at CMakeLists.txt:8 (find_package): By not providing "FindKernelHeaders.cmake" in CMAKE_MODULE_PATH this project has asked CMake to find a package configuration file provided by "KernelHeaders", but CMake did not find one.

Could not find a package configuration file provided by "KernelHeaders" with any of the following names:

KernelHeadersConfig.cmake
kernelheaders-config.cmake

Add the installation prefix of "KernelHeaders" to CMAKE_PREFIX_PATH or set "KernelHeaders_DIR" to a directory containing one of the above files. If "KernelHeaders" provides a separate development package or SDK, be sure it has been installed.

CMake Error at CMakeLists.txt:22 (message): Kernel headers not found.

=== STDOUT === Build Directory : build Bro Source Directory : -- The C compiler identification is GNU 8.3.0 -- The CXX compiler identification is GNU 8.3.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- wo`rks -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Configuring incomplete, errors occurred! See also "/root/.bro-pkg/testing/bro-af_packet-plugin/clones/bro-af_packet-plugin/build/CMakeFiles/CMakeOutput.log".

JustinAzoff commented 5 years ago

Finally gave up, compiled and installed by hand.

How exactly did you do that? bro-pkg just clones the repository and runs ./configure && make

mkgvt commented 5 years ago

On Wed, Oct 23, 2019 at 1:29 PM Justin notifications@github.com wrote:

Finally gave up, compiled and installed by hand.

How exactly did you do that? bro-pkg just clones the repository and runs ./configure && make

These are the commands I used to compile the plugin by hand on v2.6.1 on Debian 9: $ browse https://github.com/J-Gras/bro-af_packet-plugin $ CC=clang CXX=clang++ ./configure --bro-dist=/path/to/zeek --with-kernel=/usr/src/linux-headers-4.9.0-8-amd64/ 2>&1 | tee xConfig.log $ CC=clang CXX=clang++ bash -c 'time make' 2>&1 | tee xMake.log $ CC=clang CXX=clang++ bash -c 'sudo make install' 2>&1 | tee xInstall.log

Note: since I installed Bro v2.6.1 using clang and clang++, I used the same for the plugin. No idea if clang or gcc perform better but I like clang's error messages better.

Mark

JustinAzoff commented 5 years ago

Ah, so that explains it.. bro-pkg just runs ./configure, you were using ./configure --bro-dist=/path/to/zeek --with-kernel=/usr/src/linux-headers-4.9.0-8-amd64. Your problem isn't with bro-pkg, but with ./configure not working without options that you shouldn't need to be passing...

Is bro-config in your path? the configure script tries to run bro-config --bro_dist to figure that out automatically.

mkgvt commented 5 years ago

On Wed, Oct 23, 2019 at 9:07 PM Justin notifications@github.com wrote:

Ah, so that explains it.. bro-pkg just runs ./configure, you were using ./configure --bro-dist=/path/to/zeek --with-kernel=/usr/src/linux-headers-4.9.0-8-amd64. Your problem isn't with bro-pkg, but with ./configure not working without options that you shouldn't need to be passing...

Is bro-config in your path? the configure script tries to run bro-config --bro_dist to figure that out automatically.

It is found at /opt/bro/bin/bro-config and is in my path. bro-config --bro_dist returns nothing. I peeked inside the script and the bro_dist variable is not set. What should it be set to?

Mark

J-Gras commented 5 years ago

I reproduced the issue. The main problem is that Debian seems to put the headers into directories using weird suffixes like -common. I need to adapt the cmake file accordingly.

JustinAzoff commented 5 years ago

Ah, I double checked how configure works, it actually uses bro-config --cmake_dir first to see if you are on a more recent version that supports building plugins without the bro source tree.

bro-config --version is definitely outputting 2.6.4?

I'm not sure why any of the variables in that script would be empty though if you built 2.6.4 from source. Something seems off with your installation, not sure what though.

mkgvt commented 5 years ago

On Thu, Oct 24, 2019 at 10:04 AM Justin notifications@github.com wrote:

Ah, I double checked how configure works, it actually uses bro-config --cmake_dir first to see if you are on a more recent version that supports building plugins without the bro source tree.

Starting with which version does it allow building without the source tree?

bro-config --version is definitely outputting 2.6.4?

bro-config --version

2.6.4

I'm not sure why any of the variables in that script would be empty though if you built 2.6.4 from source. Something seems off with your installation, not sure what though.

I am not building from source but trying to use the pre-built Debian package. (I built the previous bro version (2.6.1) and plugin from source, also due to hitting the same issue when installing via the Debian package. Compiling from source worked fine. But I would prefer to use the pre-built Debian package and bro-pkg for ease of upgrading.)

Executing bro-config --bro_dist from the Debian package gives no output. I checked the script itself and found "bro_dist=" which explains why it returns nothing. So it appears that the Debian package doesn't create the script with the variable set like it should. (That is the only variable in the script which isn't set.)

I get the feeling that I may be one of the only folks installing pre-built packages on Debian. :-)

Mark -- Mark Gardner

JustinAzoff commented 5 years ago

2.6.0 was the first version:

  • "make install" now installs Bro's include headers (and more) into "--prefix" so that compiling plugins no longer needs access to a source/build tree. For OS distributions, this also facilitates creating "bro-devel" packages providing all files necessary to build plugins.

If you installed from a package then bro_dist should be blank, you don't actually have the source tree. However, you should have the cmake_dir. It looks like there is an issue with the binary packages missing that directory:

root@395583468d09:~# bro-config --cmake_dir
/opt/bro/share/bro/cmake
root@395583468d09:~# ls /opt/bro/share/bro/cmake
ls: cannot access '/opt/bro/share/bro/cmake': No such file or directory

which is why your build is failing with 

include could not find load file:
BroPlugin

So, it Jan is able to fix the kernel headers issue for debian, but it looks like the binary packages are missing some files that should be present that make it impossible to currently build plugins against them :-( edit: it seems we may just have some bugs in the package description files. Looks like there is supposed to be a bro-dev or zeek-dev package that has the needed files, but that isn't getting built.

In any case, none of this is a problem with bro-pkg.

wraezor commented 4 years ago

I am also getting this error with installing the bro-af_packet-plugin on Debian 10 using zkg. My Zeek instance is built from source (3.0.1), but the zkg install is failing with a message seen above.

CMake Error at CMakeLists.txt:22 (message):
  Kernel headers not found.

I do have the two header 'packages' installed.

root@bf8278d8efb0:# dpkg -l linux-headers*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                          Version      Architecture Description
+++-=============================-============-============-======================================
ii  linux-headers-4.19.0-8-amd64  4.19.98-1    amd64        Header files for Linux 4.19.0-8-amd64
ii  linux-headers-4.19.0-8-common 4.19.98-1    all          Common header files for Linux 4.19.0-8

root@bf8278d8efb0:# ls /usr/src/
linux-headers-4.19.0-8-amd64  linux-headers-4.19.0-8-common  linux-kbuild-4.19

Does FindKernelHeaders.cmake need to be modified?

J-Gras commented 4 years ago

You are right, FindKernelHeaders.cmake has to be adapted. So far I haven't found the time to do the necessary changes.

0xxon commented 4 years ago

Just FYI - the deb-based binary packages are updated to include the necessary cmake subdirectory now (after someone pointed me here :) ).

(On the rpm-based ones that always was the case - they have better notifications in place when you mess something like this up).

wraezor commented 4 years ago

Thanks @0xxon. Any idea when these changes will make it upstream and consequently into the Zeek package repository?

0xxon commented 4 years ago

This should be in all our binary packages since March 12th. As long as you install the -devel package, you should get the cmake directory in your install.

mkgvt commented 4 years ago

It appears that the problem of not finding the kernel header files still exists with 3.0.6-0.

To reproduce: install on top of Debian Live DVD debian-live-10.3.0-amd64-xfce.iso following the instructions on the web site:

# echo 'deb http://download.opensuse.org/repositories/security:/zeek/Debian_10/ /' > /etc/apt/sources.list.d/zeek.list
# apt-get install wget
# wget -nv https://download.opensuse.org/repositories/security:zeek/Debian_10/Release.key -O Release.key
# apt-key add - < Release.key
# apt-get update
# apt-get install zeek-lts zeek-lts-core-dev
# dpkg -l zeek-lts
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=====================================================================
ii  zeek-lts       3.0.6-0      amd64        Zeek is a powerful framework for network analysis and security monito

Now attempt to install zeek-af_packet-plugin:

# apt-get install python-pip git cmake
# pip install zkg
# zkg install zeek-af_packet-plugin
The following packages will be INSTALLED:
  zeek/j-gras/zeek-af_packet-plugin (2.0.0)

Proceed? [Y/n] Y
Running unit tests for "zeek/j-gras/zeek-af_packet-plugin"
error: failed to run tests for zeek/j-gras/zeek-af_packet-plugin: package build_command failed, see log in /root/.zkg/logs/zeek-af_packet-plugin-build.log
Proceed to install anyway? [N/y] N

# cat /root/.zkg/logs/zeek-af_packet-plugin-build.log
=== STDERR ===
CMake Error at CMakeLists.txt:22 (message):
  Kernel headers not found.

=== STDOUT ===
Build Directory        : build
Zeek Source Directory   :
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test cxx11_header_works
-- Performing Test cxx11_header_works - Success
-- Found BinPAC: /opt/zeek/bin/binpac
-- Found CAF: /opt/zeek/lib/libcaf_core_static.a;/opt/zeek/lib/libcaf_io_static.a;/opt/zeek/lib/libcaf_openssl_static.a  found components:  core io openssl
-- Found Broker: /opt/zeek/lib/libbroker.a
-- Zeek executable      : /opt/zeek/bin/zeek
-- Zeek source          :
-- Zeek build           :
-- Zeek install prefix  : /opt/zeek
-- Zeek plugin directory: /opt/zeek/lib/zeek/plugins
-- Zeek debug mode      :
-- Could NOT find KernelHeaders (missing: KERNELHEADERS_ROOT_DIR)
-- Configuring incomplete, errors occurred!
See also "/root/.zkg/testing/zeek-af_packet-plugin/clones/zeek-af_packet-plugin/build/CMakeFiles/CMakeOutput.log".

# uname -r
4.19.0-8-amd64

# dpkg -l linux-headers-4.19.0-8*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                          Version      Architecture Description
+++-=============================-============-============-======================================
ii  linux-headers-4.19.0-8-amd64  4.19.98-1    amd64        Header files for Linux 4.19.0-8-amd64
ii  linux-headers-4.19.0-8-common 4.19.98-1    all          Common header files for Linux 4.19.0-8

The kernel headers are install so it appears to still having problems finding the kernel header files.

0xxon commented 4 years ago

That should be a problem of the package and not of the Zeek headers - so that's probably one for @J-Gras

wschoot commented 4 years ago

While I'm with @0xxon that this is something in the package, I thought I'd share the quick and dirty hack I've found for this issue to get at least the package to compile/install

As the searches for a file named include/linux/user.h, the logic in /root/.zkg/clones/package/zeek-af_packet-plugin/cmake/FindKernelHeaders.cmake fails as Debian installs these not in /usr/src/linux-headers-$(uname -r) but in a shared / common package which is in another subdirectory.

Trick it into thinking it's there is as easy as creating a symbolic link:

ln -s /usr/src/linux-headers-4.19.0-9-common/include/linux/ /usr/src/linux-headers-4.19.0-9-amd64/include/

(mind my version is a tiny bit newer as -9 was released instead of -8)

I realise this isn't anywhere near perfect but for now, it seems to work

dspruell-s01 commented 4 years ago

I've encountered this issue as well, trying to install package for Zeek 3.0.8.

# zkg install zeek-af_packet-plugin
The following packages will be INSTALLED:
  zeek/j-gras/zeek-af_packet-plugin (2.0.0)

Proceed? [Y/n] Y
Running unit tests for "zeek/j-gras/zeek-af_packet-plugin"
error: failed to run tests for zeek/j-gras/zeek-af_packet-plugin: package build_command failed, see log in /root/.zkg/logs/zeek-af_packet-plugin-build.log
Proceed to install anyway? [N/y] N
Abort.

zeek-af_packet-plugin-build.log:

=== STDERR ===
CMake Error at CMakeLists.txt:22 (message):
  Kernel headers not found.

=== STDOUT ===
Build Directory        : build
Zeek Source Directory   : 
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test cxx11_header_works
-- Performing Test cxx11_header_works - Success
-- Found BinPAC: /opt/zeek/bin/binpac  
-- Found CAF: /opt/zeek/lib/libcaf_core_static.a;/opt/zeek/lib/libcaf_io_static.a;/opt/zeek/lib/libcaf_openssl_static.a  found components:  core io openssl 
-- Found Broker: /opt/zeek/lib/libbroker.a  
-- Zeek executable      : /opt/zeek/bin/zeek
-- Zeek source          : 
-- Zeek build           : 
-- Zeek install prefix  : /opt/zeek
-- Zeek plugin directory: /opt/zeek/lib/zeek/plugins
-- Zeek debug mode      : 
-- Could NOT find KernelHeaders (missing: KERNELHEADERS_ROOT_DIR) 
-- Configuring incomplete, errors occurred!
See also "/root/.zkg/testing/zeek-af_packet-plugin/clones/zeek-af_packet-plugin/build/CMakeFiles/CMakeOutput.log".

Zeek 3.0.8 installed from Zeek's Debian package repo.

ii  zeek-lts                       3.0.8-0             amd64    Zeek is a powerful framework for network analysis and security monitoring.
ii  zeek-lts-core                  3.0.8-0             amd64    Zeek is a powerful framework for network analysis and security monitoring.
ii  zeek-lts-core-dev              3.0.8-0             amd64    Zeek is a powerful framework for network analysis and security monitoring.
ii  zeek-lts-libcaf-dev            3.0.8-0             amd64    CAF development files
ii  zeekctl-lts                    3.0.8-0             amd64    Zeek's interactive shell for operating Zeek installations.

ii  linux-headers-4.19.0-10-amd64  4.19.132-1          amd64    Header files for Linux 4.19.0-10-amd64
ii  linux-headers-4.19.0-10-common 4.19.132-1          all      Common header files for Linux 4.19.0-10
ii  linux-headers-amd64            4.19+105+deb10u5    amd64    Header files for Linux amd64 configuration (meta-package)

I'm also partly confused that zeek-af_packet-plugin/cmake/FindKernelHeaders.cmake appears to support specifying KERNELHEADERS_ROOT_DIR but fiddling around with setting this to a couple of header file paths as an environment variable didn't seem to help.

# Variables used by this module, they can change the default behaviour and need
# to be set before calling find_package:
#
#  KERNELHEADERS_ROOT_DIR    Set this variable to the root directory of the
#                            kernel sources if the module has problems finding
#                            the proper path.
dspruell-s01 commented 4 years ago

Can verify @wschoot 's workaround enables successful installation.

# ln -s /usr/src/linux-headers-4.19.0-10-common/include/linux/ /usr/src/linux-headers-4.19.0-10-amd64/include/

# zkg install zeek-af_packet-plugin
The following packages will be INSTALLED:
  zeek/j-gras/zeek-af_packet-plugin (2.0.0)

Proceed? [Y/n] 
Running unit tests for "zeek/j-gras/zeek-af_packet-plugin"
Installing "zeek/j-gras/zeek-af_packet-plugin"..........
Installed "zeek/j-gras/zeek-af_packet-plugin" (2.0.0)
J-Gras commented 4 years ago

I just pushed a commit to master that should finally fix the detection of kernel headers on Debian. This can be tested using zkg install zeek-af_packet-plugin --version master.

wschoot commented 4 years ago

Can confirm this works. I'll use the new version with this fix as soon as it's released. Thanks!

J-Gras commented 4 years ago

Thanks a lot for confirming. The fix is now part of the 2.1.0 release.

wschoot commented 4 years ago

@J-Gras Sorry but I might've confirmed a bit too soon and did not properly read the error messages from my build. The containerbuild succeeded after tweaking your installline so I assumed the build was good, however upon further inspection, it doesn't work just yet.

Now as I'm building in a debiancontainer on an ubuntuhost I do need some way to make sure the ubuntukernel that's used in the debiancontainer matches the version numbering so I'm gonna try on vanilla debian virtual machine now, instead of container.

J-Gras commented 4 years ago

I am a bit confused. In general the packet should install fine if the headers are located on the default paths of their distribution. If that's not the case in you container setup, you can use the --with-kernel option of the manual build or the symlink workaround.

wschoot commented 4 years ago

I've tried building on a regular virtual machine and that works, except for a warning during unit tests, so closing this issue was still good. I'll try the suggested flag. Thanks.