search

stackrox / collector

Runtime data collection for the StackRox Kubernetes Security Platform using eBPF
Apache License 2.0
51 stars 24 forks source link

Collector eBPF kernel-module error in On Premise Kubernetes Cluster #896

Closed The3fon closed 2 years ago

The3fon commented 2 years ago

Hello! The collector pods are bouncing between Running and CrashLoopBackOff due to the below error:

[2022-11-08T13:30:43.219349718+03:00 I 20221108 103043 collector.cpp:329] Attempting to download eBPF probe - Candidate kernel versions: 

[I 20221108 103043 collector.cpp:331] 5.15.35-5.el7.3.x86_64
[I 20221108 103043 GetKernelObject.cpp:180] Local storage does not contain collector-ebpf-5.15.35-5.el7.3.x86_64.o
[2022-11-08T13:30:43.440220207+03:00 I2022-11-08T13:30:43.440283536+03:00  20221108 103043 FileDownloader.cpp:316] Fail to download /module/collector-ebpf.o.gz - Failed writing body (0 != 10)
[I 20221108 103043 FileDownloader.cpp:318] HTTP Request failed with error code '404' - HTTP Body Response: not found
[I 20221108 103044 FileDownloader.cpp:316] Fail to download /module/collector-ebpf.o.gz - Failed writing body (0 != 10)
[I 2022112022-11-08T13:30:44.445460645+03:00 08 103044 FileDownloader.cpp:318] HTTP Request failed with error code '404' - HTTP Body Response: not found
......
[W 20221108 103112 FileDownloader.cpp:332] Failed to download /module/collector-ebpf.o.gz
[W 20221108 103112 GetKernelObject.cpp:183] Unable to download kernel object collector-ebpf-5.15.35-5.el7.3.x86_64.o to /module/collector-ebpf.o.gz
[W 2022112022-11-08T13:31:12.584992655+03:00 08 103112 collector.cpp:343] Error getting kernel object: collector-ebpf-5.15.35-5.el7.3.x86_64.o
[I 20221108 103112 collector.cpp:215] gRPC server=sensor.stackrox.svc:443
[2022-11-08T13:31:12.585667313+03:00 I2022-11-08T13:31:12.585678270+03:00  2022-11-08T13:31:12.585689123+03:00 20222022-11-08T13:31:12.585698916+03:00 112022-11-08T13:31:12.585708326+03:00 082022-11-08T13:31:12.585717352+03:00  2022-11-08T13:31:12.585726504+03:00 102022-11-08T13:31:12.585735847+03:00 312022-11-08T13:31:12.585745043+03:00 122022-11-08T13:31:12.585754222+03:00  2022-11-08T13:31:12.585763262+03:00 collector.cpp2022-11-08T13:31:12.585772387+03:00 :2022-11-08T13:31:12.585781692+03:00 3572022-11-08T13:31:12.585790958+03:00 ] 2022-11-08T13:31:12.585800229+03:00 Attempting to connect to GRPC server2022-11-08T13:31:12.585809299+03:00 

[2022-11-08T13:31:12.585830551+03:00 E2022-11-08T13:31:12.585839485+03:00  2022-11-08T13:31:12.585849167+03:00 20222022-11-08T13:31:12.585858503+03:00 112022-11-08T13:31:12.585868634+03:00 082022-11-08T13:31:12.585877519+03:00  2022-11-08T13:31:12.585886703+03:00 102022-11-08T13:31:12.585895845+03:00 312022-11-08T13:31:12.585904982+03:00 122022-11-08T13:31:12.585913900+03:00  2022-11-08T13:31:12.585922860+03:00 collector.cpp2022-11-08T13:31:12.585931770+03:00 :2022-11-08T13:31:12.585940638+03:00 3592022-11-08T13:31:12.585949507+03:00 ] 2022-11-08T13:31:12.585958593+03:00 Unable to connect to the GRPC server.2022-11-08T13:31:12.585967404+03:00 

[2022-11-08T13:31:12.586043197+03:00 F2022-11-08T13:31:12.586052670+03:00  2022-11-08T13:31:12.586062981+03:00 20222022-11-08T13:31:12.586072428+03:00 112022-11-08T13:31:12.586081771+03:00 082022-11-08T13:31:12.586090799+03:00  2022-11-08T13:31:12.586100030+03:00 102022-11-08T13:31:12.586125944+03:00 312022-11-08T13:31:12.586135654+03:00 122022-11-08T13:31:12.586145126+03:00  2022-11-08T13:31:12.586154678+03:00 collector.cpp2022-11-08T13:31:12.586178476+03:00 :368] No suitable kernel object downloaded

How can I troubleshoot? How cain i build own kernel-module?

Molter73 commented 2 years ago

Hi @The3fon, looks like that kernel is for a RedOS system, could you confirm this? We currently don't support that system out of the box so you will need to create your own drivers and make them available for collector to use.

If you have access to a system with the headers for that kernel, you could use the make -C kernel-modules drivers at the root of the repository to build the eBPF probe and kernel modules: https://github.com/stackrox/collector/blob/0025bc7abd6b177e3462c414cc09b24bced9ddd8/kernel-modules/Makefile#L45-L53

Once the build finishes, you should see the driver under kernel-modules/container/kernel-modules.

At this point you have a few options to make them available for collector:

The3fon commented 2 years ago

Thank @Molter73 for fast answer. Yes, its RedOS system. I have trouble for build module, my steps:

  1. Clone repo https://github.com/stackrox/collector.git
  2. ChangeDir to "collector"
  3. make -C kernel-modules drivers
  4. Output log: 
    
docker build  -t build-kernel-modules-fc36 ./build -f build/Dockerfile.fc36
Sending build context to Docker daemon  32.26kB
Step 1/7 : FROM fedora:36
....
Successfully built a60654025b55
Successfully tagged build-kernel-modules-fc36:latest
docker run --rm \
    -v /root/collector/kernel-modules/..:/collector \
    -v /usr/include/bpf:/usr/include/bpf:ro \
    -v /lib/modules/:/lib/modules/:ro \
    -v /usr/src:/usr/src:ro \
    build-kernel-modules-fc36:latest \
    /collector/kernel-modules/dev/build-drivers.sh
CMake Error: The source directory "/collector/falcosecurity-libs" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.


What does include CMakeLists.txt?
Molter73 commented 2 years ago

Looks like you might have made a shallow copy of the repo, run git submodule update --init at the root of the repository and make sure there are files in the falcosecurity-libs/ subdirectory, including a CMakeLists.txt file.

$ ls falcosecurity-libs
CMakeLists.txt               COPYING                      README.md                    driver                       test
CMakeListsGtestInclude.cmake NOTICES                      cmake                        proposals                    userspace
CODE_OF_CONDUCT.md           OWNERS                       coding_conventions.md        release.md
The3fon commented 2 years ago

@Molter73, It's my mistake, I didn't read the documentation carefully. Thank you for your help, the module build without errors.