Driverkit fails to build Ubuntu releases

distortedsignal commented 2 years ago

Describe the bug

Current HEAD doesn't build successfully for supported Ubuntu releases.

AS A new developer to driverkit, I WANT to build the Ubuntu drivers to make sure that I'm doing things right WHEN I TRY... the driverkit fails with... lots of errors.

How to reproduce it

Clone the driverkit project into a new directory.
- mkdir repros && cd repros && git clone https://github.com/falcosecurity/driverkit.git && cd driverkit
Build driverkit using make build
- make build
- Output attached below in [1]
EXPECT: the driverkit binary to be built and stored at _output/bin/driverkit
- This step completed successfully
Attempt to use the new driverkit binary to build drivers for Ubuntu distros listed in the driverkit README
1. Make files that specify how drivers/eBPFs should be built.
  - These files are listed in [2]
2. Call the newly created driverkit binary with these files as the config file.
  - I initially tried this with debug logging, and... y'all love your debug logging, huh? I got like 30k lines worth of logs off one build.
  - _output/bin/driverkit docker -c /home/tom/Documents/src/falco/ubuntu-generic.yaml &> ./ubuntu-generic.log
  - _output/bin/driverkit docker -c /home/tom/Documents/src/falco/ubuntu-aws.yaml &> ./ubuntu-aws.log
3. EXPECT: the drivers to be created and placed in the specified file/directory.
  - And... this is where we fail. No compiled files are created.
  - Log files are collected in [3]

[1] Make build output

``` rm -f -R dist rm -f -R _output CGO_ENABLED=1 go build -v -ldflags '-X github.com/falcosecurity/driverkit/pkg/version.buildTime=1645123753 -X github.com/falcosecurity/driverkit/pkg/version.gitCommit=10b957d -X github.com/falcosecurity/driverkit/pkg/version.gitTag=v0.5.0 -X github.com/falcosecurity/driverkit/pkg/version.commitsFromGitTag=3 -X github.com/falcosecurity/driverkit/pkg/driverbuilder.builderBaseImage=docker.io/falcosecurity/driverkit-builder:10b957d' -tags 'sqlite_omit_load_extension linux' -o _output/bin/driverkit . # github.com/mattn/go-sqlite3 sqlite3-binding.c: In function ‘sqlite3SelectNew’: sqlite3-binding.c:128049:10: warning: function may return address of local variable [-Wreturn-local-addr] 128049 | return pNew; | ^~~~ sqlite3-binding.c:128009:10: note: declared here 128009 | Select standin; | ^~~~~~~ ```

[2] Files passed to driverkit

Ubuntu Generic File

```yaml kernelrelease: 4.15.0-72-generic kernelversion: 81 target: ubuntu-generic output: module: /home/tom/Documents/src/falco/driver-out/falco-ubuntu-generic.ko probe: /home/tom/Documents/src/falco/driver-out/falco-ubuntu-generic.o driverversion: master ```

AWS Ubuntu File

```yaml kernelrelease: 4.15.0-1057-aws kernelversion: 59 target: ubuntu-aws output: module: /home/tom/Documents/src/falco/driver-out/falco-ubuntu-aws.ko probe: /home/tom/Documents/src/falco/driver-out/falco-ubuntu-aws.o driverversion: master ```

[3] Build output log files

Ubuntu Generic Log file

``` INFO using config file file=/home/tom/Documents/src/falco/ubuntu-generic.yaml INFO driver building, it will take a few seconds processor=docker FATA exiting error="Error: No such container:path: e1135c7a0565ed73c3ecfdaeaf1fb0f32ccb68966b96f354cd3ecd3f6f0950a8:/tmp/driver/module.ko" ```

AWS Ubuntu Log file

``` INFO using config file file=/home/tom/Documents/src/falco/ubuntu-aws.yaml INFO driver building, it will take a few seconds processor=docker FATA exiting error="Error: No such container:path: befba3c012fe1c7c83f99f14976782f0783588b4f437b923343a547569c44db0:/tmp/driver/module.ko" ```

Environment

Falco version: N/A
System info: N/A
Cloud provider or hardware configuration: N/A

OS:

NAME="Manjaro Linux"
ID=manjaro
ID_LIKE=arch
BUILD_ID=rolling
PRETTY_NAME="Manjaro Linux"
ANSI_COLOR="32;1;24;144;200"
HOME_URL="https://manjaro.org/"
DOCUMENTATION_URL="https://wiki.manjaro.org/"
SUPPORT_URL="https://manjaro.org/"
BUG_REPORT_URL="https://bugs.manjaro.org/"
LOGO=manjarolinux

Kernel:

Linux tom-virtualbox 5.16.7-1-MANJARO #1 SMP PREEMPT Sun Feb 6 12:22:29 UTC 2022 x86_64 GNU/Linux

Installation method: Uh... from an ISO? Not sure how this is relevant.

Additional context

distortedsignal commented 2 years ago

Forgot to check my formatting, give me a minute.

distortedsignal commented 2 years ago

Ok, after more careful study (read: I made more files and ran the driverkit a lot), I was able to get AmazonLinux 2 to successfully compile both the kernel module and eBPF probe, I was able to successfully compile just the kernel module for AmazonLinux, and I was able to successfully compile the kernel module but not the eBPF module for the vanilla linux *.yaml file.

The vanilla linux output says that it cannot find the eBPF probe compiled output:

INFO using config file                             file=/home/tom/Documents/src/falco/vanilla.yaml
INFO driver building, it will take a few seconds   processor=docker
INFO kernel module available                       path=/home/tom/Documents/src/falco/driver-out/falco-vanilla.ko
FATA exiting                                       error="Error: No such container:path: 913092e6f8f848e672b2643b85b7d15b0a61b878eb7a72d1a20bfedb55aa1a9a:/tmp/driver/bpf/probe.o"

The other failures here are pretty interesting.

The centos compiles all get to the following log line and just... hang, without launching a docker container:

INFO using config file                             file=/home/tom/Documents/src/falco/centos(6|7|8).yaml
INFO driver building, it will take a few seconds   processor=docker

Interestingly, when run with debug logging, not a whole lot else is learned: Centos6:

INFO using config file                             file=/home/tom/Documents/src/falco/centos6.yaml
DEBU running with options                          driverversion=master kernelrelease=2.6.32-754.14.2.el6.x86_64 kernelversion=1 output-module=/home/tom/Documents/src/falco/driver-out/falco-centos6.ko target=centos
INFO driver building, it will take a few seconds   processor=docker
DEBU doing a new docker build

Centos7:

INFO using config file                             file=/home/tom/Documents/src/falco/centos7.yaml
DEBU running with options                          driverversion=master kernelrelease=3.10.0-957.12.2.el7.x86_64 kernelversion=1 output-module=/home/tom/Documents/src/falco/driver-out/falco-centos7.ko target=centos
INFO driver building, it will take a few seconds   processor=docker
DEBU doing a new docker build

Centos8:

INFO using config file                             file=/home/tom/Documents/src/falco/centos8.yaml
DEBU running with options                          driverversion=master kernelrelease=4.18.0-147.5.1.el8_1.x86_64 kernelversion=1 output-module=/home/tom/Documents/src/falco/driver-out/falco-centos8.ko target=centos
INFO driver building, it will take a few seconds   processor=docker
DEBU doing a new docker build

Debian can't find its kernel headers:

INFO using config file                             file=/home/tom/Documents/src/falco/debian.yaml
INFO driver building, it will take a few seconds   processor=docker
FATA exiting                                       error="kernel headers not found"

And while the Ubuntu drivers don't error in compilation, they do error when trying to recover the kernel module from the container:

Ubuntu AWS:

INFO using config file                             file=/home/tom/Documents/src/falco/ubuntu-aws.yaml
INFO driver building, it will take a few seconds   processor=docker
FATA exiting                                       error="Error: No such container:path: c27c7f03f871c91ef737fa13274b00c7078872a12c2e3aa80ec4925018a953ae:/tmp/driver/module.ko"

Ubuntu Generic:

INFO using config file                             file=/home/tom/Documents/src/falco/ubuntu-generic.yaml
INFO driver building, it will take a few seconds   processor=docker
FATA exiting                                       error="Error: No such container:path: e524c25a08cc4038a6f3e7a58ce34ad03500ce14820e0cd1f34d9b3825f0d9db:/tmp/driver/module.ko"

So... I'm not sure what to do with this information. Seems like there's some seriously broken stuff in this project.

distortedsignal commented 2 years ago

Bump.

dwindsor commented 2 years ago

We tracked this down to a missing package (zstd) and an out-of-date version of binutils. See #127 and PR#128.

distortedsignal commented 2 years ago

Ok, so it looks like the Ubuntu builds work now, but now the amazon builds are broken.

~/Documents/src/falco/driverkit # master > ls ..                                                                                                                                             ✔ 
amazonlinux_2.yaml  amazonlinux.yaml  centos6.yaml  centos7.yaml  centos8.yaml  debian.yaml  driverkit  driver-out  ubuntu-aws.yaml  ubuntu-generic.yaml  vanilla.yaml
~/Documents/src/falco/driverkit # master > _output/bin/driverkit docker -c /home/tom/Documents/src/falco/amazonlinux_2.yaml                                                                  ✔ 
INFO using config file                             file=/home/tom/Documents/src/falco/amazonlinux_2.yaml
INFO driver building, it will take a few seconds   processor=docker
FATA exiting                                       error="Error: No such container:path: 3db37cf770b89c4d634f68c006b7f7fe667ac1750f9ffb926886430476f872ab:/tmp/driver/module.ko"
~/Documents/src/falco/driverkit # master > _output/bin/driverkit docker -c /home/tom/Documents/src/falco/amazonlinux.yaml                                                       1 ✘  1m 43s  
INFO using config file                             file=/home/tom/Documents/src/falco/amazonlinux.yaml
INFO driver building, it will take a few seconds   processor=docker
FATA exiting                                       error="Error: No such container:path: dee0b0d98d50b67c31998843910bbf28c1c42e5f82d1025659e93d7a930fc76d:/tmp/driver/module.ko"
~/Documents/src/falco/driverkit # master > _output/bin/driverkit docker -c /home/tom/Documents/src/falco/centos6.yaml                                                            1 ✘  1m 2s  
INFO using config file                             file=/home/tom/Documents/src/falco/centos6.yaml
INFO driver building, it will take a few seconds   processor=docker
FATA exiting                                       error="Error: No such container:path: 7512d2823da0d69b0a67f1f9fcf5bc9b199ccc5f81eacb4608b6d70e722c7f43:/tmp/driver/module.ko"
~/Documents/src/falco/driverkit # master > _output/bin/driverkit docker -c /home/tom/Documents/src/falco/centos7.yaml                                                              1 ✘  26s  
INFO using config file                             file=/home/tom/Documents/src/falco/centos7.yaml
INFO driver building, it will take a few seconds   processor=docker
INFO kernel module available                       path=/home/tom/Documents/src/falco/driver-out/falco-centos7.ko
~/Documents/src/falco/driverkit # master > _output/bin/driverkit docker -c /home/tom/Documents/src/falco/centos8.yaml                                                                ✔  30s  
INFO using config file                             file=/home/tom/Documents/src/falco/centos8.yaml
INFO driver building, it will take a few seconds   processor=docker
INFO kernel module available                       path=/home/tom/Documents/src/falco/driver-out/falco-centos8.ko
~/Documents/src/falco/driverkit # master > _output/bin/driverkit docker -c /home/tom/Documents/src/falco/debian.yaml                                                                 ✔  33s  
INFO using config file                             file=/home/tom/Documents/src/falco/debian.yaml
INFO driver building, it will take a few seconds   processor=docker
FATA exiting                                       error="kernel headers not found"
~/Documents/src/falco/driverkit # master > _output/bin/driverkit docker -c /home/tom/Documents/src/falco/ubuntu-aws.yaml                                                            1 ✘  4s  
INFO using config file                             file=/home/tom/Documents/src/falco/ubuntu-aws.yaml
INFO driver building, it will take a few seconds   processor=docker
INFO kernel module available                       path=/home/tom/Documents/src/falco/driver-out/falco-ubuntu-aws.ko
INFO eBPF probe available                          path=/home/tom/Documents/src/falco/driver-out/falco-ubuntu-aws.o
~/Documents/src/falco/driverkit # master > _output/bin/driverkit docker -c /home/tom/Documents/src/falco/ubuntu-generic.yaml                                                         ✔  50s  
INFO using config file                             file=/home/tom/Documents/src/falco/ubuntu-generic.yaml
INFO driver building, it will take a few seconds   processor=docker
INFO kernel module available                       path=/home/tom/Documents/src/falco/driver-out/falco-ubuntu-generic.ko
INFO eBPF probe available                          path=/home/tom/Documents/src/falco/driver-out/falco-ubuntu-generic.o
~/Documents/src/falco/driverkit # master > _output/bin/driverkit docker -c /home/tom/Documents/src/falco/vanilla.yaml --timeout=600                                                  ✔  56s  
INFO using config file                             file=/home/tom/Documents/src/falco/vanilla.yaml
INFO driver building, it will take a few seconds   processor=docker
FATA exiting                                       error="Error: No such container:path: 3db183e923c77639fcaf87269f998a032c676d9e5d1c16f94a5182c2b76cc9eb:/tmp/driver/module.ko"
~/Documents/src/falco/driverkit # master >                                                                                                                                      1 ✘  1m 0s 

So to summarize:	Distro	Return 0?	Outputs .o file?	Outputs .ko file?
amazonlinux_2	❌	❌	❌	`Error: No such container:path: $container_id:/tmp/driver/module.ko`
amazonlinux	❌	❌	❌	`Error: No such container:path: $container_id:/tmp/driver/module.ko`
centos6	❌	❌	❌	`Error: No such container:path: $container_id:/tmp/driver/module.ko`
centos7	✅	❌	✅	❌
centos8	✅	❌	✅	❌
debian	❌	❌	❌	`kernel headers not found`
ubuntu-aws	✅	✅	✅	❌
ubuntu-generic	✅	✅	✅	❌
vanilla	❌	❌	✅	`Error: No such container:path: $container_id:/tmp/driver/module.ko`

So now I have more questions about how the return code is determined and how the driverkit determines where/when .o and .ko files come from.

distortedsignal commented 2 years ago

Wait - I might have had a dirty output directory for my prior comment. Retrying now.

EDIT - Ok, I updated the output grid after cleaning the output directory.

FedeDP commented 2 years ago

@distortedsignal did you try to increase the timeout? In #143 i was getting similar issues, and it was a timeout problem. I bumped default timeout to 120s (up from 60s).

You can just pass --timeout 120 and test.

FedeDP commented 2 years ago

@distortedsignal can you run again above tests using latest driverkit master? I am curious to see if we fixed any of those!

Btw we also have various issues that will be solved with multiple builder images; i discovered that eg: centos 3.10 is failing with:

DEBU   CC [M]  /tmp/driver/main.o         
DEBU fIn file included from include/linux/compiler.h:54:0, 
DEBU                  from include/uapi/linux/stddef.h:1, 
DEBU                  from include/linux/stddef.h:4, 
DEBU                  from /tmp/kernel/include/uapi/linux/posix_types.h:4, 
DEBU                  from include/uapi/linux/types.h:13, 
DEBU                  from include/linux/types.h:5, 
DEBU                  from include/uapi/linux/audit.h:27, 
DEBU                  from /tmp/kernel/arch/x86/include/asm/syscall.h:16, 
DEBU                  from /tmp/driver/main.c:25: 
DEBU include/linux/compiler-gcc.h:103:30: fatal error: linux/compiler-gcc5.h: No such file or directory

using gcc-5, but then using gcc 4.8 fails with:

DEBU mMakefile:619: Cannot use CONFIG_CC_STACKPROTECTOR_STRONG: -fstack-protector-strong not supported by compile

We need some more builder images :)

FedeDP commented 1 year ago

Hi! Can you retest using driverkit master and forcing --builderimage auto:master ? @distortedsignal

poiana commented 1 year ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

poiana commented 1 year ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

poiana commented 1 year ago

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community. /close

poiana commented 1 year ago

@poiana: Closing this issue.

In response to [this](https://github.com/falcosecurity/driverkit/issues/126#issuecomment-1434396627): >Rotten issues close after 30d of inactivity. > >Reopen the issue with `/reopen`. > >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Provide feedback via https://github.com/falcosecurity/community. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

falcosecurity / driverkit

Driverkit fails to build Ubuntu releases #126