Falco installation on Amazon Linux2 with ebpf probe and kubenetes version 1.24 fails

[ajinkya1986]

Describe the bug

Created a AWS EKS Cluster with Amazon Linux 2 image with ebpf probe enabled. We are getting the following error on the pods which has conflicting errors. First it is trying to load drivers for 0.32.0 then for 0.32.2

How to reproduce it

Create EKS cluster with Node Groups having Amazon Linux 2 Ami

Expected behaviour

Falco pods runs successfully after loading the ebpf drivers

Logs

falco * Setting up /usr/src links from host
falco * Running falco-driver-loader for: falco version=0.32.0, driver version=39ae7d40496793cf3d3e7890c9bbdc202263836b
falco * Running falco-driver-loader with: driver=bpf, compile=yes, download=yes
falco * Mounting debugfs
falco Detected an unsupported target system, please get in touch with the Falco community
falco Thu Apr 13 16:56:41 2023: Runtime error: Error reading config file(/etc/falco/falco.yaml): could not load plugins config: yaml-cpp: error at line 40, column 16: bad conversion. Exiting.
falco-driver-loader * Setting up /usr/src links from host
falco-driver-loader * Running falco-driver-loader for: falco version=0.32.2, driver version=2.0.0+driver
falco-driver-loader * Running falco-driver-loader with: driver=bpf, compile=yes, download=yes
falco-driver-loader * Mounting debugfs
falco-driver-loader mount: /sys/kernel/debug: permission denied.
falco-driver-loader * Trying to download a prebuilt eBPF probe from https://download.falco.org/driver/2.0.0%2Bdriver/x86_64/falco_amazonlinux2_5.10.173-154.642.amzn2.x86_64_1.o
falco-driver-loader * Skipping compilation, eBPF probe is already present in /root/.falco/falco_amazonlinux2_5.10.173-154.642.amzn2.x86_64_1.o
falco-driver-loader * eBPF probe located in /root/.falco/falco_amazonlinux2_5.10.173-154.642.amzn2.x86_64_1.o
falco-driver-loader * Success: eBPF probe symlinked to /root/.falco/falco-bpf.o

• Falco version: 0.32.2
• Kubernetes Version: 1.24
• Cloud provider or hardware configuration: AWS EKS
• OS: Amazon Linx 2
• Kernel: 5.10.173-154.642
• Installation method: helm

[mpurusottamc]

Adding more context: kubernetes version is 1.24

[Andreagit97]

The ebpf probe is successfully downloaded :thinking: There seems to be an error in the Falco configuration:

falco Thu Apr 13 16:56:41 2023: Runtime error: Error reading config file(/etc/falco/falco.yaml): could not load plugins config: yaml-cpp: error at line 40, column 16: bad conversion. Exiting.

Could you post your Falco config yaml file? Do you want to run just the ebpf probe or also some plugins?

[ajinkya1986]

I have added the falco config yaml file. Removed the rules

Name:         test-falco
Namespace:    test
Labels:       app.kubernetes.io/instance=test
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=falco
              app.kubernetes.io/version=0.32.2
              helm.sh/chart=falco-2.0.18
Annotations:  meta.helm.sh/release-name: test
              meta.helm.sh/release-namespace: test

falco.yaml:
----
buffered_outputs: false
file_output:
  enabled: false
  filename: ./events.txt
  keep_alive: false
grpc:
  bind_address: unix:///var/run/falco/falco.sock
  enabled: false
  threadiness: 0
grpc_output:
  enabled: false
http_output:
  enabled: false
  url: ""
  user_agent: falcosecurity/falco
httpOutput:
  enabled: true
  url: http://example-listener:8080/
json_include_output_property: true
json_include_tags_property: true
json_output: false
jsonIncludeOutputProperty: true
jsonOutput: true
libs_logger:
  enabled: false
  severity: debug
load_plugins: []
log_level: info
log_stderr: true
log_syslog: true
metadata_download:
  chunk_wait_us: 1000
  max_mb: 100
  watch_freq_sec: 1
output_timeout: 2000
outputs:
  max_burst: 1000
  rate: 1
plugins:
- init_config: null
  library_path: libk8saudit.so
  name: k8saudit
  open_params: http://:9765/k8s-audit
- library_path: libcloudtrail.so
  name: cloudtrail
- init_config: ""
  library_path: libjson.so
  name: json
priority: debug
program_output:
  enabled: false
  keep_alive: false
  program: 'jq ''{text: .output}'' | curl -d @- -X POST https://hooks.slack.com/services/XXX'
rules_file:
- /etc/falco/falco_rules.yaml
- /etc/falco/falco_rules.local.yaml
- /etc/falco/rules.d
stdout_output:
  enabled: true
syscall_event_drops:
  actions:
  - log
  - alert
  max_burst: 1
  rate: 0.03333
  threshold: 0.1
syscall_event_timeouts:
  max_consecutives: 1000
syslog_output:
  enabled: true
time_format_iso_8601: false
watch_config_files: true
webserver:
  enabled: true
  k8s_healthz_endpoint: /healthz
  listen_port: 8765
  ssl_certificate: /etc/falco/falco.pem
  ssl_enabled: false

BinaryData
====

Events:  <none>

[ajinkya1986]

Any update on this

[Andreagit97]

uhm here there are some strange things :thinking: It seems like you are running more than one Falco simultaneously, from your logs:

falco * Running falco-driver-loader for: falco version=0.32.0, driver version=39ae7d40496793cf3d3e7890c9bbdc202263836b
falco * Running falco-driver-loader with: driver=bpf, compile=yes, download=yes

and always in the same logs:

falco-driver-loader * Running falco-driver-loader for: falco version=0.32.2, driver version=2.0.0+driver
falco-driver-loader * Running falco-driver-loader with: driver=bpf, compile=yes, download=yes

Moreover, you are facing this error

falco Thu Apr 13 16:56:41 2023: Runtime error: Error reading config file(/etc/falco/falco.yaml): could not load plugins config: yaml-cpp: error at line 40, column 16: bad conversion. Exiting.

But the config seems right, so I wonder which Falco version you want to use... If drivers are available for your kernel I would try Falco 0.34 otherwise if you need to keep an older version I suggest you to try a clean deployment with just one Falco version and then to report here the Falco logs, something like:

Sat Apr  8 21:27:15 2023: Falco version: 0.34.1 (x86_64)
Sat Apr  8 21:27:15 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Sat Apr  8 21:27:15 2023: Loading rules from file /etc/falco/falco_rules.yaml
Sat Apr  8 21:27:15 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Sat Apr  8 21:27:15 2023: Starting health webserver with threadiness 2, listening on port 8765
Sat Apr  8 21:27:15 2023: Enabled event sources: syscall
Sat Apr  8 21:27:15 2023: Opening capture with Kernel module
...

[ajinkya1986]

Thank you @Andreagit97. We will check it with clean deployment

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[Andreagit97]

any news? can we close this?

[poiana]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

[poiana]

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community. /close

[poiana]

@poiana: Closing this issue.

In response to [this](https://github.com/falcosecurity/falco/issues/2492#issuecomment-1741794035): >Rotten issues close after 30d of inactivity. > >Reopen the issue with `/reopen`. > >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Provide feedback via https://github.com/falcosecurity/community. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.