search

falcosecurity / falco

Cloud Native Runtime Security
https://falco.org
Apache License 2.0
7.37k stars 901 forks source link

Falco keeps restarting when run with ebpf driver in least privilege mode #3106

Closed nnoromuche closed 1 month ago

nnoromuche commented 8 months ago

Describe the bug When I run falco as a demonset in least priviledge mode using the edpf driver on a red hat enterprise linux 8.9, I see multiple falco process restarts in the logs on a kubernetes environment. When I run falco with the edpf driver in privilege mode, it works fine, however running in least privilege results in multiple restarts.

How to reproduce it Add the following capabilities to the security context in the deployment.yaml file

securityContext: 
   capabilities: 
      add: 
      - SYS_ADMIN 
      - SYS_RESOURCE 
      - SYS-PTRACE

Logs shows

SUPERVISOR: 2024/02/20 16:43:12 Creating new supervisor for falco
SUPERVISOR: 2024/02/20 16:43:12 Creating new supervisor for falco-exporter
SUPERVISOR: 2024/02/20 16:43:12 Creating new supervisor for falcosidekick
SUPERVISOR: 2024/02/20 16:43:12 Starting reaper
SUPERVISOR: 2024/02/20 16:43:12 Running `/usr/bin/run.sh`
2024-02-20 16:43:12 INFO  Running falcoctl driver config
                      ├ name: falco
                      ├ version: 7.0.0+driver
                      ├ type: ebpf
                      ├ host-root: /host
                      └ repos: https://download.falco.org/driver
2024-02-20 16:43:12 WARN  Avoid updating Falco configuration
                      ├ config: /etc/falco/falco.yaml
                      └ reason: engine.kind is not driver driven: 
2024-02-20 16:43:12 INFO  Running falcoctl driver install
                      ├ driver version: 7.0.0+driver
                      ├ driver type: ebpf
                      ├ driver name: falco
                      ├ compile: true
                      ├ download: true
                      ├ arch: x86_64
                      ├ kernel release: 4.18.0-513.11.1.el8_9.x86_64
                      └ kernel version: #1 SMP Thu Dec 7 03:06:13 EST 2023
2024-02-20 16:43:12 INFO  Found distro target: rhel
2024-02-20 16:43:12 INFO  Removing eBPF probe symlink
                      └ path: /root/.falco/falco-bpf.o
2024-02-20 16:43:12 INFO  Trying to download a driver.
                      └ url: https://download.falco.org/driver/7.0.0%2Bdriver/x86_64/falco_rhel_4.18.0-513.11.1.el8_9.x86_64_1.o
2024-02-20 16:43:13 WARN  Non-200 response from url. code: 404
2024-02-20 16:43:13 WARN  unable to find a prebuilt driver 
2024-02-20 16:43:13 INFO  Mounting debugfs for bpf driver. 
2024-02-20 16:43:30 INFO  Copying built driver to its destination.
                      ├ src: /usr/src/falco-7.0.0+driver/bpf/probe.o
                      └ dst: /root/.falco/7.0.0+driver/x86_64/falco_rhel_4.18.0-513.11.1.el8_9.x86_64_1.o
2024-02-20 16:43:30 INFO  Driver built.
                      └ path: /root/.falco/7.0.0+driver/x86_64/falco_rhel_4.18.0-513.11.1.el8_9.x86_64_1.o
2024-02-20 16:43:30 INFO  Symlinking eBPF probe
                      ├ src: /root/.falco/7.0.0+driver/x86_64/falco_rhel_4.18.0-513.11.1.el8_9.x86_64_1.o
                      └ dest: /root/.falco/falco-bpf.o
2024-02-20 16:43:30 INFO  eBPF probe symlinked 
SUPERVISOR: 2024/02/20 16:43:30 Completed `/usr/bin/run.sh`
SUPERVISOR: 2024/02/20 16:43:30 Starting falco
SUPERVISOR: 2024/02/20 16:43:30 falco started successfully
SUPERVISOR: 2024/02/20 16:43:30 Starting falco-exporter
SUPERVISOR: 2024/02/20 16:43:30 falco-exporter started successfully
SUPERVISOR: 2024/02/20 16:43:30 Starting falcosidekick
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:43:31 falcosidekick started successfully
SUPERVISOR: 2024/02/20 16:43:41 falco process has stopped, restarting.
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:43:51 falco process has stopped, restarting.
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:44:01 falco process has stopped, restarting.
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:44:11 falco process has stopped, restarting.
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:44:21 falco process has stopped, restarting.

Expected behaviour

I will like falco to run in least priviledge mode on a red hat enterprise linux 8.9 machine without multiple restarts.

Environment

rashim27us commented 8 months ago

@Nnoromuche There are some possible reasons for this issue, in my case when I was running Falco in the least privileged mode it didn't have the necessary permissions to load the eBPF driver. If we use Pod Security policies in the Kubernetes cluster, they might restrict the capabilities of the Falco pods, preventing them from loading the eBPF driver.

Here are the key permissions required to run Falco with eBPF driver:

  1. SYS_ADMIN_Capability
  2. Host PID Namespace
  3. Host Networking
  4. Volume Mounts

You can check whether these necessary permissions are given or not in Kubernetes least privileged mode.

nnoromuche commented 6 months ago

@rashim27us thanks, I am running falco with the below deployment.yaml file and the SYS_ADMIN_Capability is being set. 

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: test-falco
  namespace: test
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 10%
  selector:
    matchLabels:
      app: test-falco
  template:
    metadata:
      name: test-falco
      namespace: test
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "12076"
        prometheus.istio.io/merge-metrics: "false"
        sidecar.istio.io/inject: "false"
      labels:
        app: test-falco
        addonmanager.kubernetes.io/mode: Reconcile
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      tolerations:
      - operator: Exists
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
      - name: containerd-socket
        hostPath:
          path: /run/containerd/containerd.sock
      - name: proc-fs
        hostPath:
          path: /proc
      - name: boot-fs
        hostPath:
          path: /boot
      - name: lib-modules
        hostPath:
          path: /lib/modules
      - name: usr-fs
        hostPath:
          path: /usr
      - name: etc-fs
        hostPath:
          path: /etc
      - name: root-falco-fs
        emptyDir: {}
      containers:
      - name: test-falco
        image: (( concat "{{ FALCO_IMAGE_REPO_HOSTNAME}}/test/test-falco:" metadata.annotations.version ))
        env:
        - name: HOST_ROOT
          value: "/host"
        securityContext:
          capabilities:
            drop:
            - all
            add:
            - SYS_ADMIN
            - SYS_RESOURCE
            - SYS_PTRACE
        livenessProbe:
          httpGet:
            path: /liveness
            port: 12077
          periodSeconds: 60
          failureThreshold: 10
        readinessProbe:
          httpGet:
            path: /readiness
            port: 12077
          periodSeconds: 60
          failureThreshold: 10
        volumeMounts:
        - mountPath: /host/run/containerd/containerd.sock
          name: containerd-socket
        - mountPath: /host/proc
          name: proc-fs
          readOnly: true
        - mountPath: /host/boot
          name: boot-fs
          readOnly: true
        - mountPath: /host/lib/modules
          name: lib-modules
          readOnly: false
        - mountPath: /host/usr
          name: usr-fs
          readOnly: true
        - mountPath: /dev/shm
          name: dshm
        - mountPath: /host/etc
          name: etc-fs
          readOnly: true
        - mountPath: /root/.falco
          name: root-falco-fs
        resources:
          requests:
            cpu: 25m
            memory: 230Mi
          limits:
            memory: 400Mi

The edpf driver is installed using falcoctl commands 

/usr/bin/falcoctl driver config --type ebpf
/usr/bin/falcoctl driver install --compile --download

Wondering is there a missing configuration in the above deployment.yaml file?

poiana commented 3 months ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

poiana commented 2 months ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

poiana commented 1 month ago

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community. /close

poiana commented 1 month ago

@poiana: Closing this issue.

In response to [this](https://github.com/falcosecurity/falco/issues/3106#issuecomment-2380591377): >Rotten issues close after 30d of inactivity. > >Reopen the issue with `/reopen`. > >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Provide feedback via https://github.com/falcosecurity/community. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.