[stable/falco] Rules volume is trying to mount in config volume

[kody-abe]

Is this a request for help?: No

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report

Which chart: stable/falco

What happened: Adding a custom rules configmap does not mount on falco pods

What you expected to happen: The configmaps should mount on the rules volume on falco pods

How to reproduce it (as minimally and precisely as possible): Follow https://github.com/helm/charts/tree/master/stable/falco#loading-custom-rules and attempt to setup the custom rules

Anything else we need to know: It seems this might be a result of Volumes can not mount onto other volumes or have hard links to other volumes https://kubernetes.io/docs/concepts/storage/volumes/

{
  "name": "config-volume",
  "mountPath": "/etc/falco"
},
{
  "name": "rules-volume",
  "mountPath": "/etc/falco/rules.d"
}

[kody-abe]

ping @nestorsalceda

[nestorsalceda]

Hey! I'm trying to reproduce the error, but I can't. So perhaps there are some detail that I' m missing. Could you paste the content of your custom-rules.yaml file?

Thanks!

[kody-abe]

@nestorsalceda Thanks for helping on this. Here is that file. I am using the script that merges the multiple files into 1. (https://github.com/draios/falco-extras/blob/master/scripts/rules2helm)

customRules:
  kube-scheduler.yaml: |-
    #
    # Auto-generated set of falco rules for kube_scheduler containers
    # Generated at 2018-05-23 09:58:49 UTC
    #

    # By default, the autogenerated rules include rules that attempt to
    # restrict the set of system calls that can be performed by
    # applications. However, we know that these rules are fairly FP-prone,
    # so they are disabled by default. If you'd like to enable them,
    # either change or override this macro's condition to "evt.num >= 0".
    - macro: kube_scheduler_consider_syscalls
      condition: (evt.num < 0)

    # These policies are limited to containers, specifically those where
    # the container image name starts with "kube_scheduler"
    - macro: app_kube_scheduler
      condition: container and container.image contains "kube-scheduler"

    # Not considering all inbound networking suspect

    # Not considering all outbound networking suspect

    # Restricting listening ports to selected set

    - list: kube_scheduler_allowed_inbound_ports_tcp
      items: [10251]

    - rule: Unexpected inbound tcp connection kube_scheduler
      desc: Detect inbound traffic to kube_scheduler using tcp on a port outside of expected set
      condition: inbound and evt.rawres >= 0 and not fd.sport in (kube_scheduler_allowed_inbound_ports_tcp) and app_kube_scheduler
      output: Inbound network connection to kube_scheduler on unexpected port (command=%proc.cmdline pid=%proc.pid connection=%fd.name sport=%fd.sport user=%user.name %container.info image=%container.image)
      priority: NOTICE

    # Restricting spawned processes to selected set

    - list: kube_scheduler_allowed_processes
      items: ["kube-scheduler"]

    - rule: Unexpected spawned process kube_scheduler
      desc: Detect a process started in a kube_scheduler container outside of an expected set
      condition: spawned_process and not proc.name in (kube_scheduler_allowed_processes) and app_kube_scheduler
      output: Unexpected process spawned in kube_scheduler container (command=%proc.cmdline pid=%proc.pid user=%user.name %container.info image=%container.image)
      priority: NOTICE

    # Restricting files read or written to specific set

    # Restricting syscalls to selected set

    - list: kube_scheduler_allowed_syscalls
      items: [accept]

    - rule: Unexpected syscall kube_scheduler
      desc: Detect a syscall in a kube_scheduler container outside of an expected set
      condition: kube_scheduler_consider_syscalls and not evt.type in ("<unknown>", kube_scheduler_allowed_syscalls) and app_kube_scheduler
      output: Unexpected syscall in kube_scheduler container (command=%proc.cmdline pid=%proc.pid user=%user.name syscall=%evt.type args=%evt.args %container.info image=%container.image)
      priority: NOTICE

Currently only using 1 file just to verify and get it working

[nestorsalceda]

Yeah! I think I got something.

I install Falco using the custom-rule you provided me with the following command. And I get the following output.

$ helm install --name sysdig-falco-1 -f custom-rules.yaml stable/falco
NAME:   sysdig-falco-1
LAST DEPLOYED: Mon Jul 23 18:47:42 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                  DATA  AGE
sysdig-falco-1-rules  1     1s
sysdig-falco-1        3     1s

==> v1/ServiceAccount
NAME            SECRETS  AGE
sysdig-falco-1  1        1s

==> v1beta1/ClusterRole
NAME            AGE
sysdig-falco-1  1s

==> v1beta1/ClusterRoleBinding
NAME            AGE
sysdig-falco-1  1s

==> v1beta1/DaemonSet
NAME            DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
sysdig-falco-1  4        4        0      4           0          <none>         1s

==> v1/Pod(related)
NAME                  READY  STATUS             RESTARTS  AGE
sysdig-falco-1-5d7vc  0/1    ContainerCreating  0         1s
sysdig-falco-1-89t2z  0/1    ContainerCreating  0         0s
sysdig-falco-1-9b8r4  0/1    ContainerCreating  0         1s
sysdig-falco-1-wxrz9  0/1    ContainerCreating  0         1s

NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.

No further action should be required.

And my pods are up and running:

$ kubectl get pod
NAME                   READY     STATUS    RESTARTS   AGE
sysdig-falco-1-5d7vc   1/1       Running   0          3m
sysdig-falco-1-89t2z   1/1       Running   0          3m
sysdig-falco-1-9b8r4   1/1       Running   0          3m
sysdig-falco-1-wxrz9   1/1       Running   0          3m

So I'm going to take a look to the logs:

$ kubectl logs sysdig-falco-1-5d7vc 
* Setting up /usr/src links from host
* Unloading falco-probe, if present
* Running dkms install for falco

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area...
make -j1 KERNELRELEASE=4.13.0-1015-gcp -C /lib/modules/4.13.0-1015-gcp/build M=/var/lib/dkms/falco/0.10.0/build....
cleaning build area...

DKMS: build completed.

falco-probe.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/4.13.0-1015-gcp/kernel/extra/
mkdir: cannot create directory '/lib/modules/4.13.0-1015-gcp/kernel/extra': Read-only file system
cp: cannot create regular file '/lib/modules/4.13.0-1015-gcp/kernel/extra/falco-probe.ko': No such file or directory

depmod...

DKMS: install completed.
* Trying to load a dkms falco-probe, if present
falco-probe found and loaded in dkms
Mon Jul 23 16:47:51 2018: Falco initialized with configuration file /etc/falco/falco.yaml
Mon Jul 23 16:47:51 2018: Loading rules from file /etc/falco/falco_rules.yaml:
Mon Jul 23 16:47:51 2018: Loading rules from file /etc/falco/falco_rules.local.yaml:
Mon Jul 23 16:47:51 2018: Loading rules from file /etc/falco/rules.d/kube-scheduler.yaml:
Rule Unexpected syscall kube_scheduler: warning (trailing-evttype):
kube_scheduler_consider_syscalls and not evt.type in ("<unknown>", accept) and app_kube_scheduler
         does not have all evt.type restrictions at the beginning of the condition,
         or uses a negative match (i.e. "not"/"!=") for some evt.type restriction.
         This has a performance penalty, as the rule can not be limited to specific event types.
         Consider moving all evt.type restrictions to the beginning of the rule and/or
         replacing negative matches with positive matches if possible.

Then I think the problem is some kind of confusion between a warning and an error loading the rule. The rule is loaded, but we receive a warning because we are considering syscalls in kube_scheduler before checking the event type and this could lead to some performance issues. So I'm going to forward your issue to my coworkers and let' s try to remove the warning.

Are you receiving an output from logs like this one?

Thanks again!

[nestorsalceda]

/assign @nestorsalceda

[kody-abe]

@nestorsalceda Nope, I don't get that - This is all I get:

* Setting up /usr/src links from host
* Unloading falco-probe, if present
* Running dkms install for falco
 Kernel preparation unnecessary for this kernel.  Skipping...
 Building module:
cleaning build area...
make -j2 KERNELRELEASE=4.4.78-k8s -C /lib/modules/4.4.78-k8s/build M=/var/lib/dkms/falco/0.10.0/build...
cleaning build area...
 DKMS: build completed.
 falco-probe.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/4.4.78-k8s/kernel/extra/
mkdir: cannot create directory '/lib/modules/4.4.78-k8s/kernel/extra': Read-only file system
cp: cannot create regular file '/lib/modules/4.4.78-k8s/kernel/extra/falco-probe.ko': No such file or directory
 depmod...
 DKMS: install completed.
* Trying to load a dkms falco-probe, if present
falco-probe found and loaded in dkms
Fri Jul 13 03:41:22 2018: Falco initialized with configuration file /etc/falco/falco.yaml
Fri Jul 13 03:41:22 2018: Loading rules from file /etc/falco/falco_rules.yaml:
Fri Jul 13 03:41:22 2018: Loading rules from file /etc/falco/falco_rules.local.yaml:

The real problem here is that the mount is not even mounting:

root@falco-10mtd:/# ls -l /etc/falco/   
..7987_13_07_03_41_19.697621788/ falco.yaml                       falco_rules.yaml                 
..data/                          falco_rules.local.yaml           
root@falco-10mtd:/# ls -l /etc/falco/rules.d
ls: cannot access '/etc/falco/rules.d': No such file or directory

[nestorsalceda]

Hey!I would like to see the command you typed for installing Falco using the chart. I'm wondering if the right parameters are passed to helm install

Thanks!

[kody-abe]

@nestorsalceda Here is the command:

helm install stable/falco --name falco --namespace falco -f generated/falco_rules.yaml -f generated/falco_values.yaml --kube-context <redacted> --version 0.1.1

falco_values.yaml:

  rbac:
    create: false
  falco:
    jsonOutput: true
    programOutput:
      enabled: true
      program: "\"jq '.output as $orig | $orig | split(\\\": \\\") | .[1] | split(\\\" (user=\\\") | .[0] as $error | $orig | split(\\\"container=\\\") | .[1] | split(\\\" \\\") | .[0] as $container | $orig | split(\\\"image=\\\") | .[1] | split(\\\" \\\") | .[0] as $image | {\\\"attachments\\\": [{\\\"pretext\\\": $error, \\\"fields\\\": [{\\\"title\\\": \\\"Container\\\", \\\"value\\\": $container, \\\"short\\\": true}, {\\\"title\\\": \\\"Image\\\", \\\"value\\\": $image, \\\"short\\\": true}, {\\\"title\\\": \\\"Full Content\\\", \\\"value\\\": $orig}]}]}' | curl -d @- -X POST https://hooks.slack.com/services/<REDACTED>\""

faclo_rules.yaml:

customRules:
  kube-scheduler.yaml: |-
    #
    # Auto-generated set of falco rules for kube_scheduler containers
    # Generated at 2018-05-23 09:58:49 UTC
    #

    # By default, the autogenerated rules include rules that attempt to
    # restrict the set of system calls that can be performed by
    # applications. However, we know that these rules are fairly FP-prone,
    # so they are disabled by default. If you'd like to enable them,
    # either change or override this macro's condition to "evt.num >= 0".
    - macro: kube_scheduler_consider_syscalls
      condition: (evt.num < 0)

    # These policies are limited to containers, specifically those where
    # the container image name starts with "kube_scheduler"
    - macro: app_kube_scheduler
      condition: container and container.image contains "kube-scheduler"

    # Not considering all inbound networking suspect

    # Not considering all outbound networking suspect

    # Restricting listening ports to selected set

    - list: kube_scheduler_allowed_inbound_ports_tcp
      items: [10251]

    - rule: Unexpected inbound tcp connection kube_scheduler
      desc: Detect inbound traffic to kube_scheduler using tcp on a port outside of expected set
      condition: inbound and evt.rawres >= 0 and not fd.sport in (kube_scheduler_allowed_inbound_ports_tcp) and app_kube_scheduler
      output: Inbound network connection to kube_scheduler on unexpected port (command=%proc.cmdline pid=%proc.pid connection=%fd.name sport=%fd.sport user=%user.name %container.info image=%container.image)
      priority: NOTICE

    # Restricting spawned processes to selected set

    - list: kube_scheduler_allowed_processes
      items: ["kube-scheduler"]

    - rule: Unexpected spawned process kube_scheduler
      desc: Detect a process started in a kube_scheduler container outside of an expected set
      condition: spawned_process and not proc.name in (kube_scheduler_allowed_processes) and app_kube_scheduler
      output: Unexpected process spawned in kube_scheduler container (command=%proc.cmdline pid=%proc.pid user=%user.name %container.info image=%container.image)
      priority: NOTICE

    # Restricting files read or written to specific set

    # Restricting syscalls to selected set

    - list: kube_scheduler_allowed_syscalls
      items: [accept]

    - rule: Unexpected syscall kube_scheduler
      desc: Detect a syscall in a kube_scheduler container outside of an expected set
      condition: kube_scheduler_consider_syscalls and not evt.type in ("<unknown>", kube_scheduler_allowed_syscalls) and app_kube_scheduler
      output: Unexpected syscall in kube_scheduler container (command=%proc.cmdline pid=%proc.pid user=%user.name syscall=%evt.type args=%evt.args %container.info image=%container.image)
      priority: NOTICE

[nestorsalceda]

I tried with your settings and your command and Falco is loading the rule successfully. So we are going to check other components and try to find the difference.

My Helm version is:

$ helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

And my Kubernetes version is:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.3", GitCommit:"9b5b719c5f295c99de68ffb5b63101b0e0175376", GitTreeState:"clean", BuildDate:"2018-05-31T18:32:23Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Thanks!

[kody-abe]

@nestorsalceda My Helm version is:

⚡ helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

And my Kubernetes version is:

⚡ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-13T22:27:55Z", GoVersion:"go1.9.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"922a86cfcd65915a9b2f69f3f193b8907d741d9c", GitTreeState:"clean", BuildDate:"2017-07-21T08:08:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

[nestorsalceda]

Hey @kody-abe it looks like is an older kubernetes server version, so I'm going to try to reproduce with that settings.

Thanks again!

[kody-abe]

@nestorsalceda We upgraded to the latest version of Kubernetes and see that the issue does not occur there. So, this is specifically an issue with an older version. This no longer affects us, but should still probably be solved for. I leave that up to you.

[nestorsalceda]

Ah! It's nice to know this.

I think we would add a note in the README saying that we support the latest three versions of Kubernetes, following the Kubernetes conventions

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew.

Thanks for the feedback Kody!

[nestorsalceda]

/close

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.