securing KubeArmor pods and policies

[nyrahul]

KubeArmor is a security engine and thus it is imperative that it follows all the security best practices. The aim is to ensure security of the KubeArmor itself. Much of the work towards following best security practices (such as not using privileged flag) is already followed. It is now time to notch up the security practices to next level. The proposition is as follows:

• [ ] Use baseline level in enforce mode Pod Security Admission for KubeArmor namespace
• [ ] Dogfooding
  • [ ] Protect AppArmor policies mount point to be accessible only by KubeArmor binary. Enforce a KubeArmor policy to ensure this.
  • [ ] Only allow access of service account token for kubearmor binary in the KubeArmor pod.
  • [ ] Use hardening policies for KubeArmor pod.
• [x] Remove use of cluster-admin role
• [ ] Satisfy SLSA 3 requirements #1164
• [ ] Ability to use signed policy YAMLs only
• [x] seccomp profile for kubearmor
• [ ] selinux/apparmor ootb profile
• [x] Fix all critical vulnerabilities across all kubearmor-images.
• [x] TLS for all intra-kubearmor communication #1526
• [x] Reducing the kubearmor base image (can we try distroless kubearmor image?) [moved to UBI]
• [x] Remove unnecessary hostPaths from the manifests
• [x] Remove unnecessary capabilities from the manifests
• [ ] attacker can disable kprobes, thus we won't get any future events Predefined hardening policy for checking any changes to /sys/kernel/debug/kprobes/enabled
• [ ] Optimize RBAC permissions (implementation detail - config will be handled through Helm values of kubearmor-operator)
  • [ ] Check if update perms can be removed for deployments/statefulsets etc - for AppArmor, patch should be suffice
  • [ ] BPF LSM doesn't need the perms to annotate the above, set RBAC accordingly. (check kubearmor-policy annotation once though)

[Ankurk99]

karmor recommend for KubeArmor Daemonset

[daemon1024]

• How would KubeArmor daemonset attach a profile to itself?
  • Failed admission if annotation added and no profile found

Potential Solution

• Snitch has the intelligence to load the default apparmor profile.

[DelusionalOptimist]

For protecting KubeArmor with KubeArmor in BPF LSM, we need to:

• How to differentiate between host process and containers with hostPid.

[daemon1024]

• Seccomp Loading
• Try running all other KA Components with Seccomp RuntimeDefault

[daemon1024]

• Remove skipping kubearmor-app logic from KubeArmor.
• Have snitch generate apparmor profile just for kubearmor daemonset with a privileged default profile
• annotate kubearmor daemonset if snitch successful
• verify if matchlabels works as expected with kubearmor pods as well

Edge Case

• kubearmore-controller should not block daemonset deployment or try to admission control daemonset (it can do it for other kubearmor pods except snitch)
• if operator does not exist, verify what happens when kubearmor tries to patch it's own daemonset
  • validate if kubearmor loads apparmor profile before patching

[dirsigler]

Regarding "Use baseline level in enforce mode Pod Security Admission for KubeArmor namespace", how about providing a Namespace manifest within the Helm chart which already provide the specific labels to set the Pod Security Standards on it.

This way users could for example deploy the Chart via Helm and use their own Namespace via --namespace --create-namespace or implement the provided namespace with all the features out of the box.

[DelusionalOptimist]

Prioritize

• Try using different manifests for AppArmor and BPF LSM
• Remove hostPath volumes
• Remove unnecessary capabilities from the manifests
• Have an FAQ on additional privilleges/host path volumes required if we can't get around them
• Optimize for best security for BPF LSM first
• Enable seccomp by default

Future

• Try using different images for AppArmor and BPF LSM
  • Use UBI image if BPF-LSM is present? (kubearmor-ubi)
    • Can't do - this doesn't work with containerd
    • Advantages - non root user not required
  • Use a more minimal image which doesn't ship with apparmor_parser or any other caps required by apparmor

[DelusionalOptimist]

Host paths

KubeArmor

• Remove:

  • /sys/fs/bpf - not needed, KubeArmor can create and mount bpffs within the container, persistence of maps not an issue

  • /sys/kernel/debug - was needed earlier for bpf_printk logs, not needed anymore

  • Init container mounts: if BTF present, compilation not needed - remove the init container entirely along with below volumes. Also document when would this be needed. (Checkout PR #1658)

    • /etc/os-release
    • /lib/modules
    • /usr/src

• Document:

  • Container runtime sockets (read-only) - needed for interacting with container runtimes, would be removed soon as we adopt OCI hooks (PR in review)
  • /etc/apparmor.d (read/write) - needed for writing apparmor profiles. Only mounted if AppArmor enforcer is being used.

Snitch

• Remove:

  • / - snitch mounts entire root, though read-only but it is not good. Mount specific paths. Also, document each of the path that snitch mounts for detection of the env, if we cannot remove them.

• Document:

  • /var/lib/kubelet/seccomp - (read/write) needed for storing seccomp profiles - if not desired inform about the option to disable seccomp.

Controller

• Remove:
  • /sys/kernel/security - to check if AppArmor enforcer. Handle this through snitch and remove relevant code in controller. Label/annotate the deployment and accordingly skip adding this volume if it is not AppArmor. (Checkout PR: https://github.com/kubearmor/KubeArmor/pull/1497)

[DelusionalOptimist]

Capabilities

KubeArmor/InitContainer

• Remove:

  • SYS_RESOURCE - for managing memory. Needed only on kernel < 5.11 by cilium/ebpf (ref). Modify snitch to handle this.

• Check:

  • SYS_ADMIN - Our usecases should ideally be satisfied with CAP_BPF itself however SYS_ADMIN is still kept
    • In order to be compatible with Kernels older than 5.8.
    • Certain problems encountered in loading eBPF code with CAP_BPF (we'll try to see if we can get around these anyhow)
  • SETUID - trace and check
  • SETGID - trace and check
  • IPC_LOCK - for locking memory using mmap and other related syscalls as mentioned in man page. (Ref)

• Document:

  • SETPCAP - For AppArmor subprofiles
  • SYS_PTRACE - for reading processes other than self from procfs (source: man capabilities)
  • MAC_ADMIN - for managing LSM rules (man capabilities and ref)
  • CAP_DAC_OVERRIDE, CAP_DAC_READ_SEARCH - mainly needed by cilium/ebpf (ref). Also possible that these are needed for managing apparmor profiles at /etc/apparmor.d

Snitch

• Check and document:
  • IPC_LOCK
  • SYS_ADMIN
  • SYS_RESOURCE

[DelusionalOptimist]

Regarding "Use baseline level in enforce mode Pod Security Admission for KubeArmor namespace", how about providing a Namespace manifest within the Helm chart which already provide the specific labels to set the Pod Security Standards on it.

This way users could for example deploy the Chart via Helm and use their own Namespace via --namespace --create-namespace or implement the provided namespace with all the features out of the box.

Yep! This seems fairly straight and user friendly. Thanks for the suggestion @dirsigler. Also, no one is looking into this aspect right now so PRs welcome! : )

[DelusionalOptimist]

Signing container images SBOM PR - https://github.com/kubearmor/KubeArmor/pull/1735