cruise-automation / k-rail

Kubernetes security tool for policy enforcement
Apache License 2.0
444 stars 55 forks source link

Exemptions do not cover DaemonSets #123

Closed Kaezon closed 2 years ago

Kaezon commented 2 years ago

I've been trying to configure k-rail for a deployment which includes kube-prometheus-stack. I've gotten most of the components working with a few exemptions. Unfortunately, it seems that the exemption system does not apply to DaemonSet objects correctly, so my deployment cannot complete.

Kubernetes version: 1.19 k-rail version: v3.5.1

# Debug exemption for kube-prometheus-stack
- resource_name: "*"
  namespace: "prometheus"
  username: "*"
  group: "*"
  exempt_policies: ["*"]

DaemonSet manifest:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: prometheus-node-exporter
    app.kubernetes.io/instance: kube-prometheus-stack
    chart: prometheus-node-exporter-2.0.4
    heritage: Helm
    jobLabel: node-exporter
    release: kube-prometheus-stack
  name: kube-prometheus-stack-prometheus-node-exporter
  namespace: prometheus
spec:
  selector:
    matchLabels:
      app: prometheus-node-exporter
      release: kube-prometheus-stack
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: 'true'
      labels:
        app: prometheus-node-exporter
        chart: prometheus-node-exporter-2.0.4
        heritage: Helm
        jobLabel: node-exporter
        release: kube-prometheus-stack
    spec:
      automountServiceAccountToken: false
      containers:
        - args:
            - '--path.procfs=/host/proc'
            - '--path.sysfs=/host/sys'
            - '--path.rootfs=/host/root'
            - '--web.listen-address=$(HOST_IP):9100'
            - >-
              --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
            - >-
              --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
          env:
            - name: HOST_IP
              value: 0.0.0.0
          image: >-
            [REDACTED]/prometheus/node-exporter:v1.2.2@sha256:a990408ed288669bbad5b5b374fe1584e54825cde4a911c1a3d6301a907a030c
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet:
              path: /
              port: 9100
          name: node-exporter
          ports:
            - containerPort: 9100
              name: metrics
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /
              port: 9100
          resources: {}
          volumeMounts:
            - mountPath: /host/proc
              name: proc
              readOnly: true
            - mountPath: /host/sys
              name: sys
              readOnly: true
            - mountPath: /host/root
              mountPropagation: HostToContainer
              name: root
              readOnly: true
      hostNetwork: true
      hostPID: true
      securityContext:
        fsGroup: 65534
        runAsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccountName: kube-prometheus-stack-prometheus-node-exporter
      tolerations:
        - effect: NoSchedule
          operator: Exists
      volumes:
        - hostPath:
            path: /proc
          name: proc
        - hostPath:
            path: /sys
          name: sys
        - hostPath:
            path: /
          name: root
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate

And here is the error passed back from k-rail:

admission webhook "k-rail.cruise-automation.github.com" denied the request: DaemonSet kube-prometheus-stack-prometheus-node-exporter had violation: Host Bind Mounts: host bind mounts are forbidden DaemonSet kube-prometheus-stack-prometheus-node-exporter had violation: Host Bind Mounts: host bind mounts are forbidden DaemonSet kube-prometheus-stack-prometheus-node-exporter had violation: Host Bind Mounts: host bind mounts are forbidden DaemonSet kube-prometheus-stack-prometheus-node-exporter had violation: No Root user: Container node-exporter can run as the root user which is forbidden DaemonSet kube-prometheus-stack-prometheus-node-exporter had violation: No Root user: Container node-exporter can run as the root user which is forbidden DaemonSet kube-prometheus-stack-prometheus-node-exporter had violation: No Host Network: Using the host network is forbidden DaemonSet kube-prometheus-stack-prometheus-node-exporter had violation: No Host PID: Using the host PID namespace is forbidden
Kaezon commented 2 years ago

After further testing, I can't replicate this anymore. I'll close this issue for now.