questions on k3s hardening

[cwrx777]

Related issue: Previous conversation in k3s-io/k3s#10458

Hi @dereknola Thank you for answering my questions. I have additoinal questions.

Yes, this check is outdated, as the rotate feature gate has been "on by default" since Kubernetes 1.11. This check is basically just "did you do something stupid and turn off the security feature".

how do I audit that RotateKubeletServerCertificate feature is enabled?

Additional questions:

• 1.1.13 why is this Not Applicable ? isn't it just changing the file permission to 600?

My worker node is run using the following config.yaml:

...
protect-kernel-defaults: true
kubelet-arg:
  - 'streaming-connection-idle-timeout=5m'
  - 'make-iptables-util-chains=true'

k3s-agent.service:

ExecStart=/usr/local/bin/k3s \
    agent \
        '--config=/test/k3s/config/config.yaml' \

• 4.1.9, 4.1.10 Does kubelet --config configuration file refer to kubelet.kubeconfig?
• 4.2.8, 4.2.10, 4.2.11 I am not able to verify the required parameters from the ps output.

  /bin/ps -fC containerd
  UID          PID    PPID  C STIME TTY          TIME CMD
  root     3002164 3001447  0 Jul08 ?        00:06:39 containerd

• 4.2.12 Does the kubelet config refer to /var/lib/rancher/k3s/agent/kubelet.kubeconfig kubelet.kubeconfig:

  apiVersion: v1
  clusters:
  - cluster:
    server: https://127.0.0.1:6444
    certificate-authority: /var/lib/rancher/k3s/agent/server-ca.crt
  name: local
  contexts:
  - context:
    cluster: local
    namespace: default
    user: user
  name: Default
  current-context: Default
  kind: Config
  preferences: {}
  users:
  - name: user
  user:
    client-certificate: /var/lib/rancher/k3s/agent/client-kubelet.crt
    client-key: /var/lib/rancher/k3s/agent/client-kubelet.key

• 4.2.1.3 Where is pod-pids supposed to be set? I dont see that this is set in the kubelet CLI arguments.

  Running kubelet 
  --address=0.0.0.0 
  --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding 
  --anonymous-auth=false 
  --authentication-token-webhook=true 
  --authorization-mode=Webhook 
  --cgroup-driver=systemd 
  --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt 
  --cloud-provider=external 
  --cluster-dns=10.43.0.10 
  --cluster-domain=cluster.local 
  --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock 
  --containerd=/run/k3s/containerd/containerd.sock 
  --eviction-hard=imagefs.available<5%,nodefs.available<5% 
  --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% 
  --fail-swap-on=false 
  --feature-gates=CloudDualStackNodeIPs=true 
  --healthz-bind-address=127.0.0.1 
  --hostname-override=skhromivpapp03.shses.shs.com.sg 
  --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig 
  --make-iptables-util-chains=true 
  --node-ip=10.236.49.161 
  --node-labels=node-type=worker 
  --pod-infra-container-image=rancher/mirrored-pause:3.6 
  --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests 
  --protect-kernel-defaults=true 
  --read-only-port=0 
  --resolv-conf=/etc/resolv.conf 
  --serialize-image-pulls=false 
  --streaming-connection-idle-timeout=5m 
  --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt 
  --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key

[dereknola]

Audit RotateKubelet feature) Look at the upstream docs, once we pass it off to Kubelet we have no control over it. https://kubernetes.io/docs/tasks/tls/certificate-rotation/

1.1.13) This is an error, it seems to be a legacy bug in the CIS scans for K3s when they originally were copied over from RKE1. That test is applicable and we already pass it. I will update the docs and scans. Here is the current results on v1.28.11+k3s1

root@server-0:/home/vagrant# /bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi'
permissions=600

4.1.9 / 4.1.10) You should look at the audit section of the docs pages, that shows you the actual command being run on the system to check for each of these tests. In this case it is /var/lib/rancher/k3s/agent/kubelet.kubeconfig

4.2.5-4.2.12) Something appears wrong with the audit and all theses test in the docs. There are discrepancies between cis1.8 and cis-1.7 that are incorrect. I know that we some pass these, I will go back and edit and ensure that the cis scans and docs are correctly displaying the results.

4.2.13) This is a manual change you will have to perform. What limit you set is your choice. You can set this in the K3s config file with:

#/etc/rancher/k3s/config.yaml
kubelet-arg:
- "pod-max-pids=XXX"