False Fails on etcd pod specification files permissions

[papiveron]

Hi,

I'm testing my k8s kubespray cluster hardening using kube-bench.

I'm getting fails that doesn't correspond to the real cluster state. Mainly I've set etcd pod specifications and data dir as recommended by CIS, but the scan sill report the issues as you can see below.

[centos@infra-vm kube-bench]$ kubectl logs kube-bench-master-h9fh6 | grep  FAIL
[FAIL] 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated)
[FAIL] 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated)
[FAIL] 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated)
[FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)
4 checks FAIL
4 checks FAIL
[centos@infra-vm ~]$

While permissions are well set, even in the container :

[centos@cpu-node0 ~]$ ps -ef | grep etcd
root      1974     1  0 14:10 ?        00:00:00 /bin/bash /usr/local/bin/etcd
root      1975  1974  0 14:10 ?        00:00:00 /usr/bin/docker run --restart=on-failure:5 --env-file=/etc/etcd.env --net=host -v /etc/ssl/certs:/etc/ssl/certs:ro -v /etc/ssl/etcd/ssl:/etc/ssl/etcd/ssl:ro -v /var/lib/etcd:/var/lib/etcd:rw --memory=0 --blkio-weight=1000 --name=etcd1 quay.io/coreos/etcd:v3.3.12 /usr/local/bin/etcd
root      2152  2104  3 14:10 ?        00:00:31 /usr/local/bin/etcd
root      9686  9633  6 14:10 ?        00:00:54 kube-apiserver --advertise-address=192.168.33.143 --allow-privileged=true --anonymous-auth=True --apiserver-count=1 --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/log/apiserver/audit.log --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --client-ca-file=/etc/kubernetes/ssl/ca.crt --cors-allowed-origins=.* --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,PodPreset,AlwaysPullImages,DenyEscalatingExec --enable-aggregator-routing=False --enable-bootstrap-token-auth=true --endpoint-reconciler-type=lease --etcd-cafile=/etc/ssl/etcd/ssl/ca.pem --etcd-certfile=/etc/ssl/etcd/ssl/node-cpu-node0.pem --etcd-keyfile=/etc/ssl/etcd/ssl/node-cpu-node0-key.pem --etcd-servers=https://192.168.33.143:2379 --insecure-port=0 --kubelet-certificate-authority=/etc/kubernetes/ssl/ca.crt --kubelet-client-certificate=/etc/kubernetes/ssl/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/ssl/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP --profiling=False --proxy-client-cert-file=/etc/kubernetes/ssl/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/ssl/front-proxy-client.key --request-timeout=1m0s --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --runtime-config= --secure-port=6443 --service-account-key-file=/etc/kubernetes/ssl/sa.pub --service-account-lookup=True --service-cluster-ip-range=10.233.0.0/18 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/etc/kubernetes/ssl/apiserver.crt --tls-private-key-file=/etc/kubernetes/ssl/apiserver.key
centos   20812 19018  0 14:24 pts/0    00:00:00 grep --color=auto etcd
[centos@cpu-node0 ~]$ stat -c %a /var/lib/etcd
700
[centos@cpu-node0 ~]$ stat -c %a /etc/kubernetes/manifests/
644
[centos@cpu-node0 ~]$ ls -ld  /var/lib/etcd
drwx------. 3 etcd etcd 20 Mar  4 14:10 /var/lib/etcd
[centos@cpu-node0 ~]$ ls -ld  /etc/kubernetes/manifests/
drw-r--r--. 2 root root 96 Mar  4 14:09 /etc/kubernetes/manifests/
[centos@cpu-node0 ~]$

Any ideal of why the bench scan behave like this and report false issues?

Thanks.

Regards, Eugène NG

[mozillazg]

@papiveron

The debug logs will give you the reason:

Running kube-bench with the -v 3 parameter will generate debug logs that can be very helpful for debugging problems.

https://github.com/aquasecurity/kube-bench#troubleshooting

See also about 1.1.12: https://github.com/aquasecurity/kube-bench/issues/842

[yoavrotems]

Thanks @mozillazg ! @papiveron is this still an issue?

[papiveron]

Yes @mozillazg, please I'll let you know once I find the root cause or solve it.

Thanks

[yoavrotems]

@papiveron Hey anything new?

[papiveron]

Hi @yoavrotems ,

Nothing new for the moment.

I think we can close the ticket for now and reopen it later if necessary.

Best regards, Eugène NG