request for clarification on security best-practices for cluster creation

[scottyhq]

What help do you need? As someone relatively inexperienced with network security, but wanting to experiment with kubernetes clusters, it would be helpful to: 1) clarify the default configuration of clusters created by eksctl. Or 2) point users to best-practices for secure configurations.

I've noticed a few things that are perhaps intentional, but inconsistent with the aws eks documentation. For example,

"We recommend a network architecture that uses private subnets for your worker nodes, and public subnets for Kubernetes to create public load balancers within." https://docs.aws.amazon.com/eks/latest/userguide/create-public-private-vpc.html

However, eksctl create cluster creates worker nodes on public subnets.

Also, in looking over this helpful documentation on security best-practices (https://github.com/freach/kubernetes-security-best-practice). I decided to run the kube-bench security benchmark tool tool on a default cluster and notice several failed tests https://github.com/aquasecurity/kube-bench

git clone https://github.com/aquasecurity/kube-bench
# edit job-eks.yaml to point to latest dockerhub image
kubectl apply -f kube-bench/job-eks.yaml
kubectl logs kube-bench-xxxx

== Summary ==
17 checks PASS
6 checks FAIL
2 checks WARN
0 checks INFO

click for full kube-bench log

[INFO] 2 Worker Node Security Configuration [INFO] 2.1 Kubelet [FAIL] 2.1.1 Ensure that the --allow-privileged argument is set to false (Scored) [PASS] 2.1.2 Ensure that the --anonymous-auth argument is set to false (Scored) [PASS] 2.1.3 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored) [PASS] 2.1.4 Ensure that the --client-ca-file argument is set as appropriate (Scored) [FAIL] 2.1.5 Ensure that the --read-only-port argument is set to 0 (Scored) [PASS] 2.1.6 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Scored) [FAIL] 2.1.7 Ensure that the --protect-kernel-defaults argument is set to true (Scored) [PASS] 2.1.8 Ensure that the --make-iptables-util-chains argument is set to true (Scored) [PASS] 2.1.9 Ensure that the --hostname-override argument is not set (Scored) [FAIL] 2.1.10 Ensure that the --event-qps argument is set to 0 (Scored) [FAIL] 2.1.11 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored) [PASS] 2.1.12 Ensure that the --cadvisor-port argument is set to 0 (Scored) [PASS] 2.1.13 Ensure that the --rotate-certificates argument is not set to false (Scored) [PASS] 2.1.14 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored) [WARN] 2.1.15 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Not Scored) [INFO] 2.2 Configuration Files [PASS] 2.2.1 Ensure that the kubelet.conf file permissions are set to 644 or more restrictive (Scored) [PASS] 2.2.2 Ensure that the kubelet.conf file ownership is set to root:root (Scored) [PASS] 2.2.3 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Scored) [PASS] 2.2.4 Ensure that the kubelet service file ownership is set to root:root (Scored) [PASS] 2.2.5 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored) [PASS] 2.2.6 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored) [WARN] 2.2.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Scored) [FAIL] 2.2.8 Ensure that the client certificate authorities file ownership is set to root:root (Scored) [PASS] 2.2.9 Ensure that the kubelet configuration file ownership is set to root:root (Scored) [PASS] 2.2.10 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored) == Remediations == 2.1.1 Edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. --allow-privileged=false Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service 2.1.5 If using a Kubelet config file, edit the file to set readOnlyPort to 0 . If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. --read-only-port=0 Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service 2.1.7 If using a Kubelet config file, edit the file to set protectKernelDefaults: true . If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. --protect-kernel-defaults=true Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service 2.1.10 If using a Kubelet config file, edit the file to set eventRecordQPS: 0 . If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. --event-qps=0 Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service 2.1.11 If using a Kubelet config file, edit the file to set tlsCertFile to the location of the certificate file to use to identify this Kubelet, and tlsPrivateKeyFile to the location of the corresponding private key file. If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameters in KUBELET_CERTIFICATE_ARGS variable. --tls-cert-file= file= Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service 2.1.15 If using a Kubelet config file, edit the file to set TLSCipherSuites: to TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 If using executable arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter. --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 2.2.7 Run the following command to modify the file permissions of the --client-ca-file chmod 644 2.2.8 Run the following command to modify the ownership of the --client-ca-file . chown root:root

[stevenmcpherson]

+1 - Same question regarding EKSctl creating workers in public subnets when you include private subnets in the config. Not sure why this is the default behavior. AWS best practice is to place compute resources in private subnets and making apps available via ELB sitting in public subnets. Our security team requires this as well - real sticking point to deploying in our environment with the tool - which works great otherwise

[affanhmalik]

@scottyhq I've spent some time doing the same for the projects I'm working on - get kube-bench to scan EKS worker nodes. Here are some findings which helped me bring the FAIL count down to 1:

• kube-bench configurations: See issue here. I followed this and created by own copy of kube-bench config.yaml with eksctl-specific paths added. Also note that some of the eksctl directories need to be mounted to your kube-bench pod

• nodeGroup properties: Adding the following nodegroup properties:

  kubeletExtraConfig:
    readOnlyPort: 0
    protectKernelDefaults: true (setting this required the preBootstrapCommands below)
    eventRecordQPS: 0      
  preBootstrapCommands:
    - "#!/bin/bash -xe"
    - "cat > /etc/sysctl.d/90-kubelet.conf << EOF \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_on_oops=1 \nEOF"
    - "sysctl -p /etc/sysctl.d/90-kubelet.conf"  

• At this point my only outstanding fail is 2.1.11 (kubelet tls cert arguments)

I hope it helps!

[Himangini]

closing this due to lack of activity

[joebowbeer]

    kubeletExtraConfig:
      eventRecordQPS: 0      

@affanhmalik Is this setting eventRecordQPS=0, which disables any rate limiting (not recommended), or is it setting the deprecated command line option (--event-qps) to 0, which uses the default eventRecordQPS=5?

[affanhmalik]

@joebowbeer To me it looks like eksctl will still process this if passed via config file (despite the CLI option being deprecated). Eksctl will pass it on to the kubelet - which will result in rate limiting getting disabled.

https://eksctl.io/usage/customizing-the-kubelet/#customizing-kubelet-configuration_1

It is important to know that the values specified in the config file for the the fields in kubeletExtraconfig will completely overwrite the default values specified by eksctl.

[joebowbeer]

@affanhmalik I was just pointing out that NOT disabling rate limiting was one of the goals of this benchmark rule.

The --event-qps flag on the Kubelet can be used to limit the rate at which events are gathered. Setting this too low could result in relevant events not being logged, however the unlimited setting of 0 could result in a denial of service on the kubelet.

Note that if using the deprecated command line arguments --event-qps=0 was a synonym for the default value of 5.

But in the KubeletConfiguration, 0 means disable rate limiting (not recommended).

https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/