chrisevett
commented
5 years ago I was curious about the same thing, is this a difference of opinion, or a gap in kops? I'd be glad to contribute to implementing some of these security improvements.
Open cheynewallace opened 5 years ago
chrisevett
commented
5 years ago I was curious about the same thing, is this a difference of opinion, or a gap in kops? I'd be glad to contribute to implementing some of these security improvements.
faheem-nadeem
commented
5 years ago Related. https://github.com/kubernetes/kops/issues/4688. Been trying to get this issue opened, as there are more discussions there. :)
justinsb
commented
5 years ago Thanks for the list - it is much easier to come up with a checklist than it is to actually implement them in the real world when people have running clusters :-) We have to think about whether each one of these options breaks people.
That said there's some good stuff here, like --anonymous-auth=false . kops passes that one, but my understanding is that most of the other configurations out there would fail that one. So we got lucky on the recent CVE there, but there's enough in here that we might not be so lucky next time.
I do think we should use this list to drive a goal of being the most secure configuration available - I have it on a discussion item for "roadmap" for today's kops office hours.
As a taster of the complexities though, I spotted AlwaysPullImages should be set - that's really difficult. As I understand it, the problem is that kubernetes doesn't keep track of whether an image was previously pulled using a secret, and so if another tenant has pulled an image with a secret, another user on the same node will be able to pull it. AlwaysPullImages forces everyone to pull, all the time. The problem is that this creates a harder dependency on the image registry being available. Some people might want to be able to run a pod even if their image registry is unavailable, if the image happens to be cached (and k8s does try to schedule pods to nodes where the image is cached). And most people likely aren't running hostile tenants on the same cluster. So ... it's complicated :-)
Let's tag for 1.13 as a target list (we can pull things into earlier releases as well). (And I think some of these are actually getting fixed in kops 1.11)
justinsb
commented
5 years ago Opened upstream issue for AlwaysPullImages: https://github.com/kubernetes/kubernetes/issues/71850
/milestone 1.12
cheynewallace
commented
5 years ago Thanks for following up @justinsb. I just updated my original post with the full list of failing tests after performing a Kops upgrade to Kubernetes 1.10.11 (as a result of CVE-2018-1002105)
There is a full list of remediation steps that comes with that checklist if you want me to post it, although it's several pages long.
Let me know if I can help at all 👍
eupestov
commented
5 years ago Hi
While it should be possible to paint green most of the API Server and Controller Manager checks after https://github.com/kubernetes/kops/pull/4799 gets merged, I am wondering what needs to be done to fix those in the '1.4 Configuration Files' group? Are there any reasons the actual permissions are broader than prescribed?
eupestov
commented
5 years ago I've just discovered that file paths for component configuration files created by kops are different to the kubeadm defaults used by kube-bench in https://github.com/aquasecurity/kube-bench/blob/master/cfg/1.11/config.yaml
Fixing the path (.manifest instead of .yaml) has turned green the checks 1.4.1-1.4.6
raesene
commented
5 years ago FWIW if you're looking for things to fix out of this list, as default settings, I'd thoroughly recommend looking at these two on the kubelet.
[FAIL] 2.1.2 Ensure that the --anonymous-auth argument is set to false (Scored)
[FAIL] 2.1.3 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored)as a priority. This combination allows anyone who can hit the kubelet port (10250/TCP) at a network level, to easily get root access on the underlying node.
Also worth noting that there are newer versions of the benchmark out now which drop some of the recommendations which could cause issues with cluster operation (e.g. the --allow-privileged=false one has gone, in favour of PSPs)
raesene
commented
5 years ago I did a default kops install today and a couple of other ones you might want to look at changing
--insecure-bind-address and --insecure-port . Even just on localhost the insecure API port can be very dangerous. It lets anyone who can get access to the localhost on the master node (which could be through a variety of routes) get full cluster admin access easily.
Using basic and token auth is generally considered a bad idea as they both store credentials in-clear on disk.
etcd without authentication. Again although this is only bound to localhost, from a security standpoint, unauthenticated etcd is dangerous as it contains all the secrets from the cluster.
A more minor point , but one worth considering. All the keys stored in /srv/kubernetes are valid for 10 years. An attacker who compromises one of those keys gets persistent access to the targeted cluster, as they don't expire for a long time.
Peter-Lankton
commented
5 years ago @raesene so are you saying to avoid using --insecure-bind-address and --insecure-port altogether or are you saying to set them to some default value other than localhost like 0.0.0.0? I'm asking because I am working on our cluster_spec that kops uses to provision and am using kube-bench to get the cluster into better shape security wise.
raesene
commented
5 years ago @llcranmer from a security point of view the current correct configuration I believe is to configure --insecure-port=0 explicitly, and don't configure --insecure-bind-address to stop it listening (without that the default will be to listen on localhost port 8080).
when you have the cluster running the best way to check if it's enabled or not is to check for running ports with something like ss -ltnp and confirm that the kube-apiserver isn't listening on port 8080/TCP.
What I can't tell you is, what kops uses the insecure port for (if anything). If it's making use of the insecure port, then obviously disabling it could have consequences for the cluster.
Peter-Lankton
commented
5 years ago @raesene Thank you for clarifying. I'll report back if setting insecure-port=0 causes any problems.
Peter-Lankton
commented
5 years ago @raesene I've done some digging and it looks like it defaults to 8080. It looks like there are plans to deprecate the flag for kops as it has been deprecated for kubernetes in the future moving forward. Consequently, when editing the cluster spec kops doesn't let me choose '0', so guessing it is being used.
raesene
commented
5 years ago FWIW, I was looking around for this today and saw https://github.com/kubernetes/kubernetes/issues/43784 . So it sounds like the reason it's not disabled in kops is so that anonymous-auth can be set to false.
Personally, I'd say that allowing a healthz endpoint to be available unauthenticated is less of a risk than enabling the use of the insecure port, even on localhost. There's a number of scenarios where an attacker might be able to make a request the API server on localhost, and the insecure port always allows cluster-admin so that's not a good thing...
fejta-bot
commented
4 years ago Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
rifelpet
commented
4 years ago /remove-lifecycle stale
fejta-bot
commented
4 years ago Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
rifelpet
commented
4 years ago /remove-lifecycle stale /lifecycle frozen
rifelpet
commented
4 years ago I ran kube-bench 0.3.1 against a cluster built from kops' master branch and these are the FAIL and WARN results.
Some of them are outside the responsibilities of kops, and some are false positives (https://github.com/aquasecurity/kube-bench/issues/667). The file permission and ownership failures should be straight forward fixes. The arguments being set on the control plane components might be a bit more involved since it requires changing default behaviors. We can audit the remaining issues and try to address them as necessary.
[INFO] 1 Master Node Security Configuration
[INFO] 1.1 API Server
[FAIL] 1.1.5 Ensure that the --insecure-bind-address argument is not set (Scored)
[FAIL] 1.1.8 Ensure that the --profiling argument is set to false (Scored)
[FAIL] 1.1.9 Ensure that the --repair-malformed-updates argument is set to false (Scored)
[FAIL] 1.1.11 Ensure that the admission control plugin AlwaysPullImages is set (Scored)
[FAIL] 1.1.12 Ensure that the admission control plugin DenyEscalatingExec is set (Scored)
[FAIL] 1.1.13 Ensure that the admission control plugin SecurityContextDeny is set (Scored)
[FAIL] 1.1.15 Ensure that the --audit-log-path argument is set as appropriate (Scored)
[FAIL] 1.1.16 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Scored)
[FAIL] 1.1.17 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Scored)
[FAIL] 1.1.18 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Scored)
[FAIL] 1.1.21 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Scored)
[FAIL] 1.1.23 Ensure that the --service-account-lookup argument is set to true (Scored)
[FAIL] 1.1.24 Ensure that the admission control plugin PodSecurityPolicy is set (Scored)
[WARN] 1.1.30 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Not Scored)
[FAIL] 1.1.32 Ensure that the --authorization-mode argument is set to Node (Scored)
[FAIL] 1.1.34 Ensure that the --experimental-encryption-provider-config argument is set as appropriate (Scored)
[WARN] 1.1.35 Ensure that the encryption provider is set to aescbc (Scored)
[FAIL] 1.1.36 Ensure that the admission control plugin EventRateLimit is set (Scored)
[FAIL] 1.1.37b Ensure that the AdvancedAuditing argument is not set to false (Scored)
[WARN] 1.1.39 Ensure that the API Server only makes use of Strong Cryptographic Ciphers ( Not Scored)
[INFO] 1.2 Scheduler
[FAIL] 1.2.1 Ensure that the --profiling argument is set to false (Scored)
[INFO] 1.3 Controller Manager
[FAIL] 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Scored)
[FAIL] 1.3.2 Ensure that the --profiling argument is set to false (Scored)
[FAIL] 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)
[INFO] 1.4 Configuration Files
[FAIL] 1.4.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Scored)
[FAIL] 1.4.8 Ensure that the etcd pod specification file ownership is set to root:root (Scored)
[WARN] 1.4.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Not Scored)
[WARN] 1.4.10 Ensure that the Container Network Interface file ownership is set to root:root (Not Scored)
[FAIL] 1.4.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Scored)
[FAIL] 1.4.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Scored)
[FAIL] 1.4.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Scored)
[FAIL] 1.4.14 Ensure that the admin.conf file ownership is set to root:root (Scored)
[FAIL] 1.4.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Scored)
[FAIL] 1.4.16 Ensure that the scheduler.conf file ownership is set to root:root (Scored)
[FAIL] 1.4.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Scored)
[FAIL] 1.4.18 Ensure that the controller-manager.conf file ownership is set to root:root (Scored)
[INFO] 1.5 etcd
[FAIL] 1.5.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Scored)
[FAIL] 1.5.2 Ensure that the --client-cert-auth argument is set to true (Scored)
[FAIL] 1.5.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Scored)
[FAIL] 1.5.5 Ensure that the --peer-client-cert-auth argument is set to true (Scored)
[WARN] 1.5.7 Ensure that a unique Certificate Authority is used for etcd (Not Scored)
[INFO] 1.6 General Security Primitives
[WARN] 1.6.1 Ensure that the cluster-admin role is only used where required (Not Scored)
[WARN] 1.6.2 Create administrative boundaries between resources using namespaces (Not Scored)
[WARN] 1.6.3 Create network segmentation using Network Policies (Not Scored)
[WARN] 1.6.4 Ensure that the seccomp profile is set to docker/default in your pod definitions (Not Scored)
[WARN] 1.6.5 Apply Security Context to Your Pods and Containers (Not Scored)
[WARN] 1.6.6 Configure Image Provenance using ImagePolicyWebhook admission controller (Not Scored)
[WARN] 1.6.7 Configure Network policies as appropriate (Not Scored)
[WARN] 1.6.8 Place compensating controls in the form of PSP and RBAC for privileged containers usage (Not Scored)
[INFO] 1.7 PodSecurityPolicies
[WARN] 1.7.1 Do not admit privileged containers (Not Scored)
[WARN] 1.7.2 Do not admit containers wishing to share the host process ID namespace (Scored)
[WARN] 1.7.3 Do not admit containers wishing to share the host IPC namespace (Scored)
[WARN] 1.7.4 Do not admit containers wishing to share the host network namespace (Scored)
[WARN] 1.7.5 Do not admit containers with allowPrivilegeEscalation (Scored)
[WARN] 1.7.6 Do not admit root containers (Not Scored)
[WARN] 1.7.7 Do not admit containers with dangerous capabilities (Not Scored)[INFO] 2 Worker Node Security Configuration
[INFO] 2.1 Kubelet
[FAIL] 2.1.1 Ensure that the --allow-privileged argument is set to false (Scored)
[FAIL] 2.1.3 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored)
[FAIL] 2.1.5 Ensure that the --read-only-port argument is set to 0 (Scored)
[FAIL] 2.1.7 Ensure that the --protect-kernel-defaults argument is set to true (Scored)
[FAIL] 2.1.10 Ensure that the --event-qps argument is set to 0 (Scored)
[FAIL] 2.1.11 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored)
[FAIL] 2.1.14 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)
[WARN] 2.1.15 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Not Scored)
[INFO] 2.2 Configuration Files
[WARN] 2.2.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Scored)[INFO] 4 Worker Node Security Configuration
[INFO] 4.1 Worker Node Configuration Files
[WARN] 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Scored)
[INFO] 4.2 Kubelet
[FAIL] 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored)
[FAIL] 4.2.4 Ensure that the --read-only-port argument is set to 0 (Scored)
[FAIL] 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored)
[WARN] 4.2.8 Ensure that the --hostname-override argument is not set (Not Scored)
[WARN] 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Not Scored)
[FAIL] 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored)
[FAIL] 4.2.12 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)
[WARN] 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Not Scored)[INFO] 5 Kubernetes Policies
[INFO] 5.1 RBAC and Service Accounts
[WARN] 5.1.1 Ensure that the cluster-admin role is only used where required (Not Scored)
[WARN] 5.1.2 Minimize access to secrets (Not Scored)
[WARN] 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Not Scored)
[WARN] 5.1.4 Minimize access to create pods (Not Scored)
[WARN] 5.1.5 Ensure that default service accounts are not actively used. (Scored)
[WARN] 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Not Scored)
[INFO] 5.2 Pod Security Policies
[WARN] 5.2.1 Minimize the admission of privileged containers (Not Scored)
[WARN] 5.2.2 Minimize the admission of containers wishing to share the host process ID namespace (Scored)
[WARN] 5.2.3 Minimize the admission of containers wishing to share the host IPC namespace (Scored)
[WARN] 5.2.4 Minimize the admission of containers wishing to share the host network namespace (Scored)
[WARN] 5.2.5 Minimize the admission of containers with allowPrivilegeEscalation (Scored)
[WARN] 5.2.6 Minimize the admission of root containers (Not Scored)
[WARN] 5.2.7 Minimize the admission of containers with the NET_RAW capability (Not Scored)
[WARN] 5.2.8 Minimize the admission of containers with added capabilities (Not Scored)
[WARN] 5.2.9 Minimize the admission of containers with capabilities assigned (Not Scored)
[INFO] 5.3 Network Policies and CNI
[WARN] 5.3.1 Ensure that the CNI in use supports Network Policies (Not Scored)
[WARN] 5.3.2 Ensure that all Namespaces have Network Policies defined (Scored)
[INFO] 5.4 Secrets Management
[WARN] 5.4.1 Prefer using secrets as files over secrets as environment variables (Not Scored)
[WARN] 5.4.2 Consider external secret storage (Not Scored)
[INFO] 5.5 Extensible Admission Control
[WARN] 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Not Scored)
[INFO] 5.6 General Policies
[WARN] 5.6.1 Create administrative boundaries between resources using namespaces (Not Scored)
[WARN] 5.6.2 Ensure that the seccomp profile is set to docker/default in your pod definitions (Not Scored)
[WARN] 5.6.3 Apply Security Context to Your Pods and Containers (Not Scored)
[WARN] 5.6.4 The default namespace should not be used (Scored)
DOboznyi
commented
3 years ago 4.2.10/2.1.11 fixed in #10022, thanks @olemarkus
DOboznyi
commented
3 years ago Most of 1.4 fixed in PR to kube-bench https://github.com/aquasecurity/kube-bench/pull/798 1.1.7 and 1.1.8 done https://github.com/aquasecurity/kube-bench/pull/797 1.2.1 also done
agilgur5
commented
3 years ago 4.2.10/2.1.11 fixed in #10022, thanks @olemarkus
Ensure that the --kubelet-certificate-authority argument is set as appropriate (Scored).It's not set by default by k8s and doesn't seem to be set by kOps by default either, but could be set now due to #10022 .
I think one can workaround this in user-land by setting kubeletCertificateAuthority to /srv/kubernetes/ca.crt via #8661. /srv/kubernetes is mounted, so I think it should work, but will try to report back on that
checking the mount:
❯ kubectl -n kube-system get pods kube-apiserver-ip-<ip-address>.ec2.internal -o yaml | grep " volumes:" -A 50 | grep "/srv/kubernetes" -B 1 -A 4
- hostPath:
path: /srv/kubernetesEDIT: the above^ didn't work for us and broke the cert chain
I just ran
kube-bench(https://github.com/aquasecurity/kube-bench) across my KOPS 1.10 provisioned clusters (Kubernetes1.10.11running kope.io/k8s-1.10-debian-jessie-amd64-hvm-ebs-2018-08-17) and I was quite shocked to see a large number of tests failed for both nodes and masters.I expected a few, but not quite this many.
Is there any reason why there seems to be so many security issues with both node and master machines provisioned with KOPS?
Some output
Workers
Masters
UPDATED WITH FULL LIST AFTER VERSION 1.10.11 UPGRADE