kops stores Digital Ocean keys insecurely

[ghost]

This was disclosed through Hacker One, but was improperly closed after review; so this disclosure is being done publicly to ensure visibility.

When using kops for Digital Ocean, it supplies API keys through user init. API keys on Digital Ocean are inheriently insecure and must be transmitted securely.

Any unprivileged user with access on a server kicked by kops in Digital Ocean can curl the metadata API and see the API key in plain text when viewing the user data supplied.

This means that an unprivileged user that may have only had access to a single server can become Digital Ocean account god.

A secure method of transmission needs to be used for Digital Ocean API keys, for which user init is not that.

[olemarkus]

Thanks for your interest in the security of kOps on Digital Ocean.

First of, the control plane must have access to the cloud provider APIs to provision the resources the cluster requires. for example volumes and load balancers.

On many cloud providers, this means access keys or equivalent must be stored at minimum on the control plane, but often also on worker nodes (for gossip in particular).

Storing keys in metadata is the only method that we have for many of our cloud providers. There are simply no other mechanism to transfer anything into instances as they boot. Technically, as we are using IAM Instance Profiles on AWS, even there, metadata contains sensitive data.

Unprivileged users of any kind should not have direct shell access to instances in a Kubernetes cluster. There are many risks to that beyond metadata access. So take measure to ensure only those that really need access have that form of access to instances.

The more serious concern is with Pods. Unrestricted Pods can simply call the metadata API directly themselves. On AWS, people deployed KIAM and similar that protected metadata access, but with AWS IMDSv2, the API is able to protect itself from Pods. For other cloud providers, your option is to use network policies to firewall the API.

In short, there may be a lot more kOps could do out of the box or at least document. But the use and risk of metadata API is fairly well-known, and the secure methods you are talking about doesn't exist to my knowledge. And even if it did, unchecked Pods would still be able to fetch the data using the same mechanism.

/cc @srikiz @timoreimann Anything you want to add?

[ghost]

AWS gets around this problem with roles and Digital Ocean does now have such a feature. However, this means that there is a higher level of care required when dealing with the API keys.

Passing API keys this way in Digital Ocean essentially turns zero day exploits that may happen inside of a container with a narrow vector into cloud account-wide compromises.

This prevents kops from achieving production-ready status on Digital Ocean as well.

To be completely fair though, I'm not 100% sure how to remedy this without having some form of tooling to orchestrate coordination during init when user-data would be ran. There hasn't been some easy fix I could think up for this that I would recommend to implement.

[olemarkus]

AWS' use of roles do not mitigate this as the credentials for assuming a role are stored in metadata as well. But IMDSv2 hop limit=1 prevents pods from being able to assume role because they need at least two network hops to connect to IMDS. Unless hostnetworking is used. Those pods will be able to assume the instance role as well.

I.e make sure you use network policies and make sure you use PSS to block pods with host networking.

[ghost]

TIL. I was blissfully unaware of that until now.

Seems there are some tools that expect access to the metadata service to work as well. If there was something like a ClusterNetworkPolicy CR implemented to make cluster-wide defaults, it seems like this would be a much easier problem to tackle.

If no one shows up with an idea for this, perhaps clearly documenting this with some example NetworkPolicies is reasonable enough.

[johngmyers]

In AWS, before IMDS hop limits existed, my employer addressed this by having a hook on worker nodes which upon nodeup completion signaled a lambda which changed the instance role to one with negligible privileges. Perhaps DO's equivalent to instance roles allows a similar mechanism for dropping privileges?

[ghost]

Digital Ocean's API doesn't even have an equivalent afaik. The API keys generated are either RW-all or RO-all. It would be a pleasant surprise to learn I'm wrong and they rolled something new out without me seeing it.

Maybe kops-controller and nodeup could be used to facilitate a handoff? Using kops-controller to share the key to the instance for use while nodeup gets it joined into the cluster seems reasonable as long as it doesn't get stored and is forgotten after first use.

The key would still need to be passed to masters via user data, but it's a much smaller surface. It just seems unavoidable I think.

[zetaab]

@protosam example of networkpolicy that we are using with calico in OpenStack https://gist.github.com/zetaab/3408b9992630133f822644463ec81906

this means that namespaces that do have label kaas.foobar.com/allow-cloud-meta: true can access metadata and others are blocked automatically. kube-system needs access to metadata but others by default does not need that.

I have no idea how other CNIs has cluster wide networkpolicies but calico at least have and its working pretty well.

[zetaab]

@johngmyers at least in OpenStack there is only one credential that is admin for everything, so its "tenant level access" not instance. I think pretty much same applies to DO, the roles does not exists in that sense

[ghost]

Having the DO token exposed to worker nodes is an unnecessarily really wide surface space to have open like this. Suppose in the future there's another linux namespace escape vulnerability that is discovered.

Add 1 linux namespace escape + 1 DO key in your metadata = 1 stolen DO account?

It would be naive to leave this as-is for the DO deployment and expect just masking it to be sufficient with how fast development happens on k8s and the linux kernel these days.

[johngmyers]

It would be up to DO to provide some mechanism to allow this to be secured. It's not possible to secure something when the necessary primitives to do so are absent.

Worker nodes at the minimum need a credential to authenticate kubelet to apiserver. It might be possible to avoid giving worker nodes the credential to the DO cloud API.

[ghost]

The kops-controller can be used to pass information in a more secure way if tooled to do so. (Have already eluded to this)

Just because Digital Ocean doesn't have the most flexible toolkit, doesn't mean it can't be tooled on top of in a more secure way. If KOPS is ever to be production ready for DO, it can't do so with API keys inside of user-data for the worker nodes in plain text.

Just a reminder, one of the stated goals of KOPS is to provide production grade clusters. That requires a bit more tooling when you start looking at the budget clouds.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[hakman]

/remove-lifecycle stale /lifecycle frozen

[timoreimann]

👋 Timo here from DigitalOcean.

As folks have pointed out correctly, DO unfortunately does not have more fine-grained capabilities in what API keys can access (except for a r/w vs r/o distinction) as of today.

I do agree that network policies could be a viable solution to restrict access. Especially with clusters where operators run untrusted workloads, this would likely be a must-have anyway. Inside DO, we use Cilium network policies heavily to lock down access to the bare minimum needed. That said, network policies definitely come with a (maintenance) cost.

Would it be possible to separate use cases where worker nodes really need API tokens vs where their usage could possibly be restricted to the control plane nodes? I understood that the gossip protocol may require it, which sounded like users not using that protocol could possibly benefit from a change in provisioning configuration where tokens are not injected into worker node metadata?

Yet another idea could be to think about different ways to deliver tokens. Using S3/Spaces with signed URLs and some kind of node-local agent pulling those comes to my mind. That's probably quite a lift though so it'd likely require more exploration.

[zetaab]

@timoreimann new "dns none" in kOps solves these issues quite well already in OpenStack. Before that we were running gossip clusters, which means that basically all nodes + control planes do have credentials to openstack project (and vfs store). However, now we have migrated to dns none and kops-controller bootstrap which means that nodes does not have any credentials anymore. Control-planes still have, but it is another discussion how to remove those. So what I am saying is that DO could also implement needed things in "dns none" and kops-controller bootstrap. If you need help, you can ping me / @hakman in Slack.

[johngmyers]

The main thing kOps needs from a cloud provider is a mechanism for instance identity: a way for an instance/droplet to prove to a service that it was genuinely provisioned to perform a particular role.

In AWS, this is the "instance role"; there are credentials that can be obtained from the metadata service wich can, with an admittedly over-complicated procedure, be used to authenticate a request. In GCP, there's a TPM. It also needs to be possible to block access to said credentials from unprivileged pods.

Given such a mechanism, nodeup on worker nodes could then obtain certificates, secrets, and config from kops-controller. It wouldn't be necessary to distribute cloud API tokens to worker nodes.