sparkEchooo commented 8 months ago

Kubecost Helm Chart Version

1.107

Kubernetes Version

s

Kubernetes Platform

GKE

Description

Summary

The Kubecost in GKE gave excessive authority when defining Service Account named "kubecost-1-cost-analyzer-serviceaccount-name-dff5" "kubecost-1-cost-analyzer-prometheus-serviceaccounts-server-name-e82d" and "kubecost-1-deployer-kvsqj". Besides, these Service Accounts are mounted into pod, witch makes it possible for attackers to raise rights to administrators.

Detailed Analysis

I deployed Kubecost in the marketplace of Google's GKE cluster by default.
The clusterrole named "default:kubecost-1:cost-analyzer.serviceAccount.name-r0" defines the "*" verb of "pods, deployments, replicationcontrollers and nodes". And this clusterrole is bound to the Service Account named "kubecost-1-cost-analyzer-serviceaccount-name-dff5". The Service Account is mounted into the pod named "kubecost-1-cost-analyzer-789fc48778-xgpkg".
The clusterrole named "default:kubecost-1:cost-analyzer.prometheus.serviceAccounts.server.name-r0" defines the "*" verb of "pods, jobs, deployments, statefulsets, replicationcontrollers and nodes". And this clusterrole is bound to the Service Account named "kubecost-1-cost-analyzer-prometheus-serviceaccounts-server-name-e82d". The Service Account is mounted into the pod named "kubecost-1-prometheus-server-6f9d5c9989-l972j".
The clusterrole named "default:kubecost-1:deployerServiceAccount-r0" defines the "*" verb of "clusterroles and clusterrolebindings". And this clusterrole is bound to the Service Account named "kubecost-1-deployer-sa". The Service Account is mounted into the pod named "kubecost-1-deployer-kvsqj".

Attacking Strategy

If a malicious user controls a specific worker node which has the pod mentioned above, or steals one of the SA token mentioned above.He/She can raise permissions to administrator level and control the whole cluster. For example,

With the "*" verb of "clusterroles and clusterrolebindings", attacker can elevate privileges by creating a clusterrolebinding resource and binding cluster-admin to their own Service Account.
With the "*" verb of "pods, jobs, deployments, statefulsets, replicationcontrollers", attacker can elevate privileges by creating a pod to mount and steal any Service Account he/she want.
With the "*" verb of nodes, attacker can hijack other components and steal token by adding a "NoExecute" taint to other nodes.

Mitigation Discussion

Developer could use the rolebinding instead of the clusterrolebinding to restrict permissions to namespace.
Developers could define precise permissions for workload resources, including pods, deployments, jobs, statefulsets, replicationcontrollers , rather than using wildcard (*).
The "kubecost-1-deployer" appears to be used for initialization, and developers can delete resources such as the corresponding pod or Service Account after they are no longer needed.

A few questions

Is it a real issue in Kubecost?
If it's a real issue, can Kubecost mitigate the risks following my suggestions discussed in the "mitigation discussion"?
If it's a real issue, does Kubecost plan to fix this issue?

Reporter list

Xingyu Liu(ll.travor@outlook.com, me)
Nanzi Yang(952508578nanziyang@gmail.com/nzyang@stu.xidian.edu.cn)
Xunqi Liu(xunqiliu@stu.xidian.edu.cn)
Xin Guo(guox@stu.xidian.edu.cn)
Wenbo Shen(shenwenbo@zju.edu.cn)
Jinku Li(jkli@xidian.edu.cn)

Looking forward to your reply. Regards Xingyu Liu

Steps to reproduce

Deploy the kubecost by default in GKE.
Use kubectl get sa to get the list of service accounts.
Use kubectl get rolebinding,clusterrolebinding --all-namespaces -o jsonpath='{range .items[?(@.subjects[0].name=="SERVICE_ACCOUNT_NAME")]}[{.roleRef.kind},{.roleRef.name}]{end}' to get the clusterrole related to the service account, and view the permission definition.

Expected behavior

This is a configuration error.

The Service Accounts mentioned above are given excessive authority, witch makes it possible for attackers to raise rights to administrators.

Impact

No response

Screenshots

No response

Logs

No response

Slack discussion

No response

Troubleshooting

[X] I have read and followed the issue guidelines and this is a bug impacting only the Helm chart.
[X] I have searched other issues in this repository and mine is not recorded.

AjayTripathy commented 8 months ago

@sparkEchooo could you share the exact command you used to install kubecost? I don't see any "kubecost-1-deployer" service accounts in the base helm chart.

sparkEchooo commented 8 months ago

Hi @AjayTripathy, thanks for your reply!

Install Kubecost

I install kubecost from the marketplace of Google GKE. Like this,

Pods

And this is the result of command kubectl get pods

ClusterRoleBinding & ClusterRole

This is the ClusterRoleBinding and the ClusterRole related to the service account "kubecost-1-deployer-sa".

Discussion

If Kubecost don't defines the service account named "kubecost-1-deployer-sa", it may be defined by GKE. I will report this to Google.

However, the high authority of "kubecost-1-cost-analyzer-serviceaccount-name-dff5" and "kubecost-1-cost-analyzer-prometheus-serviceaccounts-server-name-e82d" defined by Kubecost is still worth noting.

Thanks again.

AjayTripathy commented 8 months ago

I believe that pod is defined by GKE. You can quickly verify that with a helm install

helm install kubecost cost-analyzer \
--repo https://kubecost.github.io/cost-analyzer/ \
--namespace kubecost --create-namespace \
--set kubecostToken="YWpheUBrdWJlY29zdC5jb20=xm343yadf98"

and diffing the created service accounts.

sparkEchooo commented 7 months ago

You are right.

When I deployed Kubecost locally, I found that the latest version and v1.107.1 (used by GKE) had no problems.

Some questions

The version of kubecost in GKE is v1.107.1. I deployed this version locally, and found that the all permission verbs are "get list watch". But in GKE, the permissions given to the two Service Accounts, "cost-analyzer" and "cost-analyzer-prometheus", are both "*"

Is v1.107.1 in github patched?

chipzoller commented 6 months ago

This issue is related to the Kubecost listing on GCP Marketplace hence transferred to its parent repo. The configuration is found here which is specific to the Marketplace listing and, as pointed out earlier, not part of the main chart. This may have been a requirement for the deployment to succeed in GCP because it's the only place where the differences appear.

AjayTripathy commented 6 months ago

whoops. sorry for the accidental close. I think this is appropriately filed now but there's not much we can do on our side.

sparkEchooo commented 5 months ago

Dear kubecost maintainers,

I kindly request that you issue a relevant security advisory to alert and assist those who, like us, have installed and are using Kubecost on GKE with the default settings. This is a real issue, and we believe there are many others who may encounter this potential security problem without being aware of it beforehand. If possible, please express gratitude for our report in the advisory.

Thank you for your understanding and cooperation.

Regards, Xingyu Liu