A potential risk in piraeus which can be leveraged to make a cluster-level priviege escalation

[younaman]

I am Nanzi Yang, and I find a potential risk in piraeus which can be leveraged to make a cluster-level privilege escalation.

Detailed analysis: The piraeus has one deployment called piraeus-op-controller-manager, which has two pods running on worker nodes randomly. The pod's service account is piraeus-op, which has the piraeus-op-controller-manager cluster role via cluster role binding. The cluster role has get/list/watch verbs of secret resources, has create/patch/update verbs of clusterrolebindings.rbac.authorization.k8s.io, and has create/patch/update verbs of clusterroles.rbac.authorization.k8s.io. Thus if a malicious user can access the worker node which has piraeus-op-controller-manager:

1. He/she can leverage the service account to get ALL secrets in the entire cluster (e.g., the cluster's admin token), resulting in cluster-level privilege escalation.
2. He/she can leverage the service account to bind a high-privileged cluster role (e.g., cluster-admin cluster role) to whatever service account he/she likes, resulting in cluster-level privilege escalation.
3. He/she can modify any existing cluster role to get high-privileged permissions (e.g., GET verbs of secret resources), resulting in cluster-level privilege escalation.

Mitigation Discussion:

1. For the secret-related verbs, the piraeus' maintainers can create a separate Kubernetes namespace, and use the RoleBinding, not the ClusterRoleBinding to restrain the deployment can only access secrets in the separate namespace.
2. For the secret-related verbs, the piraeus' maintainers can also use resource names to restrain the secrets that can be accessed by the deployment.
3. For the ClusterRole&ClusterRoleBinding related verbs, perhaps the best way to mitigate the risks is removing the related permissions. However, it needs a careful review of piraeus' source code without disrupting its functionalities.

A few questions:

1. Is it a real issue in piraeus?
2. If it's a real issue, can piraeus mitigate the risks following my suggestions?

[WanzenBug]

Thank you for the report. We are aware of the potential impact of a compromised piraeus-operator-controller-manager deployment. However, we see limited possibility to change that:

The Piraeus Operator manages all resources related to Piraeus Datastore, which, at the base level, is a Storage Provider for Kubernetes using the CSI mechanism. One of the main operations a storage provider must be able to do is using the "mount()" system call, which already requires local root privileges on a node. Then, we also need to manage DRBD resources, which also require the SYS_ADMIN capability. This means we necessarily need to run high-privileged workloads.

On to the reported issues:

• cluster-wide reading of secrets is necessary to enable users to use secrets in the storage class [1]. We use those secrets so users can securely store information about the their backup location.

• The piraeus-operator deployment itself does not use these permissions, but it creates the service account and RBAC resources necessary for the storage provider to run. One way Kubernetes protects against privilege escalation is that a ServiceAccount can only create a ClusterRule resource for things it can access itself. So in order for the piraeus-operator to create the rules for the csi-provisioner, it needs to have the cluster-wide secret read permission as well.

Thus, I do not see any way to mitigate this potential issue. All of the provided mitigations would impact the functionality in some way. If you do have further suggestions, we'd be interested in hearing them.

[zerotens]

The harder part is to mitigate most attack scenarios. To check for problematic rbac permissions: https://github.com/PaloAltoNetworks/rbac-police

./rbac-police_v1.1.2_linux_amd64 eval lib/ -n piraeus-datastor
...
    "summary": {
        "failed": 8,
        "passed": 15,
        "errors": 0,
        "evaluated": 23
    }

To mitigate all you would have delegate and verify risky rbac permissions to an protected workload on an protected node which does those risky actions on behalf of the workloads which are running on the worker nodes to provide storage.

https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/#controlling-which-nodes-pods-may-access

To be fair: Piraeus is not alone with this problem.

https://i.blackhat.com/USA-22/Thursday/US-22-Avrahami-Kubernetes-Privilege-Escalation-Container-Escape-Cluster-Admin.pdf