Switch for "Priority"

[prasenforu]

Looking for some enhancement based on "Priority" Switch.

[g3rzi]

Like filtering based "Priority"?

[prasenforu]

Yes

[g3rzi]

I added support for filtering by priority. The new switch is -p or --priority and you need to specify what kind of priority (CRITICAL\HIGH\LOW) and it is not case sensitive.
For example:
python3 KubiScan.py -rcr -p critical

This will print you all the risky cluster roles with CRITICAL priority. I tested it and it works.
Please check.

[prasenforu]

Fantastic, will check & let you know the feedback.

[prasenforu]

It works !!

but there was a switch -p use for pod (though I have not used yet), not sure whether it will impact any thing or not.

Dumping tokens:
  Use the switches: pods (-p\--pods) or all pods (-ap\ --all-pods)

  -dt, --dump-tokens    Dump tokens from pod\pods Example: -dt OR -dt -ns
                        "kube-system" -dt -n "nginx1" -ns "default"

One more thing for --risky-pods it gives all pods which is comes with long run.

But in real life scenario people used to search in particular namespace, like

KubiScan.py -rp -ns "kube-system"

If we get this enhancement (risky-pods within namespace) I think that will be better.

[g3rzi]

Oh, You are right. I will change it to something else, thanks for notice !

Regarding the risky pods, it because I am using exec on each pod to get the token which is a bit slow. I am planning to had switch to make it faster by reading the pod's token from the ETCD and not by executing commands insight, but it will be less reliable.

Regarding the namespace on risky pods, I wil add this capability.

I will update.

[g3rzi]

I just got remembered that I didn't implement the switches pods (-p\--pods) or all pods (-ap\ --all-pods) because I made the -dt switch to get by default all the pods, so it was unnecessary to add another switch.

I also didn't implemented the -p because there is already the -n switch to specify the name of the pod. The documentation is wrong, I will change it.

[prasenforu]

Ok.

But if you can think

KubiScan.py -rp -ns "kube-system"

If we get this enhancement (risky-pods within namespace) I think that will be better

[g3rzi]

Added, you can check now. Example: KubiScan.py -rp -ns "kube-system"

[prasenforu]

Getting following error.

Exception when calling api_client.CoreV1Api->connect_post_namespaced_pod_exec: (0)
Reason: Handshake status 403 Forbidden

[g3rzi]

This error means that you don't have permissions to get pods. Are you using a token or config file?

[prasenforu]

I am using token.

If it's a permission issue then how following command is working?

python3 KubiScan.py -psv -ns "kube-system"

[g3rzi]

Because the KubiScan.py -rp -ns "kube-system" command is executing commands inside pods, so you need also the permission "pod/exec".
You probably have permissions to list pods but not exec inside them.

[g3rzi]

Also, make sure that the permissions is on the whole cluster (ClusterRoleBinding and ClusterRole). There is an example for service account permissions in the readme file.

[prasenforu]

I am following as per your readiness doc for remote connections.

https://github.com/cyberark/KubiScan/blob/master/README.md#with-service-account-token-good-from-remote

[prasenforu]

I don't think issue is permission related.

Though in document it mentioned

pod/exec which is not correct.

It should be pods/exec (s is missing).

But after editing I my side also it's throwing same error.

And I installed kubectl in your image also and checked that I am able to do

kubectl exec -it <podname> sh

So it not related with permission, something else but I am not sure.

[g3rzi]

You are right, I checked the token I created and I am also getting the same errors so it's good because I can test it :).

I ran the kubectl with verbose to see what is does:

root@ubuntu:~/KubiScan# kubectl exec -it -v=6 risky-pod -- sh -c 'ls'
I0327 06:31:16.708740   15206 loader.go:359] Config loaded from file /home/ubuntu/.kube/config
I0327 06:31:16.710679   15206 loader.go:359] Config loaded from file /home/ubuntu/.kube/config
I0327 06:31:16.712744   15206 loader.go:359] Config loaded from file /home/ubuntu/.kube/config
I0327 06:31:16.723992   15206 round_trippers.go:438] GET https://10.0.62.4:8443/api/v1/namespaces/default/pods/risky-pod 200 OK in 8 milliseconds
I0327 06:31:16.759524   15206 round_trippers.go:438] POST https://10.0.62.4:8443/api/v1/namespaces/default/pods/risky-pod/exec?command=sh&command=-c&command=ls&container=nginx&container=nginx&stdin=true&stdout=true&tty=true 101 Switching Protocols in 28 milliseconds

The permissions should be pods/exec liked you mentioned as it appears in the API:
/api/v1/namespaces/default/pods/risky-pod/exec

But I am also getting the same error. The problem is with some missing resource because when I added "*" in the resources it works fine:

kubectl apply -f - << EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata: 
  name: kubiscan-clusterrole
rules: 
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["get", "list"]
- apiGroups: ["*"]
  resources: ["pods/exec"]
  verbs: ["create"]
EOF

[g3rzi]

I found the problem, missing verb get in the resource pods/exec.
I added it, it should work now. Just update the ClusterRole by running:

kubectl apply -f - << EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata: 
  name: kubiscan-clusterrole
rules: 
- apiGroups: ["*"]
  resources: ["roles", "clusterroles", "rolebindings", "clusterrolebindings", "pods", "secrets"]
  verbs: ["get", "list"]
- apiGroups: ["*"]
  resources: ["pods/exec"]
  verbs: ["create", "get"]
EOF

[prasenforu]

After correction of cluster role, error not coming but it's taking longer time to execute and finally results not produced, I have to come out by executing "crtl+c".

Some problems is there.

I have a quick question, why are we doing pod exec? Can we skip this method!!

I think better approach to find in etcd then put some logic to make risky pod.

What do you think?

[g3rzi]

I think this is a good idea and I was planning to add it already. https://github.com/cyberark/KubiScan/blob/486b71173d471cad91214f213fcdf972f86cfb38/engine/utils.py#L290

The reason is that exec is more reliable in case someone changed the token. I know that the chances that it will happen are rare but it is possible.

I am planning to add the new switch that you will be able to choose by executing inside a pod or by querying ETCD which will be much faster.

[g3rzi]

Now you can run -rp\--risky-pods and it will be much faster because it will read it directly from the ETCD. I left an option (-d\--deep) to be able to read the token directly from the containers.

The new switch also works with -dt\--dump-tokens switch.

[prasenforu]

Really fast.

Thanks closing now.

Little bit interested in what basis you making pod as a risky pod?

[g3rzi]

Risky pods are pods that contains risky users. Risky users are users that contain risky permissions. For example, if you have user that have permissions to list all the secrets in the cluster and its token is mounted to containerA, then containerA will be marked as risky container.

I am sure that most of the people, if not 100% of them, will agree that listing all the secrets in the cluster is risky permission. But what about permission to delete service account or just viewing the pods? Some might say yes, some not.

Due to the subjective interpretation of "risky" permissions, I created a file that anyone can decide what risky permissions are. In this file I wrote all the permissions that I think are risky but you can change it according to your thoughts.

So, if you want KubiScan to scan for containers that have token with permissions to list all the secrets you will add the following role to the risky_roles.yaml file:

- kind: Role
  metadata:
    namespace: default
    name: risky-list-secrets
    priority: CRITICAL
  rules:
  - apiGroups: ["*"]
    resources: ["secrets"]
    verbs: ["list"]

*the namespace and the name of the role are currently not relevant for the check. Maybe they will be used in the future.

KubiScan will search for all the containers in the cluster, read their tokens. Each token is a service account, so it will list all the service account attached roles and check if they contains the verb "list" and resource "secrets". If it does, it will be marked as risky and you can specify what priority (CRITICAL\HIGH\LOW) you think it should be.