danielsagi
commented
2 years ago Thanks for reporting! This looks really weird. could you provide the full debug log data? @AljoschaP
Open aljoshare opened 2 years ago
danielsagi
commented
2 years ago Thanks for reporting! This looks really weird. could you provide the full debug log data? @AljoschaP
julien-sugg
commented
2 years ago Hi,
I started playing with kube-hunter today with my playground cluster on GKE and I am facing a similar issue when I create a dedicated NetworkPolicy for the Pod.
I will create a separate issue if @AljoschaP 's issue is not exactly the same as the one I am facing.
kube-hunter: v0.5.2 built from the Dockerfile in the repository for tag 0.5.2
kubernetes: v1.19.10-gke.1600 (Server) and v1.19.4 (Client)
The simplified manifests below are sufficient to reproduce the issue and non-cluster resources will be located in the default namespace.
Note that, for brievety, the NetworkPolicy applies to the whole namespace (podSelector: {})
NetworkPolicies are baked by Calico CNI plugin.
# kube-hunter.yaml
---
# SA
apiVersion: v1
kind: ServiceAccount
metadata:
name: ci-kube-hunter-in-cluster
namespace: default
---
# ClusterRole
# FIXME, kube-hunter project doesn't seem provide an exhaustive list of the required
# RBAC permissions, so we simply grant it read access (get, list) to all resources for now
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ci-kube-hunter-in-cluster
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["get", "list"]
---
# Bind SA to the ClusterRole.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ci-kube-hunter-in-cluster
subjects:
- kind: ServiceAccount
name: ci-kube-hunter-in-cluster
namespace: default
apiGroup: ""
roleRef:
kind: ClusterRole
name: ci-kube-hunter-in-cluster
apiGroup: rbac.authorization.k8s.io
---
# NetworkPolicy
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: ci-kube-hunter-in-cluster
namespace: default
spec:
policyTypes:
- Egress
- Ingress
podSelector: {} # simplified to match all pods for brievety
egress:
# Allow DNS
- to:
- podSelector:
matchLabels:
k8s-app: kube-dns
namespaceSelector:
matchLabels:
networking/namespace: kube-system
ports:
- protocol: TCP
port: 53
- protocol: UDP
port: 53
# Allow Kubernetes networks
# TODO, need to check in details what kube-hunter really needs to access
- to:
- ipBlock:
# Master Network (kube api server)
cidr: *redacted*
- ipBlock:
# Nodes Networks
cidr: *redacted*
- ipBlock:
# Pods IPs
cidr: *redacted*
- ipBlock:
# Service IPs
cidr: *redacted*
---
apiVersion: batch/v1
kind: Job
metadata:
name: ci-kube-hunter-in-cluster
namespace: default
spec:
template:
spec:
serviceAccountName: ci-kube-hunter-in-cluster
containers:
- name: kube-hunter-in-cluster
# @see https://github.com/aquasecurity/kube-hunter/tree/main#container
image: *redacted*/kube-hunter:0.5.2 # ${KUBE_HUNTER_VERSION}
command: ["kube-hunter"]
args: ["--pod", "--report", "plain", "--log", "debug"]
restartPolicy: Neverk apply -f kube-hunter.yamlThe logs below were generated from a kube-hunter run on a freshly generated cluster with no additional workloads.
Note that I manually redacted the auth_token and client_cert info using a regexp
# The output is too long for GitHub commentsSee output.txt
Note that when I remove the NetworkPolicy, or more precisely perform an allow all on egress (.spec.egress[0].to: {}), the error doesn't occur any more and the report is properly available in stdout.
Optionally, can you please clarify, if available, the exhaustive list of required RBAC permissions and network egress targets ?
aljoshare
commented
2 years ago Sorry for the late reply. I think the issue of @julien-sugg is the same as the one that occurs in our cluster.
Alevsk
commented
9 months ago Same issue is happening to me when running kube-hunter inside the cluster using the --pod flag
Kube-hunter deployment
apiVersion: v1
kind: Pod
metadata:
name: kube-hunter-with-sa
namespace: default
labels:
app: kube-hunter-with-sa
spec:
serviceAccountName: kube-hunter
containers:
- name: kube-hunter-with-sa
image: aquasec/kube-hunter:0.6.8
command: ["kube-hunter"]
args: ["--pod", "--log", "debug"]
restartPolicy: NeverKubernetes version:
NAME STATUS ROLES AGE VERSION
workshop-cluster-control-plane Ready control-plane 3d3h v1.27.3Full error log
2023-09-23 07:13:24,403 DEBUG kube_hunter.core.events.event_handler Event <class 'kube_hunter.core.events.types.HuntFinished'> got published to hunter - <class 'kube_hunter.modules.report. │
│ 2023-09-23 07:13:24,403 DEBUG kube_hunter.core.events.event_handler Executing <class 'kube_hunter.modules.report.collector.SendFullReport'> with {'previous': None, 'hunter': None} │
│ 2023-09-23 07:13:24,406 DEBUG kube_hunter.core.events.event_handler '<' not supported between instances of 'str' and 'IPAddress' │
│ Traceback (most recent call last): │
│ File "/usr/local/lib/python3.8/site-packages/kube_hunter/core/events/event_handler.py", line 346, in worker │
│ hook.execute() │
│ File "/usr/local/lib/python3.8/site-packages/kube_hunter/modules/report/collector.py", line 56, in execute │
│ report = config.reporter.get_report(statistics=config.statistics, mapping=config.mapping) │
│ File "/usr/local/lib/python3.8/site-packages/kube_hunter/modules/report/plain.py", line 30, in get_report │
│ output += self.nodes_table() │
│ File "/usr/local/lib/python3.8/site-packages/kube_hunter/modules/report/plain.py", line 62, in nodes_table │
│ nodes_ret = f"\nNodes\n{nodes_table}\n" │
│ File "/usr/local/lib/python3.8/site-packages/prettytable/prettytable.py", line 327, in __str__ │
│ return self.get_string() │
│ File "/usr/local/lib/python3.8/site-packages/prettytable/prettytable.py", line 1700, in get_string │
│ rows = self._get_rows(options) │
│ File "/usr/local/lib/python3.8/site-packages/prettytable/prettytable.py", line 1618, in _get_rows │
│ rows.sort(reverse=options["reversesort"], key=options["sort_key"]) │
│ TypeError: '<' not supported between instances of 'str' and 'IPAddress' │
│ 2023-09-23 07:13:24,412 DEBUG kube_hunter.__main__ Cleaned Queue
What happened
The following error happens when running kube-hunter with the following definition on k8s v1.21.2:
Expected behavior
The table should be generated and printed to stdout