[BUG] K8S - Controller throws exception on reading ingress event in Managed K8S

[Jobuen]

What happened?

Hi there!

I've run into an interesting problem when trying to deploy Bunkerweb in a managed kubernetes environment from the Stackit Cloud Provider.

I've used the same deployment in a different managed kubernetes environment (Ionos) and there it worked well.

The problem is that the controller-pod throws an exception when processing an Ingress event. I'm not sure if the problem is found in/can be fixed in Bunkerweb or Stackit, but I thought I might as well let you know.

Thanks and best wishes

How to reproduce?

Deploy Bunkerweb as an ingress controller in a Stackit Managed Kubernetes Environment

Configuration file(s) (yaml or .env)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: bunkerweb
  namespace: "bunkerweb"
  labels:
    app.kubernetes.io/instance: bunkerweb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: bunkerweb
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: bunkerweb-1.0.2
automountServiceAccountToken: true
---
apiVersion: v1
kind: Secret
metadata:
  name: bunkerweb-db
  namespace: "bunkerweb"
  labels:
    app.kubernetes.io/instance: bunkerweb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: bunkerweb
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: bunkerweb-1.0.2
type: Opaque
data:
  mariadb-root-password: XXX
  mariadb-password: XXX
---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: 'yes'
  name: bunkerweb
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 10Gi
  hostPath:
    path: /
  persistentVolumeReclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bunkerweb-pvc
  namespace: bunkerweb
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: bunkerweb
rules:
  - apiGroups: [""]
    resources: ["services", "pods", "configmaps", "secrets"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get", "watch", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: bunkerweb
  labels:
    app.kubernetes.io/instance: bunkerweb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: bunkerweb
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: bunkerweb-1.0.2
    app.kubernetes.io/part-of: bunkerweb
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: bunkerweb
subjects:
  - kind: ServiceAccount
    name: bunkerweb
    namespace: "bunkerweb"
---
apiVersion: v1
kind: Service
metadata:
  name: bunkerweb-db-svc
  namespace: bunkerweb
spec:
  type: ClusterIP
  selector:
    app: bunkerweb-db
  ports:
    - name: sql
      protocol: TCP
      port: 3306
      targetPort: 3306
---
apiVersion: v1
kind: Service
metadata:
  name: bunkerweb-redis-svc
  namespace: bunkerweb
spec:
  type: ClusterIP
  selector:
      app.kubernetes.io/instance: bunkerweb
      app.kubernetes.io/name: bunkerweb
      app.kubernetes.io/component: bunkerweb-redis
      app.kubernetes.io/part-of: bunkerweb
  ports:
    - name: redis
      protocol: TCP
      port: 6379
      targetPort: 6379
---
apiVersion: v1
kind: Service
metadata:
  namespace: bunkerweb
  name: svc-app
spec:
  selector:
    app: app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bunkerweb-controller
  namespace: "bunkerweb"
  labels:
    app.kubernetes.io/instance: bunkerweb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: bunkerweb
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: bunkerweb-1.0.2
    app.kubernetes.io/component: bunkerweb-controller
    app.kubernetes.io/part-of: bunkerweb
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/instance: bunkerweb
      app.kubernetes.io/name: bunkerweb
      app.kubernetes.io/component: bunkerweb-controller
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: bunkerweb
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: bunkerweb
        app.kubernetes.io/version: 1.0.0
        helm.sh/chart: bunkerweb-1.0.2
        app.kubernetes.io/component: bunkerweb-controller
    spec:
      serviceAccountName: bunkerweb
      affinity:
        podAffinity:

        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/instance: bunkerweb
                    app.kubernetes.io/name: bunkerweb
                    app.kubernetes.io/component: bunkerweb-controller
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:

      containers:
        - name: bunkerweb-controller
          image: bunkerity/bunkerweb-autoconf:1.5.5
          imagePullPolicy: IfNotPresent
          env:
            - name: KUBERNETES_MODE
              value: "yes"
            - name: DB_USER
              value: test
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: bunkerweb-db
                  key: mariadb-password
            - name: "DATABASE_URI"
              value: mariadb+pymysql://$(DB_USER):$(DB_PASSWORD)@bunkerweb-db-svc.bunkerweb.svc.cluster.local:3306/db
          envFrom:
          resources:
            limits:
              cpu: 100m
              memory: 500Mi
            requests:
              cpu: 50m
              memory: 100Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bunkerweb-db
  namespace: bunkerweb
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: bunkerweb-db
  template:
    metadata:
      labels:
        app: bunkerweb-db
    spec:
      affinity:
        podAffinity:

        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/instance: bunkerweb
                    app.kubernetes.io/name: bunkerweb
                    app.kubernetes.io/component: bunkerweb-redis
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:

      containers:
        - name: bunkerweb-db
          image: mariadb:10.10
          imagePullPolicy: IfNotPresent
          securityContext:
          env:
            - name: MARIADB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: bunkerweb-db
                  key: mariadb-root-password
            - name: "MYSQL_USER"
              value: "test"
            - name: "MYSQL_PASSWORD"
              valueFrom:
                secretKeyRef:
                  name: bunkerweb-db
                  key: mariadb-password
            - name: "MYSQL_DATABASE"
              value: "db"
          volumeMounts:
            - mountPath: "/var/lib/mysql"
              name: vol-db
      volumes:
        - name: vol-db
          persistentVolumeClaim:
            claimName: bunkerweb-pvc
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bunkerweb
  namespace: "bunkerweb"
  labels:
    app.kubernetes.io/instance: bunkerweb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: bunkerweb
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: bunkerweb-1.0.2  
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/instance: bunkerweb
      app.kubernetes.io/name: bunkerweb
      app.kubernetes.io/component: ingress-controller-WAF
  template:
    metadata:
      annotations:
        bunkerweb.io/INSTANCE: "yes"
      labels:
        app.kubernetes.io/instance: bunkerweb
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: bunkerweb
        app.kubernetes.io/version: 1.0.0
        helm.sh/chart: bunkerweb-1.0.2
        app.kubernetes.io/component: ingress-controller-WAF
        app.kubernetes.io/part-of: bunkerweb  
    spec:
      affinity:
        podAffinity:

        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/instance: bunkerweb
                    app.kubernetes.io/name: bunkerweb
                    app.kubernetes.io/component: ingress-controller-WAF
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:

      containers:
        - name: bunkerweb
          image: bunkerity/bunkerweb:1.5.5
          imagePullPolicy: IfNotPresent
          securityContext:
            runAsUser: 101
            runAsGroup: 101
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
          ports:
            - name: http
              containerPort: 8080
            - name: https
              containerPort: 8443
          env:
          - name: LOG_LEVEL
            value: "debug"
          - name: KUBERNETES_MODE
            value: "yes"
          - name: USE_MODSECURITY
            value: "yes"
          - name: USE_MODSECURITY_CRS
            value: "yes"
          - name: MULTISITE
            value: "yes"
          - name: USE_REVERSE_PROXY
            value: "yes"
          - name: USE_API
            value: "yes"
          - name: USE_REDIS
            value: "yes"
          - name: DNS_RESOLVERS
            value:  
          - name: API_WHITELIST_IP
            value: 
          - name: REDIS_HOST
            value: bunkerweb-redis-svc.bunkerweb.svc.cluster.local
          - name: AUTOCONF_MODE
            value: "yes"
          envFrom:
          resources:
            limits:
              cpu: 500m
              memory: 1000Mi
            requests:
              cpu: 200m
              memory: 200Mi
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
            exec:
              command:
                - /usr/share/bunkerweb/helpers/healthcheck.sh
          readinessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 1
            successThreshold: 1
            timeoutSeconds: 1
            exec:
              command:
                - /usr/share/bunkerweb/helpers/healthcheck.sh
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bunkerweb-redis
  namespace: "bunkerweb"  
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/instance: bunkerweb
      app.kubernetes.io/name: bunkerweb
      app.kubernetes.io/component: bunkerweb-redis
      app.kubernetes.io/part-of: bunkerweb
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: bunkerweb
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: bunkerweb
        app.kubernetes.io/version: 1.0.0
        helm.sh/chart: bunkerweb-1.0.2
        app.kubernetes.io/component: bunkerweb-redis
        app.kubernetes.io/part-of: bunkerweb
    spec:
      affinity:
        podAffinity:

        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/instance: bunkerweb
                    app.kubernetes.io/name: bunkerweb
                    app.kubernetes.io/component: bunkerweb-redis
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:

      containers:
        - name: bunkerweb-redis
          image: redis:7-alpine
          imagePullPolicy: IfNotPresent
          securityContext:
          envFrom:
          resources:
            limits:
              cpu: 250m
              memory: 750Mi
            requests:
              cpu: 5m
              memory: 100Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bunkerweb-scheduler
  namespace: "bunkerweb"
  labels:
    app.kubernetes.io/instance: bunkerweb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: bunkerweb
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: bunkerweb-1.0.2
    app.kubernetes.io/component: bunkerweb-scheduler
    app.kubernetes.io/part-of: bunkerweb
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/instance: bunkerweb
      app.kubernetes.io/name: bunkerweb
      app.kubernetes.io/component: bunkerweb-scheduler
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: bunkerweb
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: bunkerweb
        app.kubernetes.io/version: 1.0.0
        helm.sh/chart: bunkerweb-1.0.2
        app.kubernetes.io/component: bunkerweb-scheduler
    spec:
      serviceAccountName: bunkerweb
      affinity:
        podAffinity:

        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/instance: bunkerweb
                    app.kubernetes.io/name: bunkerweb
                    app.kubernetes.io/component: bunkerweb-scheduler
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:

      containers:
        - name: bunkerweb-scheduler
          image: bunkerity/bunkerweb-scheduler:1.5.5
          imagePullPolicy: IfNotPresent
          env:
            - name: KUBERNETES_MODE
              value: "yes"
            - name: DB_USER
              value: 
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: bunkerweb-db
                  key: mariadb-password
            - name: "DATABASE_URI"
              value: mariadb+pymysql://$(DB_USER):$(DB_PASSWORD)@bunkerweb-db-svc.bunkerweb.svc.cluster.local:3306/db
          envFrom:
          resources:
            limits:
              cpu: 250m
              memory: 750Mi
            requests:
              cpu: 50m
              memory: 100Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: bunkerweb
  labels:
    app: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "app"
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
      - name: app
        image: tutum/hello-world
        ports:
        - containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: bunkerweb-settings
  namespace: "bunkerweb"
  labels:
    app.kubernetes.io/instance: bunkerweb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: bunkerweb
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: bunkerweb-1.0.2
  annotations:
    bunkerweb.io/AUTO_REDIRECT_HTTP_TO_HTTPS: "yes"
    bunkerweb.io/LISTEN_HTTP: "yes"
    bunkerweb.io/REDIRECT_HTTP_TO_HTTPS: "no"
    bunkerweb.io/SEND_ANONYMOUS_REPORTS: "no"
    bunkerweb.io/USE_BAD_BEHAVIOUR: "no"
    bunkerweb.io/USE_BUNKERNET: "no"
    bunkerweb.io/USE_LIMIT_REQ: "no"
    bunkerweb.io/AUTO_LETS_ENCRYPT: "yes"
    bunkerweb.io/EMAIL_LETS_ENCRYT: "example.email@dummy.com" 
    bunkerweb.io/USE_LETS_ENCRYPT_STAGING: "yes"
    bunkerweb.io/empty.ingress_AUTO_LETS_ENCRYPT: "no"

spec:
  rules:
  - host: empty.ingress
    http:
      paths:
        - path: /
          pathType: Exact
          backend:
            service:
              name: bunkerweb
              port: 
                number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    bunkerweb.io/???_SERVER_NAME: ???
  name: ???
  namespace: bunkerweb
spec:
  rules:
    - host: ???
      http:
        paths:
          - backend:
              service:
                name: svc-app
                port:
                  number: 80
            path: /
            pathType: Prefix
---
apiVersion: v1
kind: Service
metadata:
  name: bunkerweb
  namespace: "bunkerweb"
  labels:
    app.kubernetes.io/instance: bunkerweb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: bunkerweb
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: bunkerweb-1.0.2
    app.kubernetes.io/component: ingress-controller-WAF
    app.kubernetes.io/part-of: bunkerweb
  annotations:
    "helm.sh/hook": post-install
spec:
  type: LoadBalancer
  sessionAffinity: None
  externalTrafficPolicy: "Cluster"
  loadBalancerIP: ???
  ports:
    - name: http
      port: 80
      targetPort: 8080
      protocol: TCP
    - name: https
      port: 443
      targetPort: 8443
      protocol: TCP
  selector:
    app.kubernetes.io/instance: bunkerweb
    app.kubernetes.io/name: bunkerweb
    app.kubernetes.io/component: ingress-controller-WAF

Relevant log output

[2024-03-22 09:32:16] - KUBERNETES-CONTROLLER - ❌ - Unknown exception while reading k8s event (type = ingress) :
Traceback (most recent call last):
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 761, in _update_chunk_length
    self.chunk_left = int(line, 16)
                      ^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 444, in _error_catcher
    yield
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 828, in read_chunked
    self._update_chunk_length()
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 765, in _update_chunk_length
    raise InvalidChunkLength(self, line)
urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/bunkerweb/autoconf/IngressController.py", line 265, in __watch
    for event in w.stream(what):
  File "/usr/share/bunkerweb/deps/python/kubernetes/watch/watch.py", line 165, in stream
    for line in iter_resp_lines(resp):
  File "/usr/share/bunkerweb/deps/python/kubernetes/watch/watch.py", line 56, in iter_resp_lines
    for seg in resp.stream(amt=None, decode_content=False):
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 624, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 816, in read_chunked
    with self._error_catcher():
  File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 461, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

[2024-03-22 09:32:16] - KUBERNETES-CONTROLLER - ⚠️  - Got exception, retrying in 10 seconds ...
[2024-03-22 09:32:17] - KUBERNETES-CONTROLLER - ❌ - Unknown exception while reading k8s event (type = service) :
Traceback (most recent call last):
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 761, in _update_chunk_length
    self.chunk_left = int(line, 16)
                      ^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 444, in _error_catcher
    yield
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 828, in read_chunked
    self._update_chunk_length()
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 765, in _update_chunk_length
    raise InvalidChunkLength(self, line)
urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/bunkerweb/autoconf/IngressController.py", line 265, in __watch
    for event in w.stream(what):
  File "/usr/share/bunkerweb/deps/python/kubernetes/watch/watch.py", line 165, in stream
    for line in iter_resp_lines(resp):
  File "/usr/share/bunkerweb/deps/python/kubernetes/watch/watch.py", line 56, in iter_resp_lines
    for seg in resp.stream(amt=None, decode_content=False):
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 624, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 816, in read_chunked
    with self._error_catcher():
  File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/usr/share/bunkerweb/deps/python/urllib3/response.py", line 461, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

[2024-03-22 09:32:17] - KUBERNETES-CONTROLLER - ⚠️  - Got exception, retrying in 10 seconds ...

BunkerWeb version

1.5.5

What integration are you using?

Kubernetes

Linux distribution (if applicable)

No response

Removed private data

• [X] I have removed all private data from the configuration file and the logs

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[fl0ppy-d1sk]

Hello @Jobuen,

Thanks for the feedback.

Can you tell me which version of k8s do you have installed please ? both ionos and stackit

[Jobuen]

Of course, stackit is 1.27.11 and ionos 1.28.7