[Bug] Kyverno does not prevent creation of Services with type LoadBalancer

cybra-northco commented 3 months ago

Kyverno Version

1.12.4

Kubernetes Version

1.29.x

Kubernetes Platform

EKS

Kyverno Rule Type

Validate

Description

Kyverno was installed using the following values file:

admissionController:
  replicas: 3
  tolerations:
  - effect: NoSchedule
    key: dedicated
    operator: Equal
    value: admin
backgroundController:
  tolerations:
  - effect: NoSchedule
    key: dedicated
    operator: Equal
    value: admin
cleanupController:
  tolerations:
  - effect: NoSchedule
    key: dedicated
    operator: Equal
    value: admin
cleanupJobs:
  admissionReports:
    tolerations:
    - effect: NoSchedule
      key: dedicated
      operator: Equal
      value: admin
  clusterAdmissionReports:
    tolerations:
    - effect: NoSchedule
      key: dedicated
      operator: Equal
      value: admin
  clusterEphemeralReports:
    tolerations:
    - effect: NoSchedule
      key: dedicated
      operator: Equal
      value: admin
  ephemeralReports:
    tolerations:
    - effect: NoSchedule
      key: dedicated
      operator: Equal
      value: admin
  updateRequests:
    tolerations:
    - effect: NoSchedule
      key: dedicated
      operator: Equal
      value: admin
crds:
  migration:
    tolerations:
    - effect: NoSchedule
      key: dedicated
      operator: Equal
      value: admin
policyReportsCleanup:
  tolerations:
  - effect: NoSchedule
    key: dedicated
    operator: Equal
    value: admin
reportsController:
  resources:
    limits:
      cpu: 1500m
      memory: 1024Mi
    requests:
      cpu: 384m
      memory: 512Mi
  tolerations:
  - effect: NoSchedule
    key: dedicated
    operator: Equal
    value: admin

In spite of the following policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  annotations:
    policies.kyverno.io/category: Pod Security Standards (Baseline)
    policies.kyverno.io/description: 'This policy restricts use of the Service type LoadBalancer.'
    policies.kyverno.io/minversion: 1.6.0
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Service
    policies.kyverno.io/title: Disallow Service Type LoadBalancer
  labels:
  name: ok-disallow-load-balancer
spec:
  admission: true
  background: true
  rules:
  - match:
      any:
      - resources:
          kinds:
          - Service
          namespaceSelector:
            matchExpressions:
            - key: bloomreach.com/team
              operator: In
              values:
              - pulsar
              - merchandising
              - analytics
              - delivery
              - brc
    name: validate-no-loadbalancer
    skipBackgroundRequests: true
    validate:
      message: Service of type LoadBalancer is not allowed.
      pattern:
        spec:
          type: '!LoadBalancer'
  validationFailureAction: Enforce

It is possible to apply the following yaml:

apiVersion: v1
kind: Service
metadata:
  name: lb-d
  namespace: default
spec:
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: garuda-example
  sessionAffinity: None
  type: LoadBalancer

When trying this on a real EKS cluster, the request is not denied and there are no logs in the kyverno admission controller. The same setup in the local Kind cluster results in the request to create a service to be denied, and there are logs in the kyverno admission controller that resource creation was denied due to a policy violation.

Steps to reproduce

Install kyverno using the aforementioned values
Apply the aforementioned policy yaml
Apply the aforementioned service yaml
The service is successfully created and assigned an external IP

Expected behavior

Kyverno rejects the creation of the k8s service

Screenshots

No response

Kyverno logs

No response

Slack discussion

No response

Troubleshooting

[X] I have read and followed the documentation AND the troubleshooting guide.
[X] I have searched other issues in this repository and mine is not recorded.

welcome[bot] commented 3 months ago

Thanks for opening your first issue here! Be sure to follow the issue template!

JimBugwadia commented 3 months ago

Are you using a custom CNI on EKS?

https://kyverno.io/docs/troubleshooting/#kyverno-fails-on-eks

cybra-northco commented 3 months ago

Yes, we are using VPC CNI, version v1.18.2-eksbuild.1. I have read the documentation that was linked, it mentions that sometimes the VPC CNI might be outdated. It seems like the version we are using is good enough for the kubernetes version we have.

Another concern mentioned in the doc - that some of the traffic from the control plane to the worker nodes might be blocked - seems not to be relevant, at least according to our SRE team:

the traffic is allowed on all ports within cluster , master and worker nodes

kyverno / kyverno