search

kyverno / kyverno

Kubernetes Native Policy Management
https://kyverno.io
Apache License 2.0
5.31k stars 798 forks source link

[Bug] VAP feature not working with admissionregistration.k8s.io/v1beta1 #10558

Closed fafarun closed 1 day ago

fafarun commented 5 days ago

Kyverno Version

1.12.4

Description

Hi,

K8s version : 1.28.10 (v1.28.10+rke2r1) Kyverno helm chart : 3.2.5 Kyverno version : 1.12.4

I tried to use the feature VAP with Kyverno. with the - "runtime-config=admissionregistration.k8s.io/v1alpha1" (k8s 1.26.14) : it's OK .

But when I tried with a k8s 1.28.10, so with the - "runtime-config=admissionregistration.k8s.io/v1beta1", kyverno return an error : 

│ kyverno 2024-06-27T13:29:14Z    INFO    config    config/config.go:460    webhookLabels not set    {"name": "kyverno", "namespace": "kyverno"}                                                                      │
│ kyverno 2024-06-27T13:29:14Z    INFO    config    config/config.go:474    matchConditions not set    {"name": "kyverno", "namespace": "kyverno"}                                                                    │
│ kyverno 2024-06-27T13:29:14Z    INFO    setup.registry-client    internal/registry.go:18    setup registry client...    {"secrets": "", "insecure": false}                                                          │
│ kyverno 2024-06-27T13:29:14Z    DEBUG    config-controller    controller/run.go:58    starting ...                                                                                                                  │
│ kyverno 2024-06-27T13:29:14Z    DEBUG    config-controller.worker    controller/run.go:71    starting worker    {"id": 0}                                                                                           │
│ kyverno 2024-06-27T13:29:14Z    LEVEL(-2)    klog    cache/reflector.go:351    Caches populated for *v1.Secret from k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229                                           │
│ kyverno 2024-06-27T13:29:14Z    INFO    setup.image-verify-cache    internal/imageverifycache.go:10    setup image verify cache...    {"enabled": true, "maxsize": 1000, "ttl": "1h0m0s"}                           │
│ kyverno 2024-06-27T13:29:14Z    INFO    setup.kube-client    internal/client.go:43    create kube client...    {"kubeconfig": "", "qps": 20, "burst": 50}                                                           │
│ kyverno 2024-06-27T13:29:14Z    INFO    setup.kyverno-client    internal/client.go:51    create kyverno client...    {"kubeconfig": "", "qps": 20, "burst": 50}                                                     │
│ kyverno 2024-06-27T13:29:14Z    INFO    setup.dynamic-client    internal/client.go:59    create dynamic client...    {"kubeconfig": "", "qps": 20, "burst": 50}                                                     │
│ kyverno 2024-06-27T13:29:14Z    INFO    setup.apiserver-client    internal/client.go:75    create apiserver client...    {"kubeconfig": "", "qps": 20, "burst": 50}                                                 │
│ kyverno 2024-06-27T13:29:14Z    INFO    setup.d-client    internal/client.go:83    create the kyverno dynamic client...    {"kubeconfig": "", "qps": 20, "burst": 50}                                               │
│ kyverno 2024-06-27T13:29:14Z    INFO    setup.events-client    internal/client.go:91    create the events client...    {"kubeconfig": "", "qps": 20, "burst": 50}                                                   │
│ kyverno 2024-06-27T13:29:14Z    INFO    setup.events-client.kube-client    internal/client.go:43    create kube client...    {"kubeconfig": "", "qps": 20, "burst": 50}                                             │
│ kyverno 2024-06-27T13:29:14Z    ERROR    setup    kyverno/main.go:315    validating admission policies aren't supported.    {"error": "the server could not find the requested resource"}

My configuration file : kube-api configuration : 

Args:                                                                                                                                                                                                           │
│       --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml                                                                                                                                               │
│       --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml                                                                                                                                                       │
│       --audit-log-maxage=30                                                                                                                                                                                         │
│       --audit-log-maxbackup=10                                                                                                                                                                                      │
│       --audit-log-maxsize=100                                                                                                                                                                                       │
│       --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log                                                                                                                                                  │
│       --allow-privileged=true                                                                                                                                                                                       │
│       --anonymous-auth=false                                                                                                                                                                                        │
│       --api-audiences=https://kubernetes.default.svc.cluster.local,rke2                                                                                                                                             │
│       --authorization-mode=Node,RBAC                                                                                                                                                                                │
│       --bind-address=0.0.0.0                                                                                                                                                                                        │
│       --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs                                                                                                                                                   │
│       --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt                                                                                                                                               │
│       --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml                                                                                                                    │
│       --enable-admission-plugins=NodeRestriction                                                                                                                                                                    │
│       --enable-aggregator-routing=true                                                                                                                                                                              │
│       --enable-bootstrap-token-auth=true                                                                                                                                                                            │
│       --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json                                                                                                                         │
│       --encryption-provider-config-automatic-reload=true                                                                                                                                                            │
│       --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt                                                                                                                                             │
│       --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt                                                                                                                                              │
│       --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key                                                                                                                                               │
│       --etcd-servers=https://127.0.0.1:2379                                                                                                                                                                         │
│       --feature-gates=ValidatingAdmissionPolicy=true                                                                                                                                                                │
│       --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt                                                                                                                                │
│       --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt                                                                                                                       │
│       --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key                                                                                                                               │
│       --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname                                                                                                                                              │
│       --profiling=false                                                                                                                                                                                             │
│       --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt                                                                                                                               │
│       --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key                                                                                                                                │
│       --requestheader-allowed-names=system:auth-proxy                                                                                                                                                               │
│       --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt                                                                                                                         │
│       --requestheader-extra-headers-prefix=X-Remote-Extra-                                                                                                                                                          │
│       --requestheader-group-headers=X-Remote-Group                                                                                                                                                                  │
│       --requestheader-username-headers=X-Remote-User                                                                                                                                                                │
│       --runtime-config=admissionregistration.k8s.io/v1beta1                                                                                                                                                         │
│       --secure-port=6443                                                                                                                                                                                            │
│       --service-account-issuer=https://kubernetes.default.svc.cluster.local                                                                                                                                         │
│       --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key                                                                                                                                       │
│       --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key                                                                                                                       │
│       --service-cluster-ip-range=10.43.0.0/16                                                                                                                                                                       │
│       --service-node-port-range=30000-32767                                                                                                                                                                         │
│       --storage-backend=etcd3                                                                                                                                                                                       │
│       --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt                                                                                                                                   │
│       --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20 │
│ _POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305                                                                                                                                                                      │
│       --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key

Kyverno chart configuration : 

# https://github.com/kyverno/kyverno/issues/6078
apiVersionOverride:
  podDisruptionBudget: policy/v1
admissionController:
  replicas: 1
  serviceMonitor:
    enabled: true
  tracing:
    enabled: true
    address: opentelemetry-collector.opentelemetry
    port: 4317
  rbac:
validatingadmissionpolicybindings
    clusterRole:
      extraResources:
      - apiGroups:
        - admissionregistration.k8s.io
        resources:
        - validatingadmissionpolicies
        - validatingadmissionpolicybindings
        verbs:
        - create
        - update
        - delete
        - list
backgroundController:
  replicas: 1
  serviceMonitor:
    enabled: true
  tracing:
    enabled: true
    address: opentelemetry-collector.opentelemetry
    port: 4317
cleanupController:
  replicas: 1
  serviceMonitor:
    enabled: true
  tracing:
    enabled: true
    address: opentelemetry-collector.opentelemetry
    port: 4317
reportsController:
  replicas: 1
  serviceMonitor:
    enabled: true
  tracing:
    enabled: true
    address: opentelemetry-collector.opentelemetry
    port: 4317
features:
  generateValidatingAdmissionPolicy:
    enabled: true

Slack discussion

No response

Troubleshooting

welcome[bot] commented 5 days ago

Thanks for opening your first issue here! Be sure to follow the issue template!

MariamFahmy98 commented 4 days ago

For k8s 1.28, you have to enable both versions; admissionregistration.k8s.io/v1beta1 and admissionregistration.k8s.io/v1alpha1 because Kyverno still uses v1alpha1 version in the codebase. Let me know if this solves the issue.

fafarun commented 1 day ago

Thx. It solved my issue. The strange think, when I execute kubectl api-resources | grep admin I still not watch the alpha api, only the beta. But I can deploy kyverno.