Enabling Mutating/Validation webhook for a CRD makes apiserver to throw 500 internal server error

On Enabling Mutating/Validation Webhook (using Kubebuilder 2.3.2) for a CRD, apiserver throws internal server error.

Expectation was, apiserver would forward the request to the webhook.

Using this self-signed cert and issuer and webhooks:

---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: myproject-serving-cert
  namespace: myproject
spec:
  dnsNames:
  - myproject-webhook-service.myproject.svc
  - myproject-webhook-service.myproject.svc.cluster.local
  issuerRef:
    kind: Issuer
    name: myproject-selfsigned-issuer
  secretName: webhook-server-cert
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: myproject-selfsigned-issuer
  namespace: myproject
spec:
  selfSigned: {}
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from: myproject/myproject-serving-cert
  name: myproject-mutating-webhook-configuration
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    service:
      name: myproject-webhook-service
      namespace: myproject
      path: /mutate-mw-mycompany-com-v1alpha1-server
  failurePolicy: Fail
  name: mserver.kb.io
  rules:
  - apiGroups:
    - mw.mycompany.com
    apiVersions:
    - v1alpha1
    operations:
    - CREATE
    - UPDATE
    resources:
    - servers
  sideEffects: None
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from: myproject/myproject-serving-cert
  name: myproject-validating-webhook-configuration
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    service:
      name: myproject-webhook-service
      namespace: myproject
      path: /validate-mw-mycompany-com-v1alpha1-server
  failurePolicy: Fail
  name: vserver.kb.io
  rules:
  - apiGroups:
    - mw.mycompany.com
    apiVersions:
    - v1alpha1
    operations:
    - CREATE
    - UPDATE
    - DELETE
    resources:
    - servers
  sideEffects: None

The debug log from kubernetes apiserver:

W0409 18:36:48.431725       1 dispatcher.go:182] Failed calling webhook, failing closed mserver.kb.io: failed calling webhook "mserver.kb.io": expected response.uid="fd42a853-695c-4616-b38d-e0ea8a222b46", got ""
I0409 18:36:48.432233       1 httplog.go:89] "HTTP" verb="POST" URI="/apis/mw.mycompany.com/v1alpha1/namespaces/myproject/servers?fieldManager=kubectl-create" latency="4.3971ms" userAgent="kubectl/v1.19.7 (linux/amd64) kubernetes/1dd5338" srcIP="172.18.0.1:58066" resp=500 statusStack="
cggoroutine 67063 [running]:\rcgk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc01f767570, 0x1f4)\r\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:237 +0xcf
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc01f767570, 0x1f4)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:216 +0x35
gk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc01f773d20, 0x1f4)
g\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:226 +0xb2
gk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc01fa31710, 0x1f4)
g\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:571 +0x45
gk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.(*deferredResponseWriter).Write(0xc006d90e60, 0xc01b000000, 0x199, 0x2a2df, 0x0, 0x0, 0x0)
g\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:204 +0x1f7
gencoding/json.(*Encoder).Encode(0xc01fe96538, 0x2920ea0, 0xc01fe341e0, 0x0, 0x40f99b)
g\t/usr/local/go/src/encoding/json/stream.go:231 +0x1cb
gk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).doEncode(0xc006d90c30, 0x2ea32a0, 0xc01fe341e0, 0x2e94000, 0xc006d90e60, 0x0, 0x0)
g\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:327 +0x2e9
gk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Encode(0xc006d90c30, 0x2ea32a0, 0xc01fe341e0, 0x2e94000, 0xc006d90e60, 0x242053a, 0x6)
g\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:301 +0x169
gk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).doEncode(0xc01fe34280, 0x2ea32a0, 0xc01fe341e0, 0x2e94000, 0xc006d90e60, 0x0, 0x0)
g\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:228 +0x396
gk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).Encode(0xc01fe34280, 0x2ea32a0, 0xc01fe341e0, 0x2e94000, 0xc006d90e60, 0x29abca1, 0x17)
g\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:184 +0x170
gk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x299d749, 0x10, 0x7f757efc14c8, 0xc01fe34280, 0x2ee9be0, 0xc01cbd9e40, 0xc01fa2fe00, 0x1f4, 0x2ea32a0, 0xc01fe341e0)
g\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:96 +0x12c
gk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x2eecc20, 0xc016bcd3e0, 0x2eed160, 0x422ffa8, 0xc00a089930, 0xb, 0xc00a089990, 0x8, 0x2ee9be0, 0xc01cbd9e40, ...)
g\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:253 +0x572
gk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x2e93860, 0xc01fe34140, 0x2eecc20, 0xc016bcd3e0, 0xc00a089930, 0xb, 0xc00a089990, 0x8, 0x2ee9be0, 0xc01cbd9e40, ...)
g\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:272 +0x16f
gk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err(...)
g\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:103
gk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.createHandler.func1(0x2ee9be0, 0xc01cbd9e40, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:187 +0x1a85
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentHandlerFunc.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:474 +0x2ba
net/http.HandlerFunc.ServeHTTP(0xc01f745900, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:59 +0x137
net/http.HandlerFunc.ServeHTTP(0xc01fa316e0, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.(*crdHandler).ServeHTTP(0xc00096e240, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:380 +0x867
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc006813100, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x47a
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc000350c40, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8c
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x29aba90, 0x17, 0xc00015bb90, 0xc000350c40, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x87f
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc00a0fdcc0, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x23d
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc00092e1c0, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8c
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x2999e93, 0xe, 0xc000774120, 0xc00092e1c0, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x87f
k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.(*proxyHandler).ServeHTTP(0xc004cd3c20, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:121 +0x183
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc00553d0c0, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x47a
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc006f1e930, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8c
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x299be0e, 0xf, 0xc002cdc240, 0xc006f1e930, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x87f
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:95 +0x165
net/http.HandlerFunc.ServeHTTP(0xc006f54de0, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x59a
net/http.HandlerFunc.ServeHTTP(0xc006efff00, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:71 +0x186
net/http.HandlerFunc.ServeHTTP(0xc006efff40, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:95 +0x165
net/http.HandlerFunc.ServeHTTP(0xc006f54e10, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fe00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithPriorityAndFairness.func1.4()
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/priority-and-fairness.go:127 +0x3c6
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flowcontrol.(*configController).Handle.func2()
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flowcontrol/apf_filter.go:133 +0x1aa
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flowcontrol.immediateRequest.Finish(...)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flowcontrol/apf_controller.go:660
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flowcontrol.(*configController).Handle(0xc000334840, 0x2eef6e0, 0xc01fa31530, 0xc01f740a50, 0x2eeff60, 0xc01f74b6c0, 0xc01f771720, 0xc01f771730, 0xc01fa10d20)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flowcontrol/apf_filter.go:123 +0x86a
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithPriorityAndFairness.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/priority-and-fairness.go:130 +0x5c3
net/http.HandlerFunc.ServeHTTP(0xc006f54e40, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:71 +0x186
net/http.HandlerFunc.ServeHTTP(0xc006efff80, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:95 +0x165
net/http.HandlerFunc.ServeHTTP(0xc006f54e70, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x23dd
net/http.HandlerFunc.ServeHTTP(0xc006efffc0, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:71 +0x186
net/http.HandlerFunc.ServeHTTP(0xc006f62000, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:95 +0x165
net/http.HandlerFunc.ServeHTTP(0xc006f54ea0, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:71 +0x186
net/http.HandlerFunc.ServeHTTP(0xc006f62040, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:95 +0x165
net/http.HandlerFunc.ServeHTTP(0xc006f54f00, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fd00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fb00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:70 +0x6d2
net/http.HandlerFunc.ServeHTTP(0xc006ed7f40, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fb00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fa00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:80 +0x38a
net/http.HandlerFunc.ServeHTTP(0xc006f62080, 0x7f757e9f8be8, 0xc01cbd9e30, 0xc01fa2fa00)
\t/usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc01f727bc0, 0xc006f56740, 0x2ef0060, 0xc01cbd9e30, 0xc01fa2fa00)
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:111 +0xb8
created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:97 +0x1cc
" addedInfo="
logging error output: \"{\\\"kind\\\":\\\"Status\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"metadata\\\":{},\\\"status\\\":\\\"Failure\\\",\\\"message\\\":\\\"Internal error occurred: failed calling webhook \\\\\\\"mserver.kb.io\\\\\\\": expected response.uid=\\\\\\\"fd42a853-695c-4616-b38d-e0ea8a222b46\\\\\\\", got \\\\\\\"\\\\\\\"\\\",\\\"reason\\\":\\\"InternalError\\\",\\\"details\\\":{\\\"causes\\\":[{\\\"message\\\":\\\"failed calling webhook \\\\\\\"mserver.kb.io\\\\\\\": expected response.uid=\\\\\\\"fd42a853-695c-4616-b38d-e0ea8a222b46\\\\\\\", got \\\\\\\"\\\\\\\"\\\"}]},\\\"code\\\":500}\
\"

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-21T01:11:42Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration:


Deployed kubernetes using kind
Patched API server with:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:

role: control-plane kubeadmConfigPatches:
- | kind: ClusterConfiguration apiServer: extraArgs: enable-admission-plugins: NodeRestrictionNamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,PersistentVolumeClaimResize,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota v: "2"

- OS (e.g: `cat /etc/os-release`):

NAME="Ubuntu" VERSION="20.04.1 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.1 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal

- Kernel (e.g. `uname -a`):

Linux W10HHQGN13 4.19.128-microsoft-standard #1 SMP Tue Jun 23 12:58:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

kubernetes / kubernetes

Enabling Mutating/Validation webhook for a CRD makes apiserver to throw 500 internal server error #100982

Environment: