search

loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.3k stars 402 forks source link

Workload no longer works after upgrading vcluster 0.15.7 to 0.18.1 #1417

Closed felipecrs closed 7 months ago

felipecrs commented 9 months ago

What happened?

I have a CI/CD pipeline that creates a vcluster and deploys some workload to it. When I was using vcluster 0.15.7, everything worked fine. After upgrading to latest version, 0.18.1, it's no longer working. I also noticed the exact same issue at least with 0.17.1 and 0.18.0 as well. Possibly also 0.16.0.

Note that this issue is a little different from https://github.com/loft-sh/vcluster/issues/1285. There, most of my workloads stopped working after upgrading from 0.15.7 to 0.16.0, but started working again after 0.17.1.

This time, a specific workload/pipeline is not working.

This is the apparent failure in the pipeline, which happens during a given helm install we have:

E1213 17:13:48.153535     500 round_tripper.go:63] CancelRequest not implemented by *cli.retryingRoundTripper
[2023-12-13T17:26:07.401Z]   Error: failed post-install: unable to build kubernetes object for post-install hook hidden-chart-name/templates/hooklauncher-job-inventory-secret.yaml: error validating "": error validating data: the server was unable to return a response in the time allotted, but may still be processing the request
[2023-12-13T17:26:07.401Z]   helm.go:84: [debug] failed post-install: unable to build kubernetes object for post-install hook hidden-chart-name/templates/hooklauncher-job-inventory-secret.yaml: error validating "": error validating data: the server was unable to return a response in the time allotted, but may still be processing the request
Error: failed post-install: unable to build kubernetes object for post-install hook hidden-chart-name/templates/job-inventory-secret.yaml: error validating "": error validating data: Get "https://hidden.domain/openapi/v3?timeout=32s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
helm.go:84: [debug] failed post-install: unable to build kubernetes object for post-install hook hidden-chart-name/templates/job-inventory-secret.yaml: error validating "": error validating data: Get "https://hidden.domain/openapi/v3?timeout=32s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Listing the vcluster pods:

❯ k get pods
NAME                                                        READY   STATUS    RESTARTS      AGE
coredns-68559449b6-2gdpt-x-kube-system-x-vcluster           1/1     Running   0             29m
coredns-68559449b6-7fsgs-x-kube-system-x-vcluster           1/1     Running   0             29m
coredns-68559449b6-rnlh9-x-kube-system-x-vcluster           1/1     Running   0             29m
vcluster-7c55958845-8hsl7                                   1/1     Running   0             30m
vcluster-7c55958845-g7h2s                                   1/1     Running   0             30m
vcluster-7c55958845-lct7s                                   1/1     Running   0             30m
vcluster-api-748f6596f-88qgc                                1/1     Running   2 (30m ago)   30m
vcluster-api-748f6596f-b6tc4                                1/1     Running   1 (30m ago)   30m
vcluster-api-748f6596f-ct9gn                                1/1     Running   1 (30m ago)   30m
vcluster-controller-5f9c99d5f7-fp488                        1/1     Running   0             30m
vcluster-etcd-0                                             1/1     Running   0             30m
vcluster-etcd-1                                             1/1     Running   0             30m
vcluster-etcd-2                                             1/1     Running   0             30m

Checking logs for the vcluster-api-748f6596f-88qgc:

Click here to show ```console $ stern vcluster-api-748f6596f-88qgc --no-follow vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:48.767921 1 server.go:553] external host was not specified, using 192.168.163.48 vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:48.770733 1 server.go:166] Version: v1.27.6 vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:48.770792 1 server.go:168] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:49.485774 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,AlwaysPullImages,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:49.485823 1 plugins.go:161] Loaded 14 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,AlwaysPullImages,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,ClusterTrustBundleAttest,CertificateSubjectRestriction,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook,ResourceQuota. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:49.534817 1 handler.go:232] Adding GroupVersion apiextensions.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:49.534906 1 genericapiserver.go:752] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:49.536866 1 instance.go:282] Using reconciler: none vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:49.883357 1 handler.go:232] Adding GroupVersion v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:49.884061 1 instance.go:651] API group "internal.apiserver.k8s.io" is not enabled, skipping. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.463045 1 instance.go:651] API group "resource.k8s.io" is not enabled, skipping. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.480527 1 handler.go:232] Adding GroupVersion authentication.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.480600 1 genericapiserver.go:752] Skipping API authentication.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.480612 1 genericapiserver.go:752] Skipping API authentication.k8s.io/v1alpha1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.483767 1 handler.go:232] Adding GroupVersion authorization.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.483790 1 genericapiserver.go:752] Skipping API authorization.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.487322 1 handler.go:232] Adding GroupVersion autoscaling v2 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.488734 1 handler.go:232] Adding GroupVersion autoscaling v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.488761 1 genericapiserver.go:752] Skipping API autoscaling/v2beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.488768 1 genericapiserver.go:752] Skipping API autoscaling/v2beta2 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.492888 1 handler.go:232] Adding GroupVersion batch v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.492928 1 genericapiserver.go:752] Skipping API batch/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.496258 1 handler.go:232] Adding GroupVersion certificates.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.496292 1 genericapiserver.go:752] Skipping API certificates.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.496299 1 genericapiserver.go:752] Skipping API certificates.k8s.io/v1alpha1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.500067 1 handler.go:232] Adding GroupVersion coordination.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.500121 1 genericapiserver.go:752] Skipping API coordination.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.502063 1 genericapiserver.go:752] Skipping API discovery.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.502861 1 handler.go:232] Adding GroupVersion discovery.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.507386 1 handler.go:232] Adding GroupVersion networking.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.507420 1 genericapiserver.go:752] Skipping API networking.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.507428 1 genericapiserver.go:752] Skipping API networking.k8s.io/v1alpha1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.509828 1 handler.go:232] Adding GroupVersion node.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.509853 1 genericapiserver.go:752] Skipping API node.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.509860 1 genericapiserver.go:752] Skipping API node.k8s.io/v1alpha1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.512623 1 handler.go:232] Adding GroupVersion policy v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.512649 1 genericapiserver.go:752] Skipping API policy/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.517140 1 handler.go:232] Adding GroupVersion rbac.authorization.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.517180 1 genericapiserver.go:752] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.517202 1 genericapiserver.go:752] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.519703 1 handler.go:232] Adding GroupVersion scheduling.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.519735 1 genericapiserver.go:752] Skipping API scheduling.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.519742 1 genericapiserver.go:752] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.531435 1 handler.go:232] Adding GroupVersion storage.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.531654 1 genericapiserver.go:752] Skipping API storage.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.531674 1 genericapiserver.go:752] Skipping API storage.k8s.io/v1alpha1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.537922 1 handler.go:232] Adding GroupVersion flowcontrol.apiserver.k8s.io v1beta3 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.539610 1 handler.go:232] Adding GroupVersion flowcontrol.apiserver.k8s.io v1beta2 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.539638 1 genericapiserver.go:752] Skipping API flowcontrol.apiserver.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.539646 1 genericapiserver.go:752] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.548145 1 handler.go:232] Adding GroupVersion apps v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.548196 1 genericapiserver.go:752] Skipping API apps/v1beta2 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.548204 1 genericapiserver.go:752] Skipping API apps/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.552205 1 handler.go:232] Adding GroupVersion admissionregistration.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.552232 1 genericapiserver.go:752] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.552240 1 genericapiserver.go:752] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.555449 1 handler.go:232] Adding GroupVersion events.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.555477 1 genericapiserver.go:752] Skipping API events.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:50.604037 1 handler.go:232] Adding GroupVersion apiregistration.k8s.io v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 13:56:50.604092 1 genericapiserver.go:752] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.494988 1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/run/config/pki/ca.crt" vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.494982 1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/run/config/pki/front-proxy-ca.crt" vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.495977 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/run/config/pki/apiserver.crt::/run/config/pki/apiserver.key" vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.497312 1 secure_serving.go:210] Serving securely on [::]:6443 vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.497450 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.497582 1 available_controller.go:423] Starting AvailableConditionController vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.497640 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.497719 1 controller.go:83] Starting OpenAPI AggregationController vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.497606 1 aggregator.go:150] waiting for initial CRD sync... vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.497603 1 handler_discovery.go:412] Starting ResourceDiscoveryManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.497850 1 crdregistration_controller.go:111] Starting crd-autoregister controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.497946 1 shared_informer.go:311] Waiting for caches to sync for crd-autoregister vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.498042 1 gc_controller.go:78] Starting apiserver lease garbage collector vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.497663 1 system_namespaces_controller.go:67] Starting system namespaces controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.498133 1 apiservice_controller.go:97] Starting APIServiceRegistrationController vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.498154 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.498155 1 controller.go:121] Starting legacy_token_tracking_controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.498178 1 shared_informer.go:311] Waiting for caches to sync for configmaps vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.498246 1 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/run/config/pki/front-proxy-client.crt::/run/config/pki/front-proxy-client.key" vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.498508 1 gc_controller.go:78] Starting apiserver lease garbage collector vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.498547 1 apf_controller.go:373] Starting API Priority and Fairness config controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.498630 1 controller.go:80] Starting OpenAPI V3 AggregationController vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.499587 1 customresource_discovery_controller.go:289] Starting DiscoveryController vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.499712 1 controller.go:85] Starting OpenAPI controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.499790 1 controller.go:85] Starting OpenAPI V3 controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.499849 1 naming_controller.go:291] Starting NamingConditionController vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.499892 1 establishing_controller.go:76] Starting EstablishingController vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.499940 1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.499978 1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.500017 1 crd_finalizer.go:266] Starting CRDFinalizer vcluster-api-748f6596f-88qgc kube-apiserver E1215 13:56:51.500345 1 controller.go:153] Found stale data, removed previous endpoints on kubernetes service, apiserver didn't exit successfully previously vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.500890 1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.500935 1 shared_informer.go:311] Waiting for caches to sync for cluster_authentication_trust_controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.501016 1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/run/config/pki/ca.crt" vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.501633 1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/run/config/pki/front-proxy-ca.crt" vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.539268 1 controller.go:624] quota admission added evaluator for: leases.coordination.k8s.io vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.598318 1 shared_informer.go:318] Caches are synced for configmaps vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.598407 1 cache.go:39] Caches are synced for AvailableConditionController controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.598643 1 apf_controller.go:378] Running API Priority and Fairness config worker vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.598675 1 shared_informer.go:318] Caches are synced for crd-autoregister vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.598656 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.598680 1 apf_controller.go:381] Running API Priority and Fairness periodic rebalancing process vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.598707 1 aggregator.go:152] initial CRD sync complete... vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.598946 1 autoregister_controller.go:141] Starting autoregister controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.598967 1 cache.go:32] Waiting for caches to sync for autoregister controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.598988 1 cache.go:39] Caches are synced for autoregister controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.601025 1 shared_informer.go:318] Caches are synced for cluster_authentication_trust_controller vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.645466 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:51.652574 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:52.055459 1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue). vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:56:52.506630 1 storage_scheduling.go:111] all system priority classes are created successfully or already exist. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:40.138089 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:40.153119 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:40.172023 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:40.226242 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:40.226293 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:40.237424 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:40.237490 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:40.258229 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:40.264553 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:40.264666 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:40.273375 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:40.295094 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:41.013573 1 handler.go:232] Adding GroupVersion wcdbcd.data.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:41.197875 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:41.197922 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha2 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:41.231537 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:41.231595 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:42.236232 1 handler.go:232] Adding GroupVersion iam.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:57:51.506010 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:58:51.504063 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver E1215 13:59:39.983804 1 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:59:39.983836 1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue. vcluster-api-748f6596f-88qgc kube-apiserver I1215 13:59:51.504291 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:00:51.503460 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.503185 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.522910 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.522984 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.523103 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.523198 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.523228 1 handler.go:232] Adding GroupVersion wcdbcd.data.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.523442 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.523551 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha2 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.523627 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.523687 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.523727 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.523760 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.523797 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:01:51.523829 1 handler.go:232] Adding GroupVersion iam.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:02:29.342815 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:02:51.503252 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:03:51.503397 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:04:51.503939 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:05:51.506461 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:06:29.614421 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:06:51.504184 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:06:51.522836 1 handler.go:232] Adding GroupVersion iam.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:06:51.523059 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:06:51.523191 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:06:51.523353 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:06:51.523540 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:06:51.523635 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:06:51.523696 1 handler.go:232] Adding GroupVersion wcdbcd.data.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:06:51.523742 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:06:51.523787 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha2 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:06:51.523828 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:06:51.523866 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:06:51.523917 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:07:20.556363 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:07:29.344384 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:07:51.504182 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:08:51.503632 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:09:02.574053 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:09:14.618044 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:09:16.192313 1 handler.go:232] Adding GroupVersion wcdbcd.data.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:09:16.192683 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:09:16.193080 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:09:16.193245 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:09:16.193298 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:09:16.193344 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:09:16.193369 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha2 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:09:16.193400 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:09:16.193432 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:09:16.193454 1 handler.go:232] Adding GroupVersion iam.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:09:16.193490 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:09:51.503823 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:10:23.590511 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:10:26.562762 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:10:36.624792 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:10:51.503765 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:11:32.590480 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:11:35.545401 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:11:39.593821 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:11:41.640424 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:11:45.603774 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:11:51.504508 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:11:51.558001 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:11:53.544423 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:12:11.614955 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:12:22.544241 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:12:23.550957 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:12:29.345078 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:12:51.502857 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:12:53.638312 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:13:13.562824 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:13:41.634850 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:13:51.503468 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:13:58.628517 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:14:15.613483 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:14:16.193213 1 handler.go:232] Adding GroupVersion wcdbcd.data.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:14:16.193315 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:14:16.193341 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha2 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:14:16.193386 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:14:16.193410 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:14:16.193898 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:14:16.194321 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:14:16.194364 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:14:16.194454 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:14:16.194541 1 handler.go:232] Adding GroupVersion iam.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver E1215 14:14:29.341377 1 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:14:29.341410 1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue. vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:14:51.502830 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:14:53.572404 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:15:34.623765 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:15:51.502999 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:16:06.626794 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:16:51.503613 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:17:29.347809 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:17:48.916825 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:17:51.504484 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:18:10.937997 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:18:14.158051 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:18:51.504002 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:19:08.231537 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:19:09.088527 1 handler.go:232] Adding GroupVersion wcdbcd.data.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:19:09.088613 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:19:09.088709 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:19:09.088815 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:19:09.088859 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:19:09.088895 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:19:09.088926 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha2 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:19:09.089431 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:19:09.089536 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:19:09.089562 1 handler.go:232] Adding GroupVersion iam.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:19:09.089593 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:19:09.089639 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:19:29.060999 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:19:51.504081 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:20:23.582157 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:20:51.504650 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:21:51.504509 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:22:01.174404 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:22:04.308784 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:22:12.577853 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:22:51.503347 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:23:01.969411 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:23:08.649576 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:23:44.195342 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:23:51.504291 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:09.089469 1 handler.go:232] Adding GroupVersion wcdbcd.data.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:09.089569 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:09.089603 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha2 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:09.089759 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:09.089787 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:09.089826 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:09.089875 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:09.089945 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:09.089977 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:09.090077 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:09.090198 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:09.090268 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:09.090308 1 handler.go:232] Adding GroupVersion iam.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:24:26.692736 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver E1215 14:24:36.552795 1 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:36.552855 1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue. vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:24:51.503720 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:25:13.603174 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:25:51.504525 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:26:02.197812 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:26:09.205484 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:26:15.074201 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:26:25.297453 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:26:33.186959 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:34.082720 1 handler.go:232] Adding GroupVersion wcdbcd.data.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:34.082829 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:34.082956 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:34.083061 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:34.083102 1 handler.go:232] Adding GroupVersion certm.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:34.083224 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:34.083269 1 handler.go:232] Adding GroupVersion com.hidden.sec.certm v1alpha2 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:34.083385 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:34.083456 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:34.083585 1 handler.go:232] Adding GroupVersion iam.sec.hidden.com v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:34.083652 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:34.083693 1 handler.go:232] Adding GroupVersion siptls.sec.hidden.com v1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:34.083856 1 handler.go:232] Adding GroupVersion com.hidden.sec.tls v1alpha1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:26:51.504006 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:26:59.879998 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:27:01.148985 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:27:08.991629 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:27:22.816912 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:27:23.790652 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:27:40.119574 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:27:41.597466 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted vcluster-api-748f6596f-88qgc kube-apiserver I1215 14:27:51.503548 1 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager vcluster-api-748f6596f-88qgc kube-apiserver W1215 14:28:25.163651 1 watcher.go:249] watch chan error: etcdserver: mvcc: required revision has been compacted ```

What did you expect to happen?

To work as it worked with 0.15.7.

How can we reproduce it (as minimally and precisely as possible)?

Sorry, I was not able to create a minimal reproduction environment, but I can easily and reliably replicate the error here. I can also replicate the fact that 0.15.7 works.

Anything else we need to know?

Here is how I create the vcluster:

vcluster create "${VCLUSTER_NAME}" \
  --namespace="${VCLUSTER_NAMESPACE}" \
  --distro=k8s \
  --kubernetes-version=1.27 \
  --connect=false \
  --extra-values=<(
    cat <<EOF
enableHA: true

syncer:
  replicas: 3

etcd:
  replicas: 3

etcd:
  replicas: 3

coredns:
  replicas: 3

api:
  replicas: 3
  # https://github.com/loft-sh/vcluster/issues/1157
  extraArgs:
    - --enable-admission-plugins=AlwaysPullImages

ingress:
  enabled: true
  host: ${VCLUSTER_FQDN}

storage:
  className: "${STORAGE_CLASS:-}"

isolation:
  # microservices such as elastic-search do not support isolation mode due hostPath
  enabled: false

  resourceQuota:
    enabled: false

  limitRange:
    enabled: false

  networkPolicy:
    # https://github.com/loft-sh/vcluster/issues/873
    enabled: false

sync:
  # Needed to use ingresses from within the vcluster
  ingresses:
    enabled: true
  # Makes the list of storageclasses from the host cluster available to the vcluster
  hoststorageclasses:
    enabled: true

proxy:
  metricsServer:
    nodes:
      enabled: true
    pods:
      enabled: true

telemetry:
  disabled: true

Host cluster Kubernetes version

```console $ kubectl version Client Version: v1.29.0 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.6 ```

Host cluster Kubernetes distribution

Vanilla K8s.

vlcuster version

```console $ vcluster --version vcluster version 0.18.1 ```

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

K8s with HA.

OS and Arch

``` OS: Ubuntu 22.04 LTS Arch: amd64 ```
FabianKramm commented 9 months ago

@felipecrs thanks for creating this issue! It looks to me you are using a custom helm chart that still includes the install hook we had in version v0.15.7, that was removed in 0.17, which is probably why this fails, because there shouldn't be any post-install hooks with the latest versions anymore.

felipecrs commented 9 months ago

@FabianKramm, that's weird. I'm not using any custom helm chart, I'm just using --distro k8s and --kubernetes-version 1.27. But I'll do some investigation here.

felipecrs commented 9 months ago

Oh, I see the confusion here. @FabianKramm, the install-hook mentioned in my logs are actually the logs of when I try to install my application against the vcluster.

I posted the vcluster logs after those logs.

FabianKramm commented 9 months ago

@felipecrs that might be a timing issue I believe, vCluster might take a little longer to startup as some replicas might not be up yet or are restarting, we will refactor the k8s distro for 0.19 to include similar to how we do it with k3s and k0s the binaries in the syncer container which should also solve this problem.

felipecrs commented 9 months ago

Got it. Will wait for 0.19 then. Thanks a lot.

felipecrs commented 7 months ago

Confirmed to work with vcluster 0.19.0. Thanks a ton.

felipecrs commented 7 months ago

Ok, to be fair I realized it only worked because I had upgraded other stuff in my services. If I try vcluster 0.19.0 on an old version of my services, it still fails with the same problem (but with vcluster 0.15.7 works fine).

However, this is no longer a problem for me. Just posting in case someone finds the same issue and reads this.