search

loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.26k stars 398 forks source link

Syncer readiness probe failed after disable service sync to host cluster #399

Closed moyuduo closed 2 years ago

moyuduo commented 2 years ago

What happened?

I'm using helm to deploy a vcluster use command helm install vcluster005 vcluster --repo https://charts.loft.sh --namespace vcluster005 --version 0.7.0-alpha.2 --repository-config='' --values values.yaml and the syncr seems readiness probe failed.

[root@centos72 .kube]# cat values.yaml 
vcluster:
  image: rancher/k3s:v1.19.5-k3s2    
  extraArgs:
    - --service-cidr=10.96.0.0/16    

syncer:
  extraArgs:
    - --tls-san=apiserver-abc.moyuduo.com

sync:
  nodes:
    enabled: true
    syncAllNodes: true
  services:
    enabled: false
#  ingresses:
#    enabled: false

[root@centos72 .kube]# k get all -n vcluster005
NAME                READY   STATUS    RESTARTS   AGE
pod/vcluster005-0   2/2     Running   1          41m

NAME                           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/vcluster005            ClusterIP   10.96.38.202   <none>        443/TCP   41m
service/vcluster005-headless   ClusterIP   None           <none>        443/TCP   41m

NAME                           READY   AGE
statefulset.apps/vcluster005   1/1     41m

[root@centos72 .kube]# k describe pod/vcluster005-0 -n vcluster005
Name:         vcluster005-0
Namespace:    vcluster005
Priority:     0
Node:         worker-s002/172.18.0.3
Start Time:   Tue, 15 Mar 2022 15:58:13 +0800
Labels:       app=vcluster
              controller-revision-hash=vcluster005-756c5f5bdb
              release=vcluster005
              statefulset.kubernetes.io/pod-name=vcluster005-0
Annotations:  cni.projectcalico.org/podIP: 10.10.131.186/32
              cni.projectcalico.org/podIPs: 10.10.131.186/32
Status:       Running
IP:           10.10.131.186
IPs:
  IP:           10.10.131.186
Controlled By:  StatefulSet/vcluster005
Containers:
  vcluster:
    Container ID:  docker://3ec921c94b882e9e63e5aa97497dd12cad8aaea4335e6442c4e709a49721c84c
    Image:         rancher/k3s:v1.23.3-k3s1
    Image ID:      docker-pullable://rancher/k3s@sha256:b7c1dc027fa5ea9f9b066dec446204c979ced69d88fbba7d6f4b7ec3a8a55da1
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      /bin/k3s server --write-kubeconfig=/data/k3s-config/kube-config.yaml --data-dir=/data --disable=traefik,servicelb,metrics-server,local-storage,coredns --disable-network-policy --disable-agent --disable-scheduler --disable-cloud-controller --flannel-backend=none --kube-controller-manager-arg=controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle && true
    State:          Running
      Started:      Tue, 15 Mar 2022 15:58:48 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  2Gi
    Requests:
      cpu:        200m
      memory:     256Mi
    Environment:  <none>
    Mounts:
      /data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6dp5g (ro)
  syncer:
    Container ID:  docker://0432f884d540172fbfd1692f741cf9ab4fa338be614a6748979366006babbff7
    Image:         loftsh/vcluster:0.7.0-alpha.2
    Image ID:      docker-pullable://loftsh/vcluster@sha256:10d1436c09e8fa78beb61859d01919f28cce1c10638d25d8b8db8c2b22582975
    Port:          <none>
    Host Port:     <none>
    Args:
      --name=vcluster005
    State:          Running
      Started:      Tue, 15 Mar 2022 15:58:59 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Tue, 15 Mar 2022 15:58:48 +0800
      Finished:     Tue, 15 Mar 2022 15:58:58 +0800
    Ready:          True
    Restart Count:  1
    Limits:
      memory:  1Gi
    Requests:
      cpu:        100m
      memory:     128Mi
    Liveness:     http-get https://:8443/healthz delay=60s timeout=1s period=2s #success=1 #failure=10
    Readiness:    http-get https://:8443/readyz delay=0s timeout=1s period=2s #success=1 #failure=30
    Environment:  <none>
    Mounts:
      /data from data (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6dp5g (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-vcluster005-0
    ReadOnly:   false
  kube-api-access-6dp5g:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Warning  FailedScheduling        40m (x3 over 40m)  default-scheduler        0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled               40m                default-scheduler        Successfully assigned vcluster005/vcluster005-0 to worker-s002
  Normal   SuccessfulAttachVolume  40m                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-6e9330ce-9418-4aa6-845d-b76801cdf11f"
  Normal   Pulling                 40m                kubelet                  Pulling image "rancher/k3s:v1.23.3-k3s1"
  Normal   Pulled                  39m                kubelet                  Successfully pulled image "rancher/k3s:v1.23.3-k3s1" in 14.901541037s
  Normal   Created                 39m                kubelet                  Created container vcluster
  Normal   Started                 39m                kubelet                  Started container vcluster
  Normal   Pulled                  39m (x2 over 39m)  kubelet                  Container image "loftsh/vcluster:0.7.0-alpha.2" already present on machine
  Normal   Created                 39m (x2 over 39m)  kubelet                  Created container syncer
  Normal   Started                 39m (x2 over 39m)  kubelet                  Started container syncer
  Warning  Unhealthy               39m (x8 over 39m)  kubelet                  Readiness probe failed: Get "https://10.10.131.186:8443/readyz": dial tcp 10.10.131.186:8443: connect: connection refused

after serveral time test i find Readiness probe failed: Get "https://10.10.131.186:8443/readyz": dial tcp 10.10.131.186:8443: connect: connection refused error is due to disable service sync in values.yaml，is this a bug？

more detail: syncer log:

[root@centos72 .kube]# k logs pod/vcluster005-0 -n vcluster005 -c syncer
I0315 07:59:00.448076       1 start.go:255] Using physical cluster at https://10.96.0.1:443
I0315 07:59:00.479042       1 start.go:286] Can connect to virtual cluster with version v1.23.3+k3s1
I0315 07:59:00.681779       1 plugins.go:158] Loaded 1 mutating admission controller(s) successfully in the following order: MutatingAdmissionWebhook.
I0315 07:59:00.681796       1 plugins.go:161] Loaded 1 validating admission controller(s) successfully in the following order: ValidatingAdmissionWebhook.
I0315 07:59:00.681931       1 start.go:316] Start Plugins Manager...
I0315 07:59:00.681959       1 logr.go:249] Start secrets sync controller
I0315 07:59:00.683690       1 logr.go:249] Plugin server listening on localhost:10099
I0315 07:59:00.683984       1 logr.go:249] Start fake-persistentvolumes sync controller
I0315 07:59:00.683991       1 logr.go:249] Start endpoints sync controller
I0315 07:59:00.683996       1 logr.go:249] Start pods sync controller
I0315 07:59:00.684240       1 logr.go:249] Start events sync controller
I0315 07:59:00.684245       1 logr.go:249] Start persistentvolumeclaims sync controller
I0315 07:59:00.684257       1 logr.go:249] Start services sync controller
I0315 07:59:00.684263       1 logr.go:249] Start configmaps sync controller
I0315 07:59:00.684268       1 logr.go:249] Start ingresses sync controller
I0315 07:59:00.685421       1 logr.go:249] Start fake-nodes sync controller
serviceaccount/coredns unchanged
clusterrole.rbac.authorization.k8s.io/system:coredns unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:coredns unchanged
configmap/coredns unchanged
deployment.apps/coredns configured
service/kube-dns unchanged
I0315 07:59:00.734604       1 start.go:354] CoreDNS configuration from the manifest file applied successfully
I0315 07:59:01.299774       1 server.go:227] Starting tls proxy server at 0.0.0.0:8443
I0315 07:59:01.300203       1 syncer.go:143] Generating serving cert for service ips: [10.96.38.202]
I0315 07:59:01.300270       1 dynamic_cafile_content.go:156] "Starting controller" name="request-header::/data/server/tls/request-header-ca.crt"
I0315 07:59:01.300275       1 dynamic_cafile_content.go:156] "Starting controller" name="client-ca-bundle::/data/server/tls/client-ca.crt"
I0315 07:59:01.300862       1 secure_serving.go:200] Serving securely on [::]:8443
I0315 07:59:01.300937       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
secret/vc-vcluster005 created
I0315 07:59:03.947058       1 logr.go:249] reconciler group  reconciler kind Secret: controller: secret: Starting Controller
I0315 07:59:03.947073       1 logr.go:249] reconciler group  reconciler kind Secret: controller: secret: Starting workers worker count 1
I0315 07:59:03.947096       1 logr.go:249] reconciler group  reconciler kind Secret: controller: secret: Starting EventSource source &source.Kind{Type:(*v1.Secret)(0xc000ec6dc0), cache:(*cache.informerCache)(0xc000314020), started:(chan error)(nil), startCancel:(func())(nil)}
I0315 07:59:03.947120       1 logr.go:249] reconciler group  reconciler kind Secret: controller: secret: Starting EventSource source &source.kindWithCache{kind:source.Kind{Type:(*v1.Secret)(0xc000ec6c80), cache:(*cache.informerCache)(0xc00000e298), started:(chan error)(nil), startCancel:(func())(nil)}}
I0315 07:59:03.947148       1 logr.go:249] reconciler group  reconciler kind Secret: controller: secret: Starting EventSource source &source.Kind{Type:(*v1.Ingress)(0xc0005c0480), cache:(*cache.informerCache)(0xc000314020), started:(chan error)(nil), startCancel:(func())(nil)}
I0315 07:59:03.947150       1 logr.go:249] reconciler group  reconciler kind ConfigMap: controller: coredns_nodehosts: Starting EventSource source kind source: *v1.ConfigMap
I0315 07:59:03.947166       1 logr.go:249] reconciler group  reconciler kind Secret: controller: secret: Starting EventSource source &source.Kind{Type:(*v1.Pod)(0xc000a92400), cache:(*cache.informerCache)(0xc000314020), started:(chan error)(nil), startCancel:(func())(nil)}
I0315 07:59:03.947170       1 logr.go:249] reconciler group  reconciler kind ConfigMap: controller: coredns_nodehosts: Starting EventSource source kind source: *v1.Node
I0315 07:59:03.947177       1 logr.go:249] reconciler group  reconciler kind ConfigMap: controller: coredns_nodehosts: Starting Controller
I0315 07:59:03.947264       1 logr.go:249] reconciler group  reconciler kind Endpoints: controller: endpoints: Starting Controller
I0315 07:59:03.947279       1 logr.go:249] reconciler group  reconciler kind Endpoints: controller: endpoints: Starting workers worker count 1
I0315 07:59:03.947294       1 logr.go:249] reconciler group  reconciler kind Endpoints: controller: endpoints: Starting EventSource source &source.Kind{Type:(*v1.Endpoints)(0xc000ec7040), cache:(*cache.informerCache)(0xc000314020), started:(chan error)(nil), startCancel:(func())(nil)}
I0315 07:59:03.947305       1 logr.go:249] reconciler group  reconciler kind Endpoints: controller: endpoints: Starting EventSource source &source.kindWithCache{kind:source.Kind{Type:(*v1.Endpoints)(0xc000ec6f00), cache:(*cache.informerCache)(0xc00000e298), started:(chan error)(nil), startCancel:(func())(nil)}}
I0315 07:59:03.947308       1 logr.go:249] reconciler group  reconciler kind PersistentVolume: controller: fake-persistentvolume: Starting EventSource source kind source: *v1.PersistentVolume
I0315 07:59:03.947321       1 logr.go:249] reconciler group  reconciler kind PersistentVolume: controller: fake-persistentvolume: Starting EventSource source kind source: *v1.PersistentVolumeClaim
I0315 07:59:03.947331       1 logr.go:249] reconciler group  reconciler kind PersistentVolume: controller: fake-persistentvolume: Starting Controller
I0315 07:59:03.947401       1 logr.go:249] reconciler group  reconciler kind Event: controller: event: Starting Controller
I0315 07:59:03.947412       1 logr.go:249] reconciler group  reconciler kind Event: controller: event: Starting workers worker count 1
I0315 07:59:03.947480       1 logr.go:249] reconciler group  reconciler kind Event: controller: event: Starting EventSource source &source.Kind{Type:(*v1.Event)(0xc000b3af00), cache:(*cache.informerCache)(0xc000314020), started:(chan error)(nil), startCancel:(func())(nil)}
I0315 07:59:03.947491       1 logr.go:249] reconciler group  reconciler kind Pod: controller: pod: Starting EventSource source kind source: *v1.Pod
I0315 07:59:03.947507       1 logr.go:249] reconciler group  reconciler kind Event: controller: event: Starting EventSource source &source.kindWithCache{kind:source.Kind{Type:(*v1.Event)(0xc000b3ac80), cache:(*cache.informerCache)(0xc00000e298), started:(chan error)(nil), startCancel:(func())(nil)}}
I0315 07:59:03.947613       1 logr.go:249] reconciler group  reconciler kind Pod: controller: pod: Starting EventSource source &{{%!s(*v1.Pod=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} {[] [] [] []  <nil> <nil>  map[]   <nil>  false false false <nil> <nil> []   <nil>  [] []  <nil> <nil> [] <nil> <nil> <nil> map[] [] <nil> <nil>} { []      [] <nil> [] []  []}}) %!s(*cache.informerCache=&{0xc0008763a0}) %!s(chan error=<nil>) %!s(func()=<nil>)}}
I0315 07:59:03.947632       1 logr.go:249] reconciler group  reconciler kind Pod: controller: pod: Starting EventSource source kind source: *v1.Namespace
I0315 07:59:03.947640       1 logr.go:249] reconciler group  reconciler kind Pod: controller: pod: Starting Controller
I0315 07:59:03.947700       1 logr.go:249] reconciler group  reconciler kind Service: controller: service: Starting Controller
I0315 07:59:03.947709       1 logr.go:249] reconciler group  reconciler kind Service: controller: service: Starting workers worker count 1
I0315 07:59:03.947722       1 logr.go:249] reconciler group  reconciler kind PersistentVolumeClaim: controller: persistent-volume-claim: Starting EventSource source kind source: *v1.PersistentVolumeClaim
I0315 07:59:03.947729       1 logr.go:249] reconciler group  reconciler kind Service: controller: service: Starting EventSource source &source.Kind{Type:(*v1.Service)(0xc0007ca280), cache:(*cache.informerCache)(0xc000314020), started:(chan error)(nil), startCancel:(func())(nil)}
I0315 07:59:03.947747       1 logr.go:249] reconciler group  reconciler kind Service: controller: service: Starting EventSource source &source.kindWithCache{kind:source.Kind{Type:(*v1.Service)(0xc0007ca000), cache:(*cache.informerCache)(0xc00000e298), started:(chan error)(nil), startCancel:(func())(nil)}}
I0315 07:59:03.947757       1 logr.go:249] reconciler group  reconciler kind PersistentVolumeClaim: controller: persistent-volume-claim: Starting EventSource source &{{%!s(*v1.PersistentVolumeClaim=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} {[] <nil> {map[] map[]}  <nil> <nil> <nil> <nil>} { [] map[] [] map[] <nil>}}) %!s(*cache.informerCache=&{0xc0008763a0}) %!s(chan error=<nil>) %!s(func()=<nil>)}}
I0315 07:59:03.949747       1 logr.go:249] reconciler group  reconciler kind PersistentVolumeClaim: controller: persistent-volume-claim: Starting Controller
I0315 07:59:03.950505       1 logr.go:249] endpoints: kube-dns: create physical endpoints vcluster005/kube-dns-x-kube-system-x-vcluster005
I0315 07:59:03.951265       1 logr.go:249] reconciler group  reconciler kind ConfigMap: controller: configmap: Starting EventSource source kind source: *v1.ConfigMap
I0315 07:59:03.951348       1 logr.go:249] reconciler group  reconciler kind ConfigMap: controller: configmap: Starting EventSource source &{{%!s(*v1.ConfigMap=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} <nil> map[] map[]}) %!s(*cache.informerCache=&{0xc0008763a0}) %!s(chan error=<nil>) %!s(func()=<nil>)}}
I0315 07:59:03.951370       1 logr.go:249] reconciler group  reconciler kind ConfigMap: controller: configmap: Starting EventSource source kind source: *v1.Pod
I0315 07:59:03.951386       1 logr.go:249] reconciler group  reconciler kind ConfigMap: controller: configmap: Starting Controller
I0315 07:59:03.951655       1 logr.go:249] reconciler group  reconciler kind Node: controller: fake-node: Starting Controller
I0315 07:59:03.951666       1 logr.go:249] reconciler group  reconciler kind Node: controller: fake-node: Starting workers worker count 1
I0315 07:59:03.951907       1 logr.go:249] reconciler group  reconciler kind Node: controller: fake-node: Starting EventSource source &source.Kind{Type:(*v1.Node)(0xc00151e000), cache:(*cache.informerCache)(0xc000314020), started:(chan error)(nil), startCancel:(func())(nil)}
I0315 07:59:03.951950       1 logr.go:249] reconciler group  reconciler kind Node: controller: fake-node: Starting EventSource source &source.kindWithCache{kind:source.Kind{Type:(*v1.Pod)(0xc000601000), cache:(*cache.informerCache)(0xc00000e298), started:(chan error)(nil), startCancel:(func())(nil)}}
I0315 07:59:03.951963       1 logr.go:249] reconciler group networking.k8s.io reconciler kind Ingress: controller: ingress: Starting EventSource source kind source: *v1.Ingress
I0315 07:59:03.951974       1 logr.go:249] reconciler group  reconciler kind Node: controller: fake-node: Starting EventSource source &source.Kind{Type:(*v1.Pod)(0xc000601800), cache:(*cache.informerCache)(0xc000314020), started:(chan error)(nil), startCancel:(func())(nil)}
I0315 07:59:03.952028       1 logr.go:249] reconciler group networking.k8s.io reconciler kind Ingress: controller: ingress: Starting EventSource source &{{%!s(*v1.Ingress=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} {<nil> <nil> [] []} {{[]}}}) %!s(*cache.informerCache=&{0xc0008763a0}) %!s(chan error=<nil>) %!s(func()=<nil>)}}
I0315 07:59:03.952036       1 logr.go:249] reconciler group networking.k8s.io reconciler kind Ingress: controller: ingress: Starting Controller
E0315 07:59:03.962811       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
I0315 07:59:03.962882       1 logr.go:249] service: kube-dns: create physical service vcluster005/kube-dns-x-kube-system-x-vcluster005
I0315 07:59:03.973040       1 logr.go:249] service: kube-dns: recreating virtual service kube-system/kube-dns, because cluster ip differs 10.96.111.248 != 10.43.64.149
I0315 07:59:03.978528       1 logr.go:249] endpoints: kube-dns: delete physical vcluster005/kube-dns-x-kube-system-x-vcluster005, because virtual object was deleted
E0315 07:59:03.982102       1 syncer.go:131] error recreating virtual service: kube-system/kube-dns: Service "kube-dns" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.111.248"}: failed to allocate IP 10.96.111.248: the provided IP (10.96.111.248) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 07:59:03.982125       1 logr.go:265] service: kube-dns:  error creating virtual service: kube-system/kube-dns
E0315 07:59:03.982156       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kube-dns namespace kube-system: Reconciler error Service "kube-dns" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.111.248"}: failed to allocate IP 10.96.111.248: the provided IP (10.96.111.248) is not in the valid range. The range of valid IPs is 10.43.0.0/16
I0315 07:59:03.982212       1 logr.go:249] service: kube-dns: delete physical vcluster005/kube-dns-x-kube-system-x-vcluster005, because virtual object was deleted
I0315 07:59:04.048451       1 logr.go:249] reconciler group  reconciler kind ConfigMap: controller: coredns_nodehosts: Starting workers worker count 1
I0315 07:59:04.048466       1 logr.go:249] reconciler group  reconciler kind PersistentVolume: controller: fake-persistentvolume: Starting workers worker count 1
I0315 07:59:04.048775       1 logr.go:249] reconciler group  reconciler kind Pod: controller: pod: Starting workers worker count 1
E0315 07:59:04.048960       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
I0315 07:59:04.050546       1 logr.go:249] reconciler group  reconciler kind PersistentVolumeClaim: controller: persistent-volume-claim: Starting workers worker count 1
E0315 07:59:04.055045       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
I0315 07:59:04.056053       1 logr.go:249] reconciler group networking.k8s.io reconciler kind Ingress: controller: ingress: Starting workers worker count 1
I0315 07:59:04.058240       1 logr.go:249] reconciler group  reconciler kind ConfigMap: controller: configmap: Starting workers worker count 1
I0315 07:59:04.058398       1 logr.go:249] configmap: kube-root-ca.crt: create physical configmap vcluster005/kube-root-ca.crt-x-kube-system-x-vcluster005
E0315 07:59:04.065849       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
I0315 07:59:04.072100       1 logr.go:249] configmap: coredns: create physical configmap vcluster005/coredns-x-kube-system-x-vcluster005
E0315 07:59:04.086821       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
E0315 07:59:04.127258       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
E0315 07:59:04.207687       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
E0315 07:59:04.368453       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
E0315 07:59:04.689042       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
E0315 07:59:05.329804       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
E0315 07:59:06.611018       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
E0315 07:59:09.171766       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
E0315 07:59:12.434754       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 07:59:14.292547       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
E0315 07:59:22.411786       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 07:59:24.533713       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
E0315 07:59:32.409653       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 07:59:42.417055       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 07:59:45.014150       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
E0315 07:59:52.428030       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 08:00:02.411004       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 08:00:12.410705       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 08:00:22.410223       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 08:00:22.411337       1 logr.go:265] reconciler group  reconciler kind Endpoints: controller: endpoints: name kubernetes namespace default: Reconciler error Operation cannot be fulfilled on endpoints "kubernetes": StorageError: invalid object, Code: 4, Key: /registry/services/endpoints/default/kubernetes, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 1b20d25e-66f8-445f-875a-d7e20134e9b4, UID in object meta: 
E0315 08:00:25.974970       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
E0315 08:00:32.412353       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 08:00:32.413974       1 logr.go:265] reconciler group  reconciler kind Endpoints: controller: endpoints: name kubernetes namespace default: Reconciler error Operation cannot be fulfilled on endpoints "kubernetes": StorageError: invalid object, Code: 4, Key: /registry/services/endpoints/default/kubernetes, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 5aaf5d56-91e0-4c1d-8a0b-d6f027ef5e65, UID in object meta: 
E0315 08:00:42.414355       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 08:00:42.416687       1 logr.go:265] reconciler group  reconciler kind Endpoints: controller: endpoints: name kubernetes namespace default: Reconciler error Operation cannot be fulfilled on endpoints "kubernetes": StorageError: invalid object, Code: 4, Key: /registry/services/endpoints/default/kubernetes, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 10bf728d-9958-4b06-b5ca-791b2ed35cf6, UID in object meta: 
E0315 08:00:52.416832       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 08:01:02.416542       1 logr.go:265] reconciler group  reconciler kind Endpoints: controller: endpoints: name kubernetes namespace default: Reconciler error Operation cannot be fulfilled on endpoints "kubernetes": StorageError: invalid object, Code: 4, Key: /registry/services/endpoints/default/kubernetes, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: df743de0-b4eb-4102-ba07-3e98a8d918e4, UID in object meta: 
E0315 08:01:02.417220       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 08:01:12.414618       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 08:01:12.416257       1 logr.go:265] reconciler group  reconciler kind Endpoints: controller: endpoints: name kubernetes namespace default: Reconciler error Operation cannot be fulfilled on endpoints "kubernetes": StorageError: invalid object, Code: 4, Key: /registry/services/endpoints/default/kubernetes, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 23525aa0-36cc-4212-bc86-7e4a9ea7c7ff, UID in object meta: 
E0315 08:01:22.417832       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 08:01:32.416229       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 08:01:42.416737       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 08:01:47.895755       1 logr.go:265] reconciler group  reconciler kind Pod: controller: pod: name coredns-66dcd89f55-pbsk9 namespace kube-system: Reconciler error waiting for DNS service IP
E0315 08:01:52.421000       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16
E0315 08:02:02.418498       1 logr.go:265] reconciler group  reconciler kind Service: controller: service: name kubernetes namespace default: Reconciler error Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.38.202"}: failed to allocate IP 10.96.38.202: the provided IP (10.96.38.202) is not in the valid range. The range of valid IPs is 10.43.0.0/16

and if you deploy a pod in vcluster, that pod will always pending and never sync to host cluster

What did you expect to happen?

disable service and ingress sync to host cluster and no readiness probe failed error and deploy pod successfully

How can we reproduce it (as minimally and precisely as possible)?

how to fix

Anything else we need to know?

No response

Host cluster Kubernetes version

```console $ kubectl version # paste output here ```

Host cluster Kubernetes distribution

``` # Write here ```

vlcuster version

```console $ vcluster --version # paste output here ```

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

``` k3s ```

OS and Arch

``` OS: Arch: ```
FabianKramm commented 2 years ago

@moyuduo thanks for creating this issue! Why are you disabling service sync? Certain options will leave vcluster in a broken state, so disabling services is not a good idea in general and rather intended to be replaced through a custom plugin sync

moyuduo commented 2 years ago

@moyuduo thanks for creating this issue! Why are you disabling service sync? Certain options will leave vcluster in a broken state, so disabling services is not a good idea in general and rather intended to be replaced through a custom plugin sync

Why i want to disable service sync is that i think if i want to create many vcluster and each vcluster's nodePort servcie is sync to host cluster,so that port become a rare resource,i just want to expose nodePort service on pod/vcluster005-0, so i can use pod ip + port to visit