search

loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.84k stars 423 forks source link

Waiting for vCluster to come up... #41

Closed randyguptaIKEA closed 3 years ago

randyguptaIKEA commented 3 years ago

I can't connect to the vcluster:

vcluster connect vcluster-2 --update-current --debug
[info]   Waiting for vCluster to come up...
[fatal]  timed out waiting for the condition
wait for vCluster
github.com/loft-sh/vcluster/cmd/vclusterctl/cmd.(*ConnectCmd).Run
    /Users/runner/work/vcluster/vcluster/cmd/vclusterctl/cmd/connect.go:106
github.com/loft-sh/vcluster/cmd/vclusterctl/cmd.NewConnectCmd.func1
    /Users/runner/work/vcluster/vcluster/cmd/vclusterctl/cmd/connect.go:61
github.com/spf13/cobra.(*Command).execute
    /Users/runner/work/vcluster/vcluster/vendor/github.com/spf13/cobra/command.go:850
github.com/spf13/cobra.(*Command).ExecuteC
    /Users/runner/work/vcluster/vcluster/vendor/github.com/spf13/cobra/command.go:958
github.com/spf13/cobra.(*Command).Execute
    /Users/runner/work/vcluster/vcluster/vendor/github.com/spf13/cobra/command.go:895
github.com/loft-sh/vcluster/cmd/vclusterctl/cmd.Execute
    /Users/runner/work/vcluster/vcluster/cmd/vclusterctl/cmd/root.go:36
main.main
    /Users/runner/work/vcluster/vcluster/cmd/vclusterctl/main.go:16
runtime.main
    /Users/runner/hostedtoolcache/go/1.16.3/x64/src/runtime/proc.go:225
runtime.goexit
    /Users/runner/hostedtoolcache/go/1.16.3/x64/src/runtime/asm_amd64.s:1371

All pod seem to be started fine:

k get po -o wide
NAME                                                  READY   STATUS    RESTARTS   AGE   IP           NODE                    NOMINATED NODE   READINESS GATES
coredns-854c77959c-9vh84-x-kube-system-x-vcluster-2   1/1     Running   0          16m   10.42.0.10   t320.xxx   <none>           <none>
vcluster-2-0                                          2/2     Running   1          17m   10.42.0.8    t320.xxx   <none>           <none>

To logs from the controle plane:

k logs -f vcluster-01-0 -c vcluster
time="2021-06-04T21:55:41.921673812Z" level=info msg="Starting k3s v1.20.4+k3s1 (838a906a)"
time="2021-06-04T21:55:41.930902651Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2021-06-04T21:55:41.930934416Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2021-06-04T21:55:41.934174422Z" level=info msg="Database tables and indexes are up to date"
time="2021-06-04T21:55:41.935341771Z" level=info msg="Kine listening on unix://kine.sock"
time="2021-06-04T21:55:41.960523842Z" level=info msg="certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.961718174Z" level=info msg="certificate CN=system:kube-controller-manager signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.962882249Z" level=info msg="certificate CN=system:kube-scheduler signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.964065893Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.965215732Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.966185326Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.967318198Z" level=info msg="certificate CN=cloud-controller-manager signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.969107829Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-server-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.970873906Z" level=info msg="certificate CN=system:auth-proxy signed by CN=k3s-request-header-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.972648892Z" level=info msg="certificate CN=etcd-server signed by CN=etcd-server-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.973729851Z" level=info msg="certificate CN=etcd-client signed by CN=etcd-server-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.975470544Z" level=info msg="certificate CN=etcd-peer signed by CN=etcd-peer-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:42.105260943Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:42 +0000 UTC"
time="2021-06-04T21:55:42.105588248Z" level=info msg="Active TLS secret  (ver=) (count 7): map[listener.cattle.io/cn-10.42.0.8:10.42.0.8 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=BE9B09BF2487E6F5BD28D40EF63D5F11C05D60B6]"
time="2021-06-04T21:55:42.111807715Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/data/server/tls/temporary-certs --client-ca-file=/data/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --feature-gates=ServiceAccountIssuerDiscovery=false --insecure-port=0 --kubelet-certificate-authority=/data/server/tls/server-ca.crt --kubelet-client-certificate=/data/server/tls/client-kube-apiserver.crt --kubelet-client-key=/data/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/data/server/tls/client-auth-proxy.crt --proxy-client-key-file=/data/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/data/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=k3s --service-account-key-file=/data/server/tls/service.key --service-account-signing-key-file=/data/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/data/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/data/server/tls/serving-kube-apiserver.key"
Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
I0604 21:55:42.113032       1 server.go:659] external host was not specified, using 10.42.0.8
I0604 21:55:42.113285       1 server.go:196] Version: v1.20.4+k3s1
I0604 21:55:42.842273       1 shared_informer.go:240] Waiting for caches to sync for node_authorizer
I0604 21:55:42.843434       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0604 21:55:42.843449       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0604 21:55:42.844685       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0604 21:55:42.844699       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0604 21:55:42.876519       1 instance.go:289] Using reconciler: lease
I0604 21:55:42.961512       1 rest.go:131] the default service ipfamily for this cluster is: IPv4
W0604 21:55:43.317390       1 genericapiserver.go:419] Skipping API batch/v2alpha1 because it has no resources.
W0604 21:55:43.333438       1 genericapiserver.go:419] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
W0604 21:55:43.348550       1 genericapiserver.go:419] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0604 21:55:43.359028       1 genericapiserver.go:419] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0604 21:55:43.363156       1 genericapiserver.go:419] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0604 21:55:43.369765       1 genericapiserver.go:419] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0604 21:55:43.372706       1 genericapiserver.go:419] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0604 21:55:43.378781       1 genericapiserver.go:419] Skipping API apps/v1beta2 because it has no resources.
W0604 21:55:43.378797       1 genericapiserver.go:419] Skipping API apps/v1beta1 because it has no resources.
I0604 21:55:43.389090       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0604 21:55:43.389119       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
time="2021-06-04T21:55:43.400014907Z" level=info msg="Waiting for API server to become available"
time="2021-06-04T21:55:43.400037410Z" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/data/server/tls/client-ca.crt --cluster-signing-key-file=/data/server/tls/client-ca.key --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle --kubeconfig=/data/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/data/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/data/server/tls/service.key --use-service-account-credentials=true"
time="2021-06-04T21:55:43.401426513Z" level=info msg="Node token is available at /data/server/token"
time="2021-06-04T21:55:43.401510420Z" level=info msg="To join node to cluster: k3s agent -s https://10.42.0.8:6443 -t ${NODE_TOKEN}"
time="2021-06-04T21:55:43.403352418Z" level=info msg="Wrote kubeconfig /k3s-config/kube-config.yaml"
time="2021-06-04T21:55:43.403659056Z" level=info msg="Run: k3s kubectl"
I0604 21:55:46.078612       1 dynamic_cafile_content.go:167] Starting request-header::/data/server/tls/request-header-ca.crt
I0604 21:55:46.078630       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/data/server/tls/client-ca.crt
I0604 21:55:46.079004       1 dynamic_serving_content.go:130] Starting serving-cert::/data/server/tls/serving-kube-apiserver.crt::/data/server/tls/serving-kube-apiserver.key
I0604 21:55:46.079611       1 secure_serving.go:197] Serving securely on 127.0.0.1:6444
I0604 21:55:46.079631       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0604 21:55:46.079690       1 apf_controller.go:261] Starting API Priority and Fairness config controller
I0604 21:55:46.079693       1 available_controller.go:475] Starting AvailableConditionController
I0604 21:55:46.079736       1 dynamic_serving_content.go:130] Starting aggregator-proxy-cert::/data/server/tls/client-auth-proxy.crt::/data/server/tls/client-auth-proxy.key
I0604 21:55:46.079740       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0604 21:55:46.079770       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0604 21:55:46.079791       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0604 21:55:46.079739       1 customresource_discovery_controller.go:209] Starting DiscoveryController
I0604 21:55:46.079824       1 controller.go:86] Starting OpenAPI controller
I0604 21:55:46.079851       1 naming_controller.go:291] Starting NamingConditionController
I0604 21:55:46.079872       1 establishing_controller.go:76] Starting EstablishingController
I0604 21:55:46.079894       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I0604 21:55:46.079914       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0604 21:55:46.079943       1 crd_finalizer.go:266] Starting CRDFinalizer
I0604 21:55:46.079987       1 autoregister_controller.go:141] Starting autoregister controller
I0604 21:55:46.079999       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0604 21:55:46.080028       1 controller.go:83] Starting OpenAPI AggregationController
I0604 21:55:46.080062       1 crdregistration_controller.go:111] Starting crd-autoregister controller
I0604 21:55:46.080071       1 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
I0604 21:55:46.080574       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0604 21:55:46.080585       1 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
I0604 21:55:46.080625       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/data/server/tls/client-ca.crt
I0604 21:55:46.081106       1 dynamic_cafile_content.go:167] Starting request-header::/data/server/tls/request-header-ca.crt
I0604 21:55:46.142405       1 shared_informer.go:247] Caches are synced for node_authorizer 
I0604 21:55:46.179817       1 apf_controller.go:266] Running API Priority and Fairness config worker
I0604 21:55:46.179817       1 cache.go:39] Caches are synced for AvailableConditionController controller
I0604 21:55:46.179843       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0604 21:55:46.180041       1 cache.go:39] Caches are synced for autoregister controller
I0604 21:55:46.180113       1 shared_informer.go:247] Caches are synced for crd-autoregister 
I0604 21:55:46.180618       1 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller 
I0604 21:55:46.186770       1 controller.go:609] quota admission added evaluator for: namespaces
E0604 21:55:46.217452       1 controller.go:151] Unable to perform initial Kubernetes service initialization: Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.43.0.1"}: failed to allocated ip:10.43.0.1 with error:cannot allocate resources of type serviceipallocations at this time
E0604 21:55:46.218258       1 controller.go:156] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/10.42.0.8, ResourceVersion: 0, AdditionalErrorMsg: 
I0604 21:55:47.078685       1 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0604 21:55:47.078732       1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0604 21:55:47.089007       1 storage_scheduling.go:132] created PriorityClass system-node-critical with value 2000001000
I0604 21:55:47.094908       1 storage_scheduling.go:132] created PriorityClass system-cluster-critical with value 2000000000
I0604 21:55:47.094951       1 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
I0604 21:55:47.654599       1 controller.go:609] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0604 21:55:47.693050       1 controller.go:609] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
W0604 21:55:47.749626       1 lease.go:233] Resetting endpoints for master service "kubernetes" to [10.42.0.8]
I0604 21:55:47.750552       1 controller.go:609] quota admission added evaluator for: endpoints
I0604 21:55:47.754111       1 controller.go:609] quota admission added evaluator for: endpointslices.discovery.k8s.io
time="2021-06-04T21:55:48.095659943Z" level=info msg="Kube API server is now running"
time="2021-06-04T21:55:48.095718982Z" level=info msg="k3s is up and running"
Flag --address has been deprecated, see --bind-address instead.
I0604 21:55:48.100911       1 controllermanager.go:176] Version: v1.20.4+k3s1
I0604 21:55:48.101263       1 deprecated_insecure_serving.go:53] Serving insecurely on 127.0.0.1:10252
time="2021-06-04T21:55:48.102840712Z" level=info msg="Creating CRD addons.k3s.cattle.io"
time="2021-06-04T21:55:48.107067601Z" level=info msg="Creating CRD helmcharts.helm.cattle.io"
time="2021-06-04T21:55:48.111963597Z" level=info msg="Creating CRD helmchartconfigs.helm.cattle.io"
time="2021-06-04T21:55:48.119053686Z" level=info msg="Waiting for CRD addons.k3s.cattle.io to become available"
time="2021-06-04T21:55:48.621757509Z" level=info msg="Done waiting for CRD addons.k3s.cattle.io to become available"
time="2021-06-04T21:55:48.621786065Z" level=info msg="Waiting for CRD helmcharts.helm.cattle.io to become available"
time="2021-06-04T21:55:49.124524060Z" level=info msg="Done waiting for CRD helmcharts.helm.cattle.io to become available"
time="2021-06-04T21:55:49.124549908Z" level=info msg="Waiting for CRD helmchartconfigs.helm.cattle.io to become available"
time="2021-06-04T21:55:49.627166966Z" level=info msg="Done waiting for CRD helmchartconfigs.helm.cattle.io to become available"
time="2021-06-04T21:55:49.638156596Z" level=info msg="Writing static file: /data/server/static/charts/traefik-1.81.0.tgz"
time="2021-06-04T21:55:49.638442987Z" level=info msg="Writing manifest: /data/server/manifests/rolebindings.yaml"
time="2021-06-04T21:55:49.638594232Z" level=info msg="Writing manifest: /data/server/manifests/coredns.yaml"
time="2021-06-04T21:55:49.739233614Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
time="2021-06-04T21:55:49.749741984Z" level=info msg="Cluster dns configmap has been set successfully"
I0604 21:55:49.751620       1 controller.go:609] quota admission added evaluator for: serviceaccounts
I0604 21:55:49.789560       1 controller.go:609] quota admission added evaluator for: deployments.apps
I0604 21:55:49.805183       1 controller.go:609] quota admission added evaluator for: addons.k3s.cattle.io
I0604 21:55:49.821433       1 request.go:655] Throttling request took 1.04843254s, request: GET:https://127.0.0.1:6444/apis/coordination.k8s.io/v1?timeout=32s
time="2021-06-04T21:55:49.840479939Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller"
time="2021-06-04T21:55:49.840494475Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
time="2021-06-04T21:55:49.840502643Z" level=info msg="Starting /v1, Kind=Service controller"
time="2021-06-04T21:55:49.840507482Z" level=info msg="Starting batch/v1, Kind=Job controller"
time="2021-06-04T21:55:49.840509119Z" level=info msg="Starting /v1, Kind=Pod controller"
time="2021-06-04T21:55:49.840515344Z" level=info msg="Starting /v1, Kind=Node controller"
time="2021-06-04T21:55:49.840518828Z" level=info msg="Starting /v1, Kind=Endpoints controller"
time="2021-06-04T21:55:50.216429888Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2021-06-04T21:55:50.221620904Z" level=info msg="Active TLS secret k3s-serving (ver=237) (count 7): map[listener.cattle.io/cn-10.42.0.8:10.42.0.8 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=BE9B09BF2487E6F5BD28D40EF63D5F11C05D60B6]"
I0604 21:55:50.727343       1 shared_informer.go:240] Waiting for caches to sync for tokens
I0604 21:55:50.744902       1 controllermanager.go:554] Started "endpointslice"
I0604 21:55:50.744976       1 endpointslice_controller.go:237] Starting endpoint slice controller
I0604 21:55:50.744994       1 shared_informer.go:240] Waiting for caches to sync for endpoint_slice
I0604 21:55:50.754798       1 controllermanager.go:554] Started "endpointslicemirroring"
W0604 21:55:50.754823       1 controllermanager.go:533] "persistentvolume-expander" is disabled
I0604 21:55:50.754826       1 endpointslicemirroring_controller.go:211] Starting EndpointSliceMirroring controller
I0604 21:55:50.754842       1 shared_informer.go:240] Waiting for caches to sync for endpoint_slice_mirroring
I0604 21:55:50.764512       1 controllermanager.go:554] Started "podgc"
W0604 21:55:50.764528       1 controllermanager.go:533] "bootstrapsigner" is disabled
W0604 21:55:50.764534       1 controllermanager.go:533] "nodeipam" is disabled
I0604 21:55:50.764622       1 gc_controller.go:89] Starting GC controller
I0604 21:55:50.764640       1 shared_informer.go:240] Waiting for caches to sync for GC
E0604 21:55:50.772476       1 core.go:92] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail
W0604 21:55:50.772498       1 controllermanager.go:546] Skipping "service"
I0604 21:55:50.780317       1 controllermanager.go:554] Started "clusterrole-aggregation"
I0604 21:55:50.780393       1 clusterroleaggregation_controller.go:149] Starting ClusterRoleAggregator
I0604 21:55:50.780407       1 shared_informer.go:240] Waiting for caches to sync for ClusterRoleAggregator
I0604 21:55:50.787585       1 controllermanager.go:554] Started "pv-protection"
W0604 21:55:50.787603       1 controllermanager.go:546] Skipping "ttl-after-finished"
W0604 21:55:50.787611       1 controllermanager.go:546] Skipping "ephemeral-volume"
I0604 21:55:50.787682       1 pv_protection_controller.go:83] Starting PV protection controller
I0604 21:55:50.787694       1 shared_informer.go:240] Waiting for caches to sync for PV protection
I0604 21:55:50.794696       1 controllermanager.go:554] Started "replicaset"
I0604 21:55:50.794718       1 replica_set.go:182] Starting replicaset controller
I0604 21:55:50.794729       1 shared_informer.go:240] Waiting for caches to sync for ReplicaSet
I0604 21:55:50.827498       1 shared_informer.go:247] Caches are synced for tokens 
I0604 21:55:50.832928       1 controllermanager.go:554] Started "ttl"
I0604 21:55:50.833063       1 ttl_controller.go:121] Starting TTL controller
I0604 21:55:50.833076       1 shared_informer.go:240] Waiting for caches to sync for TTL
I0604 21:55:50.843520       1 controllermanager.go:554] Started "root-ca-cert-publisher"
I0604 21:55:50.843540       1 publisher.go:98] Starting root CA certificate configmap publisher
I0604 21:55:50.843692       1 shared_informer.go:240] Waiting for caches to sync for crt configmap
I0604 21:55:50.853044       1 controllermanager.go:554] Started "endpoint"
I0604 21:55:50.853069       1 endpoints_controller.go:184] Starting endpoint controller
I0604 21:55:50.853081       1 shared_informer.go:240] Waiting for caches to sync for endpoint
I0604 21:55:50.881429       1 controllermanager.go:554] Started "serviceaccount"
I0604 21:55:50.881491       1 serviceaccounts_controller.go:117] Starting service account controller
I0604 21:55:50.881506       1 shared_informer.go:240] Waiting for caches to sync for service account
I0604 21:55:51.031809       1 controllermanager.go:554] Started "job"
I0604 21:55:51.031831       1 job_controller.go:148] Starting job controller
I0604 21:55:51.031842       1 shared_informer.go:240] Waiting for caches to sync for job
W0604 21:55:51.031832       1 core.go:246] configure-cloud-routes is set, but no cloud provider specified. Will not configure cloud provider routes.
W0604 21:55:51.031859       1 controllermanager.go:546] Skipping "route"
W0604 21:55:51.031866       1 controllermanager.go:533] "attachdetach" is disabled
I0604 21:55:51.181710       1 controllermanager.go:554] Started "replicationcontroller"
I0604 21:55:51.181758       1 replica_set.go:182] Starting replicationcontroller controller
I0604 21:55:51.181766       1 shared_informer.go:240] Waiting for caches to sync for ReplicationController
I0604 21:55:51.946539       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for controllerrevisions.apps
I0604 21:55:51.946628       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for deployments.apps
I0604 21:55:51.946683       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for cronjobs.batch
I0604 21:55:51.946789       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for ingresses.extensions
I0604 21:55:51.946857       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for poddisruptionbudgets.policy
I0604 21:55:51.946939       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for podtemplates
I0604 21:55:51.947000       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for endpointslices.discovery.k8s.io
W0604 21:55:51.947069       1 shared_informer.go:494] resyncPeriod 15h23m41.631502455s is smaller than resyncCheckPeriod 20h44m24.485449763s and the informer has already started. Changing it to 20h44m24.485449763s
I0604 21:55:51.947192       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for horizontalpodautoscalers.autoscaling
I0604 21:55:51.947275       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for helmchartconfigs.helm.cattle.io
I0604 21:55:51.947337       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for leases.coordination.k8s.io
I0604 21:55:51.947405       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for addons.k3s.cattle.io
I0604 21:55:51.947456       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for endpoints
I0604 21:55:51.947529       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for daemonsets.apps
I0604 21:55:51.947590       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for roles.rbac.authorization.k8s.io
I0604 21:55:51.947649       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for rolebindings.rbac.authorization.k8s.io
I0604 21:55:51.947709       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for limitranges
W0604 21:55:51.947742       1 shared_informer.go:494] resyncPeriod 16h32m22.14377626s is smaller than resyncCheckPeriod 20h44m24.485449763s and the informer has already started. Changing it to 20h44m24.485449763s
I0604 21:55:51.947828       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for serviceaccounts
I0604 21:55:51.947888       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for events.events.k8s.io
I0604 21:55:51.947951       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for helmcharts.helm.cattle.io
I0604 21:55:51.948017       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for ingresses.networking.k8s.io
I0604 21:55:51.948154       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for replicasets.apps
I0604 21:55:51.948231       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for statefulsets.apps
I0604 21:55:51.948283       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for jobs.batch
I0604 21:55:51.948342       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for networkpolicies.networking.k8s.io
I0604 21:55:51.948381       1 controllermanager.go:554] Started "resourcequota"
I0604 21:55:51.948415       1 resource_quota_controller.go:273] Starting resource quota controller
I0604 21:55:51.948446       1 shared_informer.go:240] Waiting for caches to sync for resource quota
I0604 21:55:51.948508       1 resource_quota_monitor.go:304] QuotaMonitor running
I0604 21:55:51.968160       1 controllermanager.go:554] Started "disruption"
W0604 21:55:51.968197       1 controllermanager.go:533] "persistentvolume-binder" is disabled
I0604 21:55:51.968242       1 disruption.go:331] Starting disruption controller
I0604 21:55:51.968268       1 shared_informer.go:240] Waiting for caches to sync for disruption
I0604 21:55:51.983563       1 controllermanager.go:554] Started "pvc-protection"
I0604 21:55:51.983732       1 pvc_protection_controller.go:110] Starting PVC protection controller
I0604 21:55:51.983758       1 shared_informer.go:240] Waiting for caches to sync for PVC protection
I0604 21:55:52.082201       1 controllermanager.go:554] Started "horizontalpodautoscaling"
I0604 21:55:52.082307       1 horizontal.go:169] Starting HPA controller
I0604 21:55:52.082331       1 shared_informer.go:240] Waiting for caches to sync for HPA
I0604 21:55:52.235168       1 controllermanager.go:554] Started "daemonset"
W0604 21:55:52.235203       1 controllermanager.go:533] "tokencleaner" is disabled
I0604 21:55:52.235204       1 daemon_controller.go:285] Starting daemon sets controller
W0604 21:55:52.235215       1 controllermanager.go:533] "nodelifecycle" is disabled
I0604 21:55:52.235224       1 shared_informer.go:240] Waiting for caches to sync for daemon sets
W0604 21:55:52.235226       1 controllermanager.go:533] "cloud-node-lifecycle" is disabled
I0604 21:55:52.490067       1 controllermanager.go:554] Started "namespace"
I0604 21:55:52.490117       1 namespace_controller.go:200] Starting namespace controller
I0604 21:55:52.490133       1 shared_informer.go:240] Waiting for caches to sync for namespace
I0604 21:55:52.682358       1 garbagecollector.go:142] Starting garbage collector controller
I0604 21:55:52.682397       1 shared_informer.go:240] Waiting for caches to sync for garbage collector
I0604 21:55:52.682443       1 graph_builder.go:289] GraphBuilder running
I0604 21:55:52.682690       1 controllermanager.go:554] Started "garbagecollector"
I0604 21:55:52.931843       1 controllermanager.go:554] Started "deployment"
I0604 21:55:52.931896       1 deployment_controller.go:153] Starting deployment controller
I0604 21:55:52.931905       1 shared_informer.go:240] Waiting for caches to sync for deployment
I0604 21:55:53.084603       1 controllermanager.go:554] Started "statefulset"
I0604 21:55:53.084701       1 stateful_set.go:146] Starting stateful set controller
I0604 21:55:53.084724       1 shared_informer.go:240] Waiting for caches to sync for stateful set
I0604 21:55:53.235830       1 controllermanager.go:554] Started "cronjob"
I0604 21:55:53.235894       1 cronjob_controller.go:96] Starting CronJob Manager
I0604 21:55:53.485138       1 certificate_controller.go:118] Starting certificate controller "csrsigning-kubelet-serving"
I0604 21:55:53.485181       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kubelet-serving
I0604 21:55:53.485188       1 dynamic_serving_content.go:130] Starting csr-controller::/data/server/tls/client-ca.crt::/data/server/tls/client-ca.key
I0604 21:55:53.485869       1 certificate_controller.go:118] Starting certificate controller "csrsigning-kubelet-client"
I0604 21:55:53.485897       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kubelet-client
I0604 21:55:53.485939       1 dynamic_serving_content.go:130] Starting csr-controller::/data/server/tls/client-ca.crt::/data/server/tls/client-ca.key
I0604 21:55:53.486545       1 certificate_controller.go:118] Starting certificate controller "csrsigning-kube-apiserver-client"
I0604 21:55:53.486579       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kube-apiserver-client
I0604 21:55:53.486580       1 dynamic_serving_content.go:130] Starting csr-controller::/data/server/tls/client-ca.crt::/data/server/tls/client-ca.key
I0604 21:55:53.487192       1 controllermanager.go:554] Started "csrsigning"
I0604 21:55:53.487290       1 certificate_controller.go:118] Starting certificate controller "csrsigning-legacy-unknown"
I0604 21:55:53.487318       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-legacy-unknown
I0604 21:55:53.487350       1 dynamic_serving_content.go:130] Starting csr-controller::/data/server/tls/client-ca.crt::/data/server/tls/client-ca.key
I0604 21:55:53.534160       1 controllermanager.go:554] Started "csrapproving"
I0604 21:55:53.534259       1 certificate_controller.go:118] Starting certificate controller "csrapproving"
I0604 21:55:53.534286       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrapproving
I0604 21:55:53.583704       1 controllermanager.go:554] Started "csrcleaner"
I0604 21:55:53.584604       1 cleaner.go:82] Starting CSR cleaner controller
I0604 21:55:53.632039       1 shared_informer.go:247] Caches are synced for deployment 
I0604 21:55:53.632089       1 shared_informer.go:247] Caches are synced for job 
I0604 21:55:53.633147       1 shared_informer.go:247] Caches are synced for TTL 
I0604 21:55:53.634413       1 shared_informer.go:247] Caches are synced for certificate-csrapproving 
I0604 21:55:53.635365       1 shared_informer.go:247] Caches are synced for daemon sets 
I0604 21:55:53.643906       1 shared_informer.go:247] Caches are synced for crt configmap 
I0604 21:55:53.645150       1 shared_informer.go:247] Caches are synced for endpoint_slice 
I0604 21:55:53.648666       1 shared_informer.go:247] Caches are synced for resource quota 
I0604 21:55:53.653291       1 shared_informer.go:247] Caches are synced for endpoint 
I0604 21:55:53.655024       1 shared_informer.go:247] Caches are synced for endpoint_slice_mirroring 
I0604 21:55:53.664875       1 shared_informer.go:247] Caches are synced for GC 
I0604 21:55:53.668492       1 shared_informer.go:247] Caches are synced for disruption 
I0604 21:55:53.668525       1 disruption.go:339] Sending events to api server.
I0604 21:55:53.680481       1 shared_informer.go:247] Caches are synced for ClusterRoleAggregator 
I0604 21:55:53.681645       1 shared_informer.go:247] Caches are synced for service account 
I0604 21:55:53.682001       1 shared_informer.go:247] Caches are synced for ReplicationController 
I0604 21:55:53.683151       1 shared_informer.go:247] Caches are synced for HPA 
I0604 21:55:53.684464       1 shared_informer.go:247] Caches are synced for PVC protection 
I0604 21:55:53.685604       1 shared_informer.go:247] Caches are synced for stateful set 
I0604 21:55:53.685664       1 shared_informer.go:247] Caches are synced for certificate-csrsigning-kubelet-serving 
I0604 21:55:53.686842       1 shared_informer.go:247] Caches are synced for certificate-csrsigning-kubelet-client 
I0604 21:55:53.686907       1 shared_informer.go:247] Caches are synced for certificate-csrsigning-kube-apiserver-client 
I0604 21:55:53.688163       1 shared_informer.go:247] Caches are synced for PV protection 
I0604 21:55:53.688185       1 shared_informer.go:247] Caches are synced for certificate-csrsigning-legacy-unknown 
I0604 21:55:53.690290       1 shared_informer.go:247] Caches are synced for namespace 
I0604 21:55:53.694425       1 controller.go:609] quota admission added evaluator for: replicasets.apps
I0604 21:55:53.694868       1 shared_informer.go:247] Caches are synced for ReplicaSet 
I0604 21:55:53.698059       1 event.go:291] "Event occurred" object="kube-system/coredns" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScalingReplicaSet" message="Scaled up replica set coredns-854c77959c to 1"
I0604 21:55:53.996138       1 shared_informer.go:240] Waiting for caches to sync for resource quota
I0604 21:55:53.996202       1 shared_informer.go:247] Caches are synced for resource quota 
I0604 21:55:54.142095       1 shared_informer.go:240] Waiting for caches to sync for garbage collector
E0604 21:55:54.190863       1 clusterroleaggregation_controller.go:181] view failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "view": the object has been modified; please apply your changes to the latest version and try again
E0604 21:55:54.191008       1 clusterroleaggregation_controller.go:181] admin failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "admin": the object has been modified; please apply your changes to the latest version and try again
E0604 21:55:54.195596       1 clusterroleaggregation_controller.go:181] edit failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "edit": the object has been modified; please apply your changes to the latest version and try again
E0604 21:55:54.197994       1 clusterroleaggregation_controller.go:181] admin failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "admin": the object has been modified; please apply your changes to the latest version and try again
I0604 21:55:54.291427       1 event.go:291] "Event occurred" object="kube-system/coredns-854c77959c" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: coredns-854c77959c-qtr54"
I0604 21:55:54.342313       1 shared_informer.go:247] Caches are synced for garbage collector 
I0604 21:55:54.382631       1 shared_informer.go:247] Caches are synced for garbage collector 
I0604 21:55:54.382665       1 garbagecollector.go:151] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
time="2021-06-04T21:55:54.455175576Z" level=error msg="No InternalIP found for node t320.xxx"
time="2021-06-04T21:55:54.460399269Z" level=error msg="No InternalIP found for node t320.xxx"
time="2021-06-04T21:55:54.474040009Z" level=info msg="Updated coredns node hosts entry [192.107.92.226 t320.xxx]"
time="2021-06-04T21:55:54.506725940Z" level=warning msg="Unable to remove node password: secrets \"t320.xxx.node-password.k3s\" not found"
time="2021-06-04T21:55:54.510141597Z" level=info msg="Removed coredns node hosts entry [192.107.92.226 t320.xxx]"
time="2021-06-04T21:55:54.521277608Z" level=error msg="No InternalIP found for node t320.xxx"
time="2021-06-04T21:55:54.525072518Z" level=error msg="No InternalIP found for node t320.xxx"
time="2021-06-04T21:55:54.552516661Z" level=info msg="Updated coredns node hosts entry [192.104.140.162 t320.xxx]"
W0604 21:55:57.752695       1 lease.go:233] Resetting endpoints for master service "kubernetes" to [10.42.0.8]

The host cluster is a free RKE cluster:

k get nodes -o wide
NAME                             STATUS   ROLES               AGE   VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION         CONTAINER-RUNTIME
rke-master-01.xxx   Ready    controlplane,etcd   15d   v1.20.6   192.168.140.194   <none>        Debian GNU/Linux 10 (buster)   4.19.0-16-amd64        docker://20.10.6
t320.xxx            Ready    worker              15d   v1.20.6   192.168.140.189   <none>        Debian GNU/Linux 10 (buster)   5.10.0-0.bpo.5-amd64   docker://20.10.6

Any Idea what I can do to connect to the vcluster?

FabianKramm commented 3 years ago

@randyguptaIKEA thanks for creating this issue! Are you using v0.2.0 (check with vcluster --version)? If yes, please do not use --update-current with vcluster connect as this has an issue in that version.

randyguptaIKEA commented 3 years ago

I'm running: 

vcluster --version
vcluster version 0.2.0
randyguptaIKEA commented 3 years ago

I'ts now working for me when I omit --update-current. Thank you very much @FabianKramm