loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.35k stars 406 forks source link

vcluster connection failed to sync nodes #263

Closed vumdao closed 2 years ago

vumdao commented 2 years ago

vcluster version

$ vcluster --version
vcluster version 0.5.0-beta.0

Steps:

  1. Create loadbalancer
    apiVersion: v1
    kind: Service
    metadata:
    name: vcluster-loadbalancer
    namespace: dev8
    spec:
    selector:
    app: vcluster
    release: dev8
    ports:
    - name: https
      port: 443
      targetPort: 8443
      protocol: TCP
    type: LoadBalancer
  2. Check the service
    $ kf get svc
    NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)         AGE
    vcluster-loadbalancer   LoadBalancer   172.20.111.146   a0b6ab5891bfe4d82a01b8c07eb7a238-104005730.us-west-2.elb.amazonaws.com   443:30924/TCP   65s
  3. Update values.yaml with enable --sync=nodes
    syncer:
    extraArgs:
    - --tls-san=a0b6ab5891bfe4d82a01b8c07eb7a238-104005730.us-west-2.elb.amazonaws.com
    - --sync=nodes
  4. Create vcluster
    $ vcluster create dev8 -n dev8 -f values.yaml 
    [info]   execute command: helm upgrade dev8 vcluster --repo https://charts.loft.sh --version 0.5.0-beta.0 --kubeconfig /tmp/295948009 --namespace dev8 --install --repository-config='' --values /tmp/2738491821 --values values.yaml
    [done] √ Successfully created virtual cluster dev8 in namespace dev8. Use 'vcluster connect dev8 --namespace dev8' to access the virtual cluster
  5. Check Pod
    $ kf get pod
    NAME                                            READY   STATUS    RESTARTS   AGE
    coredns-6ff7df994d-44pqg-x-kube-system-x-dev8   1/1     Running   0          49s
    dev8-0                                          2/2     Running   1          3m35s
  6. Connect cluster
    $ vcluster connect dev8 -n dev8 --server=https://a0b6ab5891bfe4d82a01b8c07eb7a238-104005730.us-west-2.elb.amazonaws.com
    [done] √ Virtual cluster kube config written to: ./kubeconfig.yaml. You can access the cluster via `kubectl --kubeconfig ./kubeconfig.yaml get namespaces`
  7. Get nodes
    $ kf get node
    NAME                                         STATUS   ROLES    AGE   VERSION
    ip-10-0-118-137.us-west-2.compute.internal   Ready    <none>   30m   v1.21.6
    ip-10-0-145-146.us-west-2.compute.internal   Ready    <none>   29m   v1.21.6
    ip-10-0-149-193.us-west-2.compute.internal   Ready    <none>   29m   v1.21.6
    ip-10-0-150-158.us-west-2.compute.internal   Ready    <none>   28m   v1.21.6
    ip-10-0-188-69.us-west-2.compute.internal    Ready    <none>   29m   v1.21.6
  8. Export kubeconfig
    $ export KUBECONFIG=./kubeconfig.yaml
  9. Issue of syncing nodes
    
    $ kf get node
    No resources found
    $ kf get pod -A
    NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
    kube-system   coredns-6ff7df994d-44pqg   0/1     Pending   0          6m6s
  10. Check Args
    $ kubectl get statefulset -n dev8 dev8 -o jsonpath='{.spec.template.spec.containers[?(@.name=="syncer")].args}'
    ["--name=dev8","--tls-san=a0b6ab5891bfe4d82a01b8c07eb7a238-104005730.us-west-2.elb.amazonaws.com","--sync=nodes"]
  11. Syncer logs syncer.log
  12. vcluster logs vcluster.log
matskiv commented 2 years ago

As discussed in Slack, adding the --sync-all-nodes flag to .syncer.extraArgs in values.yaml solved the problem. Feel free to reopen the issue if I misunderstood something.