loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.81k stars 423 forks source link

Argocd add vcluster with custom domain #893

Closed satishweb closed 10 months ago

satishweb commented 1 year ago

What happened?

I am facing an issue with vcluster helm chart version 0.13.0 and Argo CD. I tried to add cluster to ArgoCD using kubeconfig and it errored out.

I spent some time debugging this. So far, I think that vcluster/k3s is not adding custom domain in the certificate SAN list.

What did you expect to happen?

ArgoCD certificate verification should be successful with custom domain name that is known to vcluster via --tls-san argument

  extraArgs:
      - --tls-san=my.domain.com
  syncer:
    kubeConfigContextName: my-domain
    extraArgs:
      - --tls-san=my.domain.com
      - --out-kube-config-server=https://my.domain.com

How can we reproduce it (as minimally and precisely as possible)?

Vcluster helm chart values.yaml

vcluster:
    extraArgs:
      - --tls-san=my.domain.com
  syncer:
    kubeConfigContextName: my-domain
    extraArgs:
      - --tls-san=my.domain.com
      - --out-kube-config-server=https://my.domain.com
  ingress:
    enabled: true
    ingressClassName: internal
    host: "my.domain.com"
    annotations:
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
      kubernetes.io/ingress.allow-http: "false"

vcluster args

args:
        - '-c'
        - >-
          /bin/k3s server --write-kubeconfig=/data/k3s-config/kube-config.yaml
          --data-dir=/data
          --disable=traefik,servicelb,metrics-server,local-storage,coredns
          --disable-network-policy --disable-agent --disable-cloud-controller
          --flannel-backend=none --disable-scheduler
          --kube-controller-manager-arg=controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl
          --kube-apiserver-arg=endpoint-reconciler-type=none
          --service-cidr=$(SERVICE_CIDR)
          --tls-san=my.domain.com && true
      command:
        - /bin/sh

syncer args

args:
        - '--name=my-domain'
        - '--service-account=vc-workload-my-domain'
        - '--kube-config-context-name=my-domain'
        - '--leader-elect=false'
        - '--tls-san=my.domain.com'
        - '--sync=hoststorageclasses'
        - '--sync=ingresses'
        - '--tls-san=my.domain.com'
        - >-
          --out-kube-config-server=https://my.domain.com

vcluster cli error

$: vcluster connect my-domain-com -n my-domain-com
fatal  unexpected server in kubeconfig: https://my.domain.com

EDIT: below command worked
$: vcluster connect my-domain-com -n my-domain-com --server my.domain.com
done √ Switched active kube context to vcluster_my-domain-com_my-domain-com_MY-HOST-CLUSTER-NAME
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster

EDIT 2: connection was successful but kubectl command fails with same error as argocd
$: kubectl get ns
Unable to connect to the server: x509: certificate is valid for ingress.local, not my.domain.com

Adding --insecure to the vcluster conect command made that error go away. This is not expected, I should not be required to do an insecure connection.

sts logs

Defaulted container "vcluster" out of: vcluster, syncer
time="2022-12-30T12:30:48Z" level=info msg="Starting k3s v1.25.3+k3s1 (f2585c16)"
time="2022-12-30T12:30:48Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2022-12-30T12:30:48Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2022-12-30T12:30:48Z" level=info msg="Database tables and indexes are up to date"
time="2022-12-30T12:30:48Z" level=info msg="Kine available at unix://kine.sock"
time="2022-12-30T12:30:48Z" level=info msg="generated self-signed CA certificate CN=k3s-client-ca@1672403448: notBefore=2022-12-30 12:30:48.722284863 +0000 UTC notAfter=2032-12-27 12:30:48.722284863 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=system:kube-controller-manager signed by CN=k3s-client-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=system:kube-scheduler signed by CN=k3s-client-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=system:apiserver,O=system:masters signed by CN=k3s-client-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=k3s-cloud-controller-manager signed by CN=k3s-client-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="generated self-signed CA certificate CN=k3s-server-ca@1672403448: notBefore=2022-12-30 12:30:48.732061476 +0000 UTC notAfter=2032-12-27 12:30:48.732061476 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-server-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="generated self-signed CA certificate CN=k3s-request-header-ca@1672403448: notBefore=2022-12-30 12:30:48.734247807 +0000 UTC notAfter=2032-12-27 12:30:48.734247807 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=system:auth-proxy signed by CN=k3s-request-header-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="generated self-signed CA certificate CN=etcd-server-ca@1672403448: notBefore=2022-12-30 12:30:48.742083714 +0000 UTC notAfter=2032-12-27 12:30:48.742083714 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=etcd-server signed by CN=etcd-server-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=etcd-client signed by CN=etcd-server-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="generated self-signed CA certificate CN=etcd-peer-ca@1672403448: notBefore=2022-12-30 12:30:48.745228623 +0000 UTC notAfter=2032-12-27 12:30:48.745228623 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=etcd-peer signed by CN=etcd-peer-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=warning msg="dynamiclistener [::]:6443: no cached certificate available for preload - deferring certificate load until storage initialization or first client request"
time="2022-12-30T12:30:48Z" level=info msg="Active TLS secret / (ver=) (count 11): map[listener.cattle.io/cn-10.20.9.221:10.20.9.221 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.20.0.1:172.20.0.1 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-my.domain.com-963ab1:my.domain.com listener.cattle.io/cn-my-domain-0:my-domain-0 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=DF8D3CE5330E72F743A187AB7F5C884FB6E39EC2]"
time="2022-12-30T12:30:48Z" level=info msg="Tunnel server egress proxy mode: agent"
time="2022-12-30T12:30:48Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
time="2022-12-30T12:30:48Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/data/server/tls/temporary-certs --client-ca-file=/data/server/tls/client-ca.crt --egress-selector-config-file=/data/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --endpoint-reconciler-type=none --etcd-servers=unix://kine.sock --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/data/server/tls/server-ca.crt --kubelet-client-certificate=/data/server/tls/client-kube-apiserver.crt --kubelet-client-key=/data/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/data/server/tls/client-auth-proxy.crt --proxy-client-key-file=/data/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/data/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/server/tls/service.key --service-account-signing-key-file=/data/server/tls/service.key --service-cluster-ip-range=172.20.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/data/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/data/server/tls/serving-kube-apiserver.key"
time="2022-12-30T12:30:48Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/data/server/cred/controller.kubeconfig --authorization-kubeconfig=/data/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/data/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/data/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/data/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/data/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/data/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/data/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/data/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/data/server/tls/server-ca.key --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/data/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/data/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/data/server/tls/service.key --service-cluster-ip-range=172.20.0.0/16 --use-service-account-credentials=true"
time="2022-12-30T12:30:48Z" level=info msg="Waiting for API server to become available"
I1230 12:30:48.889870       7 server.go:581] external host was not specified, using 10.20.9.221
I1230 12:30:48.890346       7 server.go:171] Version: v1.25.3+k3s1
I1230 12:30:48.890446       7 server.go:173] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
time="2022-12-30T12:30:48Z" level=info msg="Server node token is available at /data/server/token"
time="2022-12-30T12:30:48Z" level=info msg="To join server node to cluster: k3s server -s https://10.20.9.221:6443 -t ${SERVER_NODE_TOKEN}"
time="2022-12-30T12:30:48Z" level=info msg="Agent node token is available at /data/server/agent-token"
time="2022-12-30T12:30:48Z" level=info msg="To join agent node to cluster: k3s agent -s https://10.20.9.221:6443 -t ${AGENT_NODE_TOKEN}"
time="2022-12-30T12:30:48Z" level=info msg="Wrote kubeconfig /data/k3s-config/kube-config.yaml"
time="2022-12-30T12:30:48Z" level=info msg="Run: k3s kubectl"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=my-domain-com-0 signed by CN=k3s-server-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="certificate CN=system:node:my-domain-com-0,O=system:nodes signed by CN=k3s-client-ca@1672403448: notBefore=2022-12-30 12:30:48 +0000 UTC notAfter=2023-12-30 12:30:48 +0000 UTC"
time="2022-12-30T12:30:48Z" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
time="2022-12-30T12:30:48Z" level=info msg="Handling backend connection request [my-domain-com-0]"
I1230 12:30:49.216778       7 shared_informer.go:255] Waiting for caches to sync for node_authorizer
I1230 12:30:49.218023       7 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I1230 12:30:49.218072       7 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I1230 12:30:49.219822       7 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I1230 12:30:49.219870       7 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W1230 12:30:49.254585       7 genericapiserver.go:656] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
I1230 12:30:49.256336       7 instance.go:261] Using reconciler: none
I1230 12:30:49.462413       7 instance.go:574] API group "internal.apiserver.k8s.io" is not enabled, skipping.

Anything else we need to know?

Argocd secret for cluster

apiVersion: v1
kind: Secret
metadata:
  name: my-domain-com-cluster
  labels:
    argocd.argoproj.io/secret-type: cluster
data:
  # after base64 decode
  name: my-domain-com
  server: https://my.domain.com
  config: |
    {
      "bearerToken": "XXXXX",
      "tlsClientConfig": {
        "insecure": false,
        "caData": "XXXX",
        "keyData": "XXXX",
        "certData": "XXXX"
      }
    }

Note: This was generated based on the vcluster secret that contains the kubeconfig

Host cluster Kubernetes version

``` $ kubectl version Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:28:30Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"darwin/arm64"} Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.13-eks-fb459a0", GitCommit:"55bd5d5cb7d32bc35e4e050f536181196fb8c6f7", GitTreeState:"clean", BuildDate:"2022-10-24T20:35:40Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"} ```

Host cluster Kubernetes distribution

``` 1.24 ```

vlcuster version

``` 0.13.0 ```

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

``` k3s ```

OS and Arch

``` OS: linux Arch: amd64 ```
FabianKramm commented 1 year ago

Hello @satishweb ! Thanks for creating this issue, when using an ingress without SSL-passthrough, you need to make sure the ingress is signing the certificate not only for itself but also for the vcluster, in your case it seems it returns the default ingress certificate which is insecure by default. Also you will need to use a service account token as described here: https://www.vcluster.com/docs/operator/external-access#ingress-without-ssl-passthrough

satishweb commented 1 year ago

hi @FabianKramm , Thanks for the quick response. I am using SSL Passthrough. Below is the error that I get when I use kubectl: Unable to connect to the server: x509: certificate is valid for ingress.local, not my.domain.com

If you think that this error happens only when SSL passthrough is not enabled, I can redo the test to confirm it. Please let me know.

FabianKramm commented 1 year ago

@satishweb there is probably a problem with the SSL passthrough then, what ingress controller are you using? This error message shows thats the ingress certificate (hence it says ingress.local) which means that the ingress controller is doing TLS termination and does not passthrough.

alexandradragodan commented 1 year ago

Heyaa, @satishweb 👋

Have you, by any chance, managed to check if your ingress certificate is validated on your desired domain?

Thanks! Waiting on your confirmation before closing this issue 🌞

satishweb commented 1 year ago

I haven't got the time to test this yet. I have added this to my personal to-do list. I will respond as soon as I am done with the testing. For now, I am using the --insecure flag for vcluster cli.

eldad87 commented 1 year ago

I got the same issue

Deploy:

$ helm upgrade --install my-vcluster vcluster \
  --values tenant/virtual_cluster_provider/vcluster/chart/values.yaml \
  --repo https://charts.loft.sh/ \
  --namespace my-vcluster \
  --repository-config=''

values.yaml

ingress:
  enabled: true
  host: my-vcluster.account.local

isolation:
  enabled: true

syncer:
  extraArgs:
  - --out-kube-config-server=https://my-vcluster.account.local/

I used the following secret

$ kubectl get secret vc-my-vcluster -n my-vcluster --template={{.data.config}} | base64 --decode

Error

$ kubectl get pods
Unable to connect to the server: x509: certificate is valid for ingress.local, not my-vcluster.account.local
satishweb commented 1 year ago

Here is the command that works right now (not ideal but works): vcluster connect --insecure my-vcluster --server my-tls-san-fqdn-address.domain.com --service-account=kube-system/developer --cluster-role=cluster-admin

rohantmp commented 1 year ago

Hi, could everyone please add the details of the ingress controller they are using? Thanks! @satishweb @eldad87

rohantmp commented 1 year ago

Just to be super clear: "Enabling SSL Passthrough" also means passing a custom flag to the nginx ingress controller. This may differ for other controllers.

satishweb commented 1 year ago

Hi, could everyone please add the details of the ingress controller they are using? Thanks! @satishweb @eldad87

Nginx ingress controller with pass through enabled.

reski-rukmantiyo commented 11 months ago

hi All,

I try to use SSL Passthrough in my ingress, but still got this _Unable to connect to the server: tls: failed to verify certificate: x509: certificate is valid for ingress.local, not cluster1.co.id_If I disable insecure then this is working, i can access my cluster. But this is something that my internal dont want.

Therefore, i want to change ingress.local certificate, and replace with cluster1.ebays.co.id. Is it possible?

rohantmp commented 11 months ago

Therefore, i want to change ingress.local certificate, and replace with cluster1.ebays.co.id. Is it possible?

please see the --tls-sans flag here: https://www.vcluster.com/docs/using-vclusters/access

reski-rukmantiyo commented 11 months ago

Hi @rohantmp

I've tried to put following values into values.yml.

syncer:
  extraArgs:
  - --tls-san=cluster1.co.id

Then i create cluster using following script

vcluster create $NAME -n $NS --connect=false --upgrade -f values.yml 

but still say the same....still lookup for ingress.local

rohantmp commented 11 months ago

Hey, this might mean that you haven't successfully enabled ingress passthrough. Could explain the steps you tried for this?

if I disable insecure then this is working, i can access my cluster.

What does it mean to "disable insecure"?

reski-rukmantiyo commented 11 months ago

Hey, this might mean that you haven't successfully enabled ingress passthrough. Could explain the steps you tried for this?

if I disable insecure then this is working, i can access my cluster.

What does it mean to "disable insecure"?

Sorry, i mean enable insecure.

satishweb commented 11 months ago

I faced the same issue and I am still using ssl passthrough with insecure flag set as false. I did not test the new version though.

reski-rukmantiyo commented 11 months ago

I faced the same issue and I am still using ssl passthrough with insecure flag set as false. I did not test the new version though.

Which version are you using right now? I'm using

root@test-for-bq:~# vcluster version
vcluster version 0.17.1
satishweb commented 11 months ago

I am still on a very old version, based on your comments, for the ssl-passthrough option, there seems to be no much difference between current version and this version: 0.13.0

mpetason commented 10 months ago

To follow up on what @rohantmp is saying, this means modifying the nginx configuration and adding a flag to the container args:

    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
        - --election-id=ingress-nginx-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --enable-ssl-passthrough <<<<<<
rohantmp commented 10 months ago

This seems to be a configuration issue, so I'm closing this for now. Feel free to re-open if you believe that was a mistake :)

dvaldivia commented 3 months ago

This is still happening on 0.19.7, I had a single syncer v0.10.2 cluster, which I torn down, then recreated with the latest vcluster binary, but using a syncer.replicas=3 on the values.yaml, and now using the same ingress to reach the vcluster k8s admin returns this error if I try to connect via kubeconfig or plain vcluster connect, both fail, excelt that the kubectl --kubeconfig works one out of every three times which leads me to belive the HA mode, has a problem.

I have ssl-passthrough, I guarantee it as this cluster was working fine on v0.10.2

Here are three subsequent tries, only 2 succeeded

➜ kubectl --kubeconfig ./kubeconfig.yaml get namespaces
NAME              STATUS   AGE
kube-system       Active   18m
kube-public       Active   18m
kube-node-lease   Active   18m
default           Active   18m
 dev-vcluster git:(master) ✗ 
➜ kubectl --kubeconfig ./kubeconfig.yaml get namespaces
Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority
 dev-vcluster git:(master) ✗ 
➜ kubectl --kubeconfig ./kubeconfig.yaml get namespaces
NAME              STATUS   AGE
kube-system       Active   18m
kube-public       Active   18m
kube-node-lease   Active   18m
default           Active   18m

again, nothing changed in my host cluster between version changes