Kubeconfig data gives me TLS error

emoreth commented 1 month ago

What happened?

I was unable to pinpoint where this changed in the code. But it happened from alpha.10 to alpha.11.

The vc-CLUSTER_NAME secret used to have (alpha.10) these 4 keys:

config
certificate-authority
client-certificate
client-key

After v.11 it now it has

config
token
client-certificate
client-key

The new token key is empty and the certificate-authority was removed.

In the config key, the certificate-authority-data was removed from clusters[0] and now I get TLS error when trying to connect to my vcluster with

Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority

What did you expect to happen?

Be able to connect to the cluster

How can we reproduce it (as minimally and precisely as possible)?

Migrate from alpha.10 to alpha.11 (still present on beta.2)

Anything else we need to know?

No response

Host cluster Kubernetes version

```console Client Version: v1.30.0 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.8-eks-a737599 ```

vcluster version

```console $ vcluster --version # paste output here ```

VCluster Config

``` # My vcluster.yaml / values.yaml here ```

ThomasK33 commented 1 month ago

Hey @emoreth, thanks for opening an issue.

That seems strange, given that vCluster creates or patches the secret on startup and that the CA cert was not removed from the secret's data in the code.

https://github.com/loft-sh/vcluster/blob/d524065f05d79ae009a6066a0a8d240198cb907b/pkg/util/kubeconfig/kubeconfig.go#L80-L84

Could you try one of the more recent betas?

If it still doesn't work, would you mind including your vCluster version and vcluster.yaml file?

emoreth commented 1 month ago

When I tried yesterday, that still happened on beta.2. I can try again with beta.3.

I'm was not using vcluster CLI for this test, I was running a pure helm deployment.

I see the code you highlighted, but I was not using the exportKubeConfig.secret setting, and even when using that, the result was the same.

Since the certificate-authority was not present on the vc-CLUSTER_NAME secret, I guess is that something is failing while creating that secret, not while populating the kubeconfig.

ThomasK33 commented 1 month ago

Oh, thanks for that tidbit of info.

However, I will need more context and instructions for reproducing this issue.

What's your host cluster version?
What's the vcluster.yaml/values.yaml that you're passing to helm?
What's the helm command you're using to install the vCluster?

Can you provide a list of commands you run that result in the secret not containing the CA data?

When I run the below steps locally, the secret contains the CA data.

vcluster on  main [$] via 🐳 orbstack via 🐹 v1.23.2
❯ kubectl create namespace team-x
namespace/team-x created

vcluster on  main [$] via 🐳 orbstack via 🐹 v1.23.2
❯ helm upgrade --install my-vcluster vcluster --repo https://charts.loft.sh --namespace team-x --version 0.21.0-beta.3
Release "my-vcluster" does not exist. Installing it now.
NAME: my-vcluster
LAST DEPLOYED: Tue Oct 15 17:20:09 2024
NAMESPACE: team-x
STATUS: deployed
REVISION: 1
TEST SUITE: None

vcluster on  main [$] via 🐳 orbstack via 🐹 v1.23.2 took 3s
❯ kubectl get secret -n team-x vc-my-vcluster -o yaml
apiVersion: v1
data:
  certificate-authority: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJZE5xS2swM1lFQ2d3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRFd01UVXhOVEl3TXpCYUZ3MHpOREV3TVRNeE5USXdNekJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURxS1JHTzVmL0s2SHoxS0VYcVdsa2VLN2Jva3V0UGkyNGJSSDI0MHE2ZFpmV1QrU1hLaG9ja0krUU0KUFRMMXZiNUlQelFmMmVEQm9lTTRGRktrc0xKa055RTVuWGY5ajlkd2p4UjIrYWZmSit2Q1h5a3dMUHg2d0RZbgpCK1JpSVl0aHR6bFZqbllQYlNmV3RJMG5JMmZGWnM4cG5JS0FSVVBGNTFGOVNIRnI5VnRDcmNudGJxRlgvMHBHCmszUkN1aDZucDk0R3d0eGxGaytEcVVEZXhLcWx1eEtRVmk2WXVpaE5DeE9kRGJOYVhjSlRlSHZicVdXMTZEREgKS1QxeDRobVhCR2dGcThicGlzNFc2OWplbkxvTVVpdkFYN2N0SjNEQ0x6dnY1S0pHUjVEQVNKeUVDYUpIa1pHUAovNTh5TGhCRTUzT05GQXNTWGg0ZVo4MHlWRzYvQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRbFI0SXNhYzVPa2pkYWRnSVFmNXFOdERXUC9UQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQVZxUWVkRUQzNQpLY1ZWVVltQ3FzWkRlU09KdkdESG5qWHJaK21YVWs4ZlF0VjJSalBmdVpKZWVXMFh0ckcyeHgxVC9OamFYeDEzCmJRMGVTK0hIdUw3dStBVHA2a056ekdWNU5qRFdWUmpHVnE5WHJnSVJqR2FNb2RuSG9CRk8vdVM0YlR0bmswMEUKQ3dBaWxSU1p4V2pEb081UktwbzZvWHpmZzVQOG1aaTJIS2dNczVzV2U1d3ZPSGpBNXZvenZwSHo4OExSU3ltRQovN3BEb3VzcjNnUDY0L09ScW4ydU9ZNVFMak9xN3NaRktkWi95TU40UEEya1c2azBvZ3VUanhWRTQwSFcrbmZiCmN0L3FLbDdPdi96MFJRMzRpeTVmVFBYbTlzaFpzRldMZjdPVllnY3N1NEM1c1JEK1JIRi9iRXVaZkY4VGUrWHYKMHkxcW96TlpMMVpzCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  client-certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZ0F3SUJBZ0lJVCtuTDErcU9ZY1V3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRFd01UVXhOVEl3TXpCYUZ3MHpOREV3TVRNeE5USXdNekZhTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTA3NlJwbjNqaXV5WnAwTU0KWS9neGtMZ25EUFNxZGxnUlVPaEh1NktnWDBWRnp5cXJpSTMybStsREgvS1JpWE1LemhIeWpCazVBdUk3ZmdlWQpNZUFpRys4dHJoWVdXeFU3U0FCKzhjVys4ZE9FZlVlcmV0TU1PaER4Lzc2SktrQnJLbjZiN05WR3ZkSTgxaVVpCkxSRkdWbkRUOVNEa212WHJFZXhTMGVyTlFLOU9lcU5kL00vcVVNSnRsQzljQmxrMDRtSi8rRVk4REdzUHVEck4KSHVJekdxenE5QkdIb1oyN2h6UG1US1ZXbTVtZmJoY0pGeVhNU0xzeC9Td1pTRDVBVlpTWFNOV0pya2dyYjZYUwoyRHFHQmJGc2drenEyVGNtSkV4RUNpMFVaMkh6SmtYc081S3l6cmNIMTRJS1BIeEZTeXordnVPcVEwT2labERYCjN2QVRrd0lEQVFBQm8xWXdWREFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RBWURWUjBUQVFIL0JBSXdBREFmQmdOVkhTTUVHREFXZ0JRbFI0SXNhYzVPa2pkYWRnSVFmNXFOdERXUAovVEFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBS1dNNFkyRTVxOU5qQngxMWx4NktTSGZET1ZYRDlVc0lybXovCkF4bDAxaVdiU1dZaHorblBFNjA3MFhqU0VWR0JlS1lpZ3lKblVsMVc0SXEzVmVPZjZ6SVhweDlTUmZqUTU4cisKTXpmVmhYbU54U1BnWnA5aFVZZW1sTHhvODJBa1AzM0dpNmJ3RVpacmpneHcyWUZ0dVhQQS9LelcyN2I2L3FHMgpQNmo0TWpQWUVuK3RRN3Q5VWYwK3p5dUExc2hoRnJmenV3cGZEMVVDL3piUndGUTc3ckZ4Ylh3am9zWmhkR2U5Ci9obzVMNUlodWR2QlBRamVKVjFhNEx5bUJ5S25IQjJ2RDhQWDJKWjNyb0FtMkprRVBxb1VsMWVlWUwrbmpLdXkKelNMOUhiZmRrUzlrcVV0QzMvdVJ5dkt3M2NOTE5xRXdlMkpBNVRQdDJFS29xRkRGcHc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  client-key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBMDc2UnBuM2ppdXlacDBNTVkvZ3hrTGduRFBTcWRsZ1JVT2hIdTZLZ1gwVkZ6eXFyCmlJMzJtK2xESC9LUmlYTUt6aEh5akJrNUF1STdmZ2VZTWVBaUcrOHRyaFlXV3hVN1NBQis4Y1crOGRPRWZVZXIKZXRNTU9oRHgvNzZKS2tCcktuNmI3TlZHdmRJODFpVWlMUkZHVm5EVDlTRGttdlhyRWV4UzBlck5RSzlPZXFOZAovTS9xVU1KdGxDOWNCbGswNG1KLytFWThER3NQdURyTkh1SXpHcXpxOUJHSG9aMjdoelBtVEtWV201bWZiaGNKCkZ5WE1TTHN4L1N3WlNENUFWWlNYU05XSnJrZ3JiNlhTMkRxR0JiRnNna3pxMlRjbUpFeEVDaTBVWjJIekprWHMKTzVLeXpyY0gxNElLUEh4RlN5eit2dU9xUTBPaVpsRFgzdkFUa3dJREFRQUJBb0lCQUFtQlhveDR6a1FvUnVXSgpUdjhWM3pPOThKTFdVT2lySDJ2Z1lZZjh6dkkzNitCVzdBaGZSeGpNcWVTL1FPVklzczE1Wk1nQlFkdnRSL3BxCllOeXNVUnVyNndNTG5zbEw5TE5Rd21JY0hHeENzSW5uTTcxNXgreTdZZ0czRGZWRm13bjNVUXlhcm1oTERjd1gKTXFlUFJQY1FiWk50ZkpXd1FITTNocWhCd0NFTktMTmNEcndYRE5mZmtPVWNkanNBamF4UUE2dlh2UzN2OFpIZQprRTNOZ3p1amdVcE10cjZEMzAveGpKQlhoWUFEbkxyaFJnY0RIaStwRW03ZDVSSEV2TU5zV3c0V3RVMlAxQ3YzCjdpZG5wTGdmb1J5Z1cwVHJnZUFpYllUbTFNSVdwZ001VDlqdzFwcm95T3hKN2oyVVBHSlNKd1VqazdEeGxPUWMKNENlRlRWRUNnWUVBNTRNU1RMVGNOQkZaeEZDbzc1aENiNnlKSVlTa3MrczlYc1RVZHpLSzVGZ1U3TUJRNjZPYgp6ZlBwRXhYZXlLN2thQm01K2RaSTAzQW9PdE0zV2QrQXZEWW5RVFF0c1ZVWjNKNE56d1NiekNiZFRQNDRZQWwwCmpnZzBBUlFTU0FZYVcxWHJvUmNGNGZnbVBVQ3JoOGtmUFZBSDRDTWlCdVhEaVNqSldrTkhBMFVDZ1lFQTZpUTYKQ3kxWUMvYjU2Uk1HdmV4ZUVIMlR3Z0tQM1p0c3U4SEtDbHliY0JMdWxHaDd4ckpzbjh5bHpKdzZlUlRQODhNNAptNnYzZnB1NlNkV0Z1S2EyVmFSMnNUd0IzMGsrM1RHbWN1cWNHdE44ZWxhSHdwQ0U2dnJVRWtwNnVCekRVZ25YCmdMandack4yNEJOZXV1RWl1K0dFQmlHMVlqdDBHdFZPQ1g2ay9QY0NnWUJKR3h1eE96OTU5YXlxRUdwNFZsVk0KWEdobFBEU2lWL3RtTExPRy9GL2RvVzdpSUF5dGpGbUwzS1RTRVFycnhrWjhGUHVrU0NWeEZ5NnMvTHV5MXFBWgpUdVJ3YzZDbUtlQmp5VWxGQVFpTm5Bd3YzdXRBY0Z2czZldzZGdk42Nk9wend4c0lJSFFNR2tkQSttWHdGL09VCmNldzZ1LzFudmpCVDVKeElzNHYrSFFLQmdRRG9IYVhWTXN5WWYrMWNuRm9RcW1UT0lFLzlBNkV2NWtjekUzaUMKTVRCL09LV0FSdHgrTEovM0EyQlNxWVptUnJXcFNKMno1aFZqVkdxbmk2QVhZRjVlSkw3dUVUa2g3SEQvdzVycApUYUFZRUJUVkN6ZG5NUEkwS1JsWVVMRzhGL2VpT3RPRGJ0UVVYRkc2N2pqNU52dkNHbDFSM3RUOFgwaEkyYkdWClJCRzRIUUtCZ1FDV1Q3VG5tWkp4ajNsZnVHM2VSY1NrRWpUcEp2dVpTUytSL3l6ejRCUUhzSkxIenNZOVh1RmkKeVV3eUo5OWFENGhPdURsWFNVdVR4NzkrN2xWdVF4aDhpbFZ1aWQ5dmM2dmVSVjZzOXFiTHJ4c29BZDRQaExjMApmZzJRNXV5MWE2d08wL2NLWEpoNmloSzRrUjFuL1U1WVRKVi9yMDg1QkZQOVNBa1RNR0FoSUE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
  config: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VSQ1ZFTkRRV1V5WjBGM1NVSkJaMGxKWkU1eFMyc3dNMWxGUTJkM1JGRlpTa3R2V2tsb2RtTk9RVkZGVEVKUlFYZEdWRVZVVFVKRlIwRXhWVVVLUVhoTlMyRXpWbWxhV0VwMVdsaFNiR042UVdWR2R6QjVUa1JGZDAxVVZYaE9WRWwzVFhwQ1lVWjNNSHBPUkVWM1RWUk5lRTVVU1hkTmVrSmhUVUpWZUFwRmVrRlNRbWRPVmtKQlRWUkRiWFF4V1cxV2VXSnRWakJhV0UxM1oyZEZhVTFCTUVkRFUzRkhVMGxpTTBSUlJVSkJVVlZCUVRSSlFrUjNRWGRuWjBWTENrRnZTVUpCVVVSeFMxSkhUelZtTDBzMlNIb3hTMFZZY1Zkc2EyVkxOMkp2YTNWMFVHa3lOR0pTU0RJME1IRTJaRnBtVjFRclUxaExhRzlqYTBrclVVMEtVRlJNTVhaaU5VbFFlbEZtTW1WRVFtOWxUVFJHUmt0cmMweEthMDU1UlRWdVdHWTVhamxrZDJwNFVqSXJZV1ptU2l0MlExaDVhM2RNVUhnMmQwUlpiZ3BDSzFKcFNWbDBhSFI2YkZacWJsbFFZbE5tVjNSSk1HNUpNbVpHV25NNGNHNUpTMEZTVlZCR05URkdPVk5JUm5JNVZuUkRjbU51ZEdKeFJsZ3ZNSEJIQ21zelVrTjFhRFp1Y0RrMFIzZDBlR3hHYXl0RWNWVkVaWGhMY1d4MWVFdFJWbWsyV1hWcGFFNURlRTlrUkdKT1lWaGpTbFJsU0haaWNWZFhNVFpFUkVnS1MxUXhlRFJvYlZoQ1IyZEdjVGhpY0dsek5GYzJPV3BsYmt4dlRWVnBka0ZZTjJOMFNqTkVRMHg2ZG5ZMVMwcEhValZFUVZOS2VVVkRZVXBJYTFwSFVBb3ZOVGg1VEdoQ1JUVXpUMDVHUVhOVFdHZzBaVm80TUhsV1J6WXZRV2ROUWtGQlIycFhWRUpZVFVFMFIwRXhWV1JFZDBWQ0wzZFJSVUYzU1VOd1JFRlFDa0puVGxaSVVrMUNRV1k0UlVKVVFVUkJVVWd2VFVJd1IwRXhWV1JFWjFGWFFrSlJiRkkwU1hOaFl6VlBhMnBrWVdSblNWRm1OWEZPZEVSWFVDOVVRVllLUW1kT1ZraFNSVVZFYWtGTloyZHdjbVJYU214amJUVnNaRWRXZWsxQk1FZERVM0ZIVTBsaU0wUlJSVUpEZDFWQlFUUkpRa0ZSUVZaeFVXVmtSVVF6TlFwTFkxWldWVmx0UTNGeldrUmxVMDlLZGtkRVNHNXFXSEphSzIxWVZXczRabEYwVmpKU2FsQm1kVnBLWldWWE1GaDBja2N5ZUhneFZDOU9hbUZZZURFekNtSlJNR1ZUSzBoSWRVdzNkU3RCVkhBMmEwNTZla2RXTlU1cVJGZFdVbXBIVm5FNVdISm5TVkpxUjJGTmIyUnVTRzlDUms4dmRWTTBZbFIwYm1zd01FVUtRM2RCYVd4U1UxcDRWMnBFYjA4MVVrdHdielp2V0hwbVp6VlFPRzFhYVRKSVMyZE5jelZ6VjJVMWQzWlBTR3BCTlhadmVuWndTSG80T0V4U1UzbHRSUW92TjNCRWIzVnpjak5uVURZMEwwOVNjVzR5ZFU5Wk5WRk1hazl4TjNOYVJrdGtXaTk1VFU0MFVFRXlhMWMyYXpCdlozVlVhbmhXUlRRd1NGY3JibVppQ21OMEwzRkxiRGRQZGk5Nk1GSlJNelJwZVRWbVZGQlliVGx6YUZwelJsZE1aamRQVmxsblkzTjFORU0xYzFKRUsxSklSaTlpUlhWYVprWTRWR1VyV0hZS01Ia3hjVzk2VGxwTU1WcHpDaTB0TFMwdFJVNUVJRU5GVWxSSlJrbERRVlJGTFMwdExTMEsKICAgIHNlcnZlcjogaHR0cHM6Ly9sb2NhbGhvc3Q6ODQ0MwogIG5hbWU6IGt1YmVybmV0ZXMKY29udGV4dHM6Ci0gY29udGV4dDoKICAgIGNsdXN0ZXI6IGt1YmVybmV0ZXMKICAgIHVzZXI6IGt1YmVybmV0ZXMtYWRtaW4KICBuYW1lOiBrdWJlcm5ldGVzLWFkbWluQGt1YmVybmV0ZXMKY3VycmVudC1jb250ZXh0OiBrdWJlcm5ldGVzLWFkbWluQGt1YmVybmV0ZXMKa2luZDogQ29uZmlnCnByZWZlcmVuY2VzOiB7fQp1c2VyczoKLSBuYW1lOiBrdWJlcm5ldGVzLWFkbWluCiAgdXNlcjoKICAgIGNsaWVudC1jZXJ0aWZpY2F0ZS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VSSlZFTkRRV2R0WjBGM1NVSkJaMGxKVkN0dVRERXJjVTlaWTFWM1JGRlpTa3R2V2tsb2RtTk9RVkZGVEVKUlFYZEdWRVZVVFVKRlIwRXhWVVVLUVhoTlMyRXpWbWxhV0VwMVdsaFNiR042UVdWR2R6QjVUa1JGZDAxVVZYaE9WRWwzVFhwQ1lVWjNNSHBPUkVWM1RWUk5lRTVVU1hkTmVrWmhUVVJSZUFwR2VrRldRbWRPVmtKQmIxUkViazQxWXpOU2JHSlVjSFJaV0U0d1dsaEtlazFTYTNkR2QxbEVWbEZSUkVWNFFuSmtWMHBzWTIwMWJHUkhWbnBNVjBackNtSlhiSFZOU1VsQ1NXcEJUa0puYTNGb2EybEhPWGN3UWtGUlJVWkJRVTlEUVZFNFFVMUpTVUpEWjB0RFFWRkZRVEEzTmxKd2JqTnFhWFY1V25Bd1RVMEtXUzluZUd0TVoyNUVVRk54Wkd4blVsVlBhRWgxTmt0bldEQldSbnA1Y1hKcFNUTXliU3RzUkVndlMxSnBXRTFMZW1oSWVXcENhelZCZFVrM1ptZGxXUXBOWlVGcFJ5czRkSEpvV1ZkWGVGVTNVMEZDS3poalZ5czRaRTlGWmxWbGNtVjBUVTFQYUVSNEx6YzJTa3RyUW5KTGJqWmlOMDVXUjNaa1NUZ3hhVlZwQ2t4U1JrZFdia1JVT1ZORWEyMTJXSEpGWlhoVE1HVnlUbEZMT1U5bGNVNWtMMDB2Y1ZWTlNuUnNRemxqUW14ck1EUnRTaThyUlZrNFJFZHpVSFZFY2s0S1NIVkpla2R4ZW5FNVFrZEliMW95TjJoNlVHMVVTMVpYYlRWdFptSm9ZMHBHZVZoTlUweHplQzlUZDFwVFJEVkJWbHBUV0ZOT1YwcHlhMmR5WWpaWVV3b3lSSEZIUW1KR2MyZHJlbkV5VkdOdFNrVjRSVU5wTUZWYU1raDZTbXRZYzA4MVMzbDZjbU5JTVRSSlMxQkllRVpUZVhvcmRuVlBjVkV3VDJsYWJFUllDak4yUVZScmQwbEVRVkZCUW04eFdYZFdSRUZQUW1kT1ZraFJPRUpCWmpoRlFrRk5RMEpoUVhkRmQxbEVWbEl3YkVKQmQzZERaMWxKUzNkWlFrSlJWVWdLUVhkSmQwUkJXVVJXVWpCVVFWRklMMEpCU1hkQlJFRm1RbWRPVmtoVFRVVkhSRUZYWjBKUmJGSTBTWE5oWXpWUGEycGtZV1JuU1ZGbU5YRk9kRVJYVUFvdlZFRk9RbWRyY1docmFVYzVkekJDUVZGelJrRkJUME5CVVVWQlMxZE5ORmt5UlRWeE9VNXFRbmd4TVd4NE5rdFRTR1pFVDFaWVJEbFZjMGx5YlhvdkNrRjRiREF4YVZkaVUxZFphSG9yYmxCRk5qQTNNRmhxVTBWV1IwSmxTMWxwWjNsS2JsVnNNVmMwU1hFelZtVlBaalo2U1Zod2VEbFRVbVpxVVRVNGNpc0tUWHBtVm1oWWJVNTRVMUJuV25BNWFGVlpaVzFzVEhodk9ESkJhMUF6TTBkcE5tSjNSVnBhY21wbmVIY3lXVVowZFZoUVFTOUxlbGN5TjJJMkwzRkhNZ3BRTm1vMFRXcFFXVVZ1SzNSUk4zUTVWV1l3SzNwNWRVRXhjMmhvUm5KbWVuVjNjR1pFTVZWREwzcGlVbmRHVVRjM2NrWjRZbGgzYW05eldtaGtSMlU1Q2k5b2J6Vk1OVWxvZFdSMlFsQlJhbVZLVmpGaE5FeDViVUo1UzI1SVFqSjJSRGhRV0RKS1dqTnliMEZ0TWtwclJWQnhiMVZzTVdWbFdVd3JibXBMZFhrS2VsTk1PVWhpWm1SclV6bHJjVlYwUXpNdmRWSjVka3QzTTJOT1RFNXhSWGRsTWtwQk5WUlFkREpGUzI5eFJrUkdjSGM5UFFvdExTMHRMVVZPUkNCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2c9PQogICAgY2xpZW50LWtleS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJTVTBFZ1VGSkpWa0ZVUlNCTFJWa3RMUzB0TFFwTlNVbEZjRUZKUWtGQlMwTkJVVVZCTURjMlVuQnVNMnBwZFhsYWNEQk5UVmt2WjNoclRHZHVSRkJUY1dSc1oxSlZUMmhJZFRaTFoxZ3dWa1o2ZVhGeUNtbEpNekp0SzJ4RVNDOUxVbWxZVFV0NmFFaDVha0pyTlVGMVNUZG1aMlZaVFdWQmFVY3JPSFJ5YUZsWFYzaFZOMU5CUWlzNFkxY3JPR1JQUldaVlpYSUtaWFJOVFU5b1JIZ3ZOelpLUzJ0Q2NrdHVObUkzVGxaSGRtUkpPREZwVldsTVVrWkhWbTVFVkRsVFJHdHRkbGh5UldWNFV6QmxjazVSU3psUFpYRk9aQW92VFM5eFZVMUtkR3hET1dOQ2JHc3dORzFLTHl0RldUaEVSM05RZFVSeVRraDFTWHBIY1hweE9VSkhTRzlhTWpkb2VsQnRWRXRXVjIwMWJXWmlhR05LQ2taNVdFMVRUSE40TDFOM1dsTkVOVUZXV2xOWVUwNVhTbkpyWjNKaU5saFRNa1J4UjBKaVJuTm5hM3B4TWxSamJVcEZlRVZEYVRCVldqSklla3ByV0hNS1R6VkxlWHB5WTBneE5FbExVRWg0UmxONWVpdDJkVTl4VVRCUGFWcHNSRmd6ZGtGVWEzZEpSRUZSUVVKQmIwbENRVUZ0UWxodmVEUjZhMUZ2VW5WWFNncFVkamhXTTNwUE9UaEtURmRWVDJseVNESjJaMWxaWmpoNmRra3pOaXRDVnpkQmFHWlNlR3BOY1dWVEwxRlBWa2x6Y3pFMVdrMW5RbEZrZG5SU0wzQnhDbGxPZVhOVlVuVnlObmROVEc1emJFdzVURTVSZDIxSlkwaEhlRU56U1c1dVRUY3hOWGdyZVRkWlowY3pSR1pXUm0xM2JqTlZVWGxoY20xb1RFUmpkMWdLVFhGbFVGSlFZMUZpV2s1MFprcFhkMUZJVFROb2NXaENkME5GVGt0TVRtTkVjbmRZUkU1bVptdFBWV05rYW5OQmFtRjRVVUUyZGxoMlV6TjJPRnBJWlFwclJUTk9aM3AxYW1kVmNFMTBjalpFTXpBdmVHcEtRbGhvV1VGRWJreHlhRkpuWTBSSWFTdHdSVzAzWkRWU1NFVjJUVTV6VjNjMFYzUlZNbEF4UTNZekNqZHBaRzV3VEdkbWIxSjVaMWN3VkhKblpVRnBZbGxVYlRGTlNWZHdaMDAxVkRscWR6RndjbTk1VDNoS04yb3lWVkJIU2xOS2QxVnFhemRFZUd4UFVXTUtORU5sUmxSV1JVTm5XVVZCTlRSTlUxUk1WR05PUWtaYWVFWkRiemMxYUVOaU5ubEtTVmxUYTNNcmN6bFljMVJWWkhwTFN6VkdaMVUzVFVKUk5qWlBZZ3A2WmxCd1JYaFlaWGxMTjJ0aFFtMDFLMlJhU1RBelFXOVBkRTB6VjJRclFYWkVXVzVSVkZGMGMxWlZXak5LTkU1NmQxTmlla05pWkZSUU5EUlpRV3d3Q21wblp6QkJVbEZUVTBGWllWY3hXSEp2VW1OR05HWm5iVkJWUTNKb09HdG1VRlpCU0RSRFRXbENkVmhFYVZOcVNsZHJUa2hCTUZWRFoxbEZRVFpwVVRZS1Eza3hXVU12WWpVMlVrMUhkbVY0WlVWSU1sUjNaMHRRTTFwMGMzVTRTRXREYkhsaVkwSk1kV3hIYURkNGNrcHpiamg1YkhwS2R6WmxVbFJRT0RoTk5BcHRObll6Wm5CMU5sTmtWMFoxUzJFeVZtRlNNbk5VZDBJek1Hc3JNMVJIYldOMWNXTkhkRTQ0Wld4aFNIZHdRMFUyZG5KVlJXdHdOblZDZWtSVloyNVlDbWRNYW5kYWNrNHlORUpPWlhWMVJXbDFLMGRGUW1sSE1WbHFkREJIZEZaUFExZzJheTlRWTBObldVSktSM2gxZUU5Nk9UVTVZWGx4UlVkd05GWnNWazBLV0Vkb2JGQkVVMmxXTDNSdFRFeFBSeTlHTDJSdlZ6ZHBTVUY1ZEdwR2JVd3pTMVJUUlZGeWNuaHJXamhHVUhWclUwTldlRVo1Tm5NdlRIVjVNWEZCV2dwVWRWSjNZelpEYlV0bFFtcDVWV3hHUVZGcFRtNUJkM1l6ZFhSQlkwWjJjelpsZHpaR2RrNDJOazl3ZW5kNGMwbEpTRkZOUjJ0a1FTdHRXSGRHTDA5VkNtTmxkeloxTHpGdWRtcENWRFZLZUVsek5IWXJTRkZMUW1kUlJHOUlZVmhXVFhONVdXWXJNV051Um05UmNXMVVUMGxGTHpsQk5rVjJOV3RqZWtVemFVTUtUVlJDTDA5TFYwRlNkSGdyVEVvdk0wRXlRbE54V1ZwdFVuSlhjRk5LTW5vMWFGWnFWa2R4Ym1rMlFWaFpSalZsU2t3M2RVVlVhMmczU0VRdmR6VnljQXBVWVVGWlJVSlVWa042Wkc1TlVFa3dTMUpzV1ZWTVJ6aEdMMlZwVDNSUFJHSjBVVlZZUmtjMk4ycHFOVTUyZGtOSGJERlNNM1JVT0Znd2FFa3lZa2RXQ2xKQ1J6UklVVXRDWjFGRFYxUTNWRzV0V2twNGFqTnNablZITTJWU1kxTnJSV3BVY0VwMmRWcFRVeXRTTDNsNmVqUkNVVWh6U2t4SWVuTlpPVmgxUm1rS2VWVjNlVW81T1dGRU5HaFBkVVJzV0ZOVmRWUjROemtyTjJ4V2RWRjRhRGhwYkZaMWFXUTVkbU0yZG1WU1ZqWnpPWEZpVEhKNGMyOUJaRFJRYUV4ak1BcG1aekpSTlhWNU1XRTJkMDh3TDJOTFdFcG9ObWxvU3pSclVqRnVMMVUxV1ZSS1ZpOXlNRGcxUWtaUU9WTkJhMVJOUjBGb1NVRTlQUW90TFMwdExVVk9SQ0JTVTBFZ1VGSkpWa0ZVUlNCTFJWa3RMUzB0TFFvPQo=
  token: ""
kind: Secret
metadata:
  creationTimestamp: "2024-10-15T15:20:40Z"
  name: vc-my-vcluster
  namespace: team-x
  ownerReferences:
  - apiVersion: v1
    controller: false
    kind: Service
    name: my-vcluster
    uid: 78969912-8525-4bad-b5f6-78398a45f536
  resourceVersion: "103464"
  uid: 4f89bddb-1914-450e-9ae5-3499910c1564
type: Opaque

emoreth commented 1 month ago

Host cluster is EKS 1.29

Values:

controlPlane:
  backingStore:
    database:
      embedded:
        enabled: true
  coredns:
    deployment:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: vcluster-install
                operator: In
                values:
                - 'yes'
    embedded: false
    enabled: true
  distro:
    k8s:
      enabled: true
      version: v1.29.6
  ingress:
    annotations:
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
      nginx.ingress.kubernetes.io/ssl-passthrough: 'true'
      nginx.ingress.kubernetes.io/ssl-redirect: 'true'
    enabled: true
    host: my.website.com
    pathType: Prefix
    spec:
      ingressClassName: nginx
  proxy:
    extraSANs:
    - my.website.com
  statefulSet:
    persistence:
      volumeClaim:
        retentionPolicy: Delete
        size: 5Gi
        storageClass: ceph-filesystem
    scheduling:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: sysbox-install
                operator: In
                values:
                - 'no'
exportKubeConfig:
  context: testv2
  insecure: false
  server: https://my.website.com
networking:
  advanced:
    fallbackHostCluster: false
rbac:
  clusterRole:
    extraRules:
    - apiGroups:
      - storage.k8s.io
      resources:
      - storageclasses
      verbs:
      - get
      - watch
      - list
sync:
  fromHost:
    storageClasses:
      enabled: false
  toHost:
    ingresses:
      enabled: true
    persistentVolumeClaims:
      enabled: true
    persistentVolumes:
      enabled: true

Helm command

helm upgrade testv2 /app/new-charts/testv2 -n dz--testorg0000000000000--testv200000000000000--1e4ca9d9023f983 --atomic --namespace vc--testorg0000000000000--testv200000000000000--1e4ca9d9023f983 --install --timeout 15m

Secret vc-testv2

apiVersion: v1
data:                                                                                                                                                                                                                                                 
  client-certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1..........0FURS0tLS0tCg==
  client-key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQp.....gUFJJVkFURSBLRVktLS0tLQo=
  config: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoK.......UzB0TFFvPQo=
  token: ""
kind: Secret
metadata:
  creationTimestamp: "2024-10-14T20:29:41Z"
  name: vc-testv2
  namespace: vc--testorg0000000000000--testv200000000000000--1e4ca9d9023f983
  ownerReferences:
  - apiVersion: v1
    controller: false
    kind: Service
    name: testv2
    uid: 021e7ef9-fbb4-4d5c-bbb1-da3fab28abbb
  resourceVersion: "111236861"
  uid: 575912ae-8b1c-4e16-8808-d5074b3ce2b6
type: Opaque

Decoded config

apiVersion: v1
clusters:
- cluster:
    server: https://my.website.com
  name: testv2
contexts:
- context:
    cluster: testv2
    user: testv2
  name: testv2
current-context: testv2
kind: Config
preferences: {}
users:
- name: testv2
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZ0F3SUJBZ0lJQTRnT.....yT1htSWxKcUIzaUJJZ3MvZUE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBdVl......VkFURSBLRVktLS0tLQo=

I can't give full output as I have to redact some values.

I now have a code workaround that adds that info to the secret, but if I run helm upgrade again without my code the certificate-authority-data goes away again.

ThomasK33 commented 1 month ago

Thanks for the details and the steps to reproduce.

The following snippet was the part that helped me reproduce it:

exportKubeConfig:
  context: testv2
  insecure: false
  server: https://my.website.com/

I've created a PR (#2231) that should fix this by adding the CA data when using exportKubeConfig.server.

loft-sh / vcluster