loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.26k stars 398 forks source link

vcluster list does not list virtual clusters not created by vcluster create #364

Closed danielhelfand closed 2 years ago

danielhelfand commented 2 years ago

From the docs, deploying virtual clusters can be done as follows using kubectl:

kubectl create namespace host-namespace-1
helm template my-vcluster vcluster --repo https://charts.loft.sh --set serviceCIDR=10.96.0.0/12 -n host-namespace-1 | kubectl apply -f -

When doing a vcluster list after running the commands above, no virtual clusters are shown:

 NAME   NAMESPACE   CREATED   AGE
 No entries found
FabianKramm commented 2 years ago

@danielhelfand thanks for creating this issue! vcluster list searches for the helm secret that was created when installing vcluster through the CLI or helm, but you are correct vcluster's are not shown if they were deployed with kubectl. I guess we could go the same route as vcluster connect does and search for pods that have the vcluster labels set, however using vcluster delete with a non helm deployed vcluster is pretty hard to do as we do not have information about what actually belongs to the vcluster if the helm state is not there.

danielhelfand commented 2 years ago

For reference, my use case is using Carvel to deploy vclusters in a more declarative fashion with kapp-controller. So I would just use helm for templating but deploy with a tool called kapp through the API here.

Since there is such a strong tie to helm though, maybe exploring ArgoCD or FluxCD would fit better with this model of more declarative approach to provisioning vclusters.

For reference, this is what you mean by helm state?

FabianKramm commented 2 years ago

@danielhelfand thanks for clarifying your use case. In general I guess we can switch to a pod based implementation for vcluster list which should work in your case. With helm state I mean the secret that helm creates within the cluster upon deploying a helm chart like vcluster, which then later is read by helm (like helm status or other commands) to identify which releases were deployed.