k0s support beside k3s?

[ThomasLohmann]

Hi everyone,

please can you also support k0s beside k3s? k0s has much more use cases (bare-metal, cloud, iot, edge etc.) compared to k3s (iot, edge). In addition, k0s is much less opinionated regarding networking, storage, ingress etc. + the size is also small (187 MB). Finally, k0s is used for conventional staging & production clusters (bare-metal or cloud) which means that dev vClusters with k0s will be much closer to staging & production. So it would be great, if you can support it. Please, see the following link: https://k0sproject.io/

Best regards, Thomas

[FabianKramm]

@ThomasLohmann thanks for creating this issue! Yeah we will think about supporting other distributions, including k0s

[danmx]

I'm interested in this too. Any suggestions how the community could start contributing?

[FabianKramm]

@danmx theoretically its already possible to use vcluster with other kubernetes distributions, as we do not have any hardcoded k3s dependencies anymore, however the vcluster chart is currently still hardwired to k3s, so you would need to do the wiring between vcluster syncer and k0s yourself.

We are currently working on making vcluster usable with vanilla k8s to allow easier HA, which would be then very similar to using k3s with k0s.

[FabianKramm]

@ThomasLohmann @danmx we actually added a guide how to use k0s to our docs and released a new version (v0.5.0-alpha.0) that supports that

[olljanat]

@FabianKramm thanks for adding support for k0s and k8s.

I did quick test on v0.5.0-alpha.3 by deploying k8s with default settings looks which to be working fine but if I enable switches --fake-persistent-volumes=false and --enable-storage-classes it fails to error:

I1126 13:49:32.854841       1 start.go:128] couldn't find virtual cluster kube-config, will retry in 1 seconds

[FabianKramm]

@olljanat good catch, the problem is you overwrite the default extraArgs there, it should look like this in your values.yaml:

syncer:
  extraArgs:
    - --request-header-ca-cert=/pki/ca.crt
    - --client-ca-cert=/pki/ca.crt
    - --server-ca-cert=/pki/ca.crt
    - --server-ca-key=/pki/ca.key
    - --kube-config=/pki/admin.conf
    - --fake-persistent-volumes=false
    - --enable-storage-classes

We'll fix this for the next version though so you can use the extraArgs as expected before

[olljanat]

@FabianKramm ok that helps to start issue but there looks to be problem with missing permissions too. Same applies to nodes sync. So I will wait next version and test again then...

[FabianKramm]

@olljanat did you enable the cluster role via:

rbac:
  clusterRole:
    create: true

[olljanat]

@FabianKramm good point that helped. Thanks.

However one more thing which I noticed. Especially k8s with HA creates a lot of containers which why separating vcluster and resources deployed inside of to separate namespaces make sense. Syncer have that --target-namespace flag but that namespace and rolebinding to it does not get created automatically. If I create those then it works but then issue it that CoreDNS will gets deployed to that namespace instead of same one with rest of the vcluster components.

[FabianKramm]

@olljanat yes, CoreDNS is deployed inside the vcluster instead of outside like etcd, api server etc., which is why it is created inside the --target-namespace.

[olljanat]

I think that this one can be closed as both k0s and k8s works nicely on v0.5.0-beta.0

[FabianKramm]

@olljanat thanks for the notification, I guess this can be indeed closed.