Support KubeVela multi-cluster

[Somefive]

What would you like to be added?

KubeVela also has multi-cluster delivery capability. I saw that clusterpedia is going to support Karmada&ClusterNet. Will it be possible to support KubeVela or OCM (Open Cluster Management) as well? Or any future plan?

If there is some documentations for instructing how to integrate with clusterpedia, I think KubeVela community might also be willing to do it.

Why is this needed?

Searching or controlling multi-cluster resources in KubeVela is also critical. For KubeVela multi-cluster applications, it also deliver resources in multi-cluster. However, there is still no convenient tools for KubeVela multi-cluster users to inspect multi-cluster resources from a unified view. Clusterpedia can be very useful for that I think. :)

[Iceber]

@Somefive Oh, and of course, clusterpedia could discover clusters that are managed by kubevela/ocm.

https://github.com/clusterpedia-io/clusterpedia/issues/185 is actually a generic solution, not specific to one or some multi-cloud platform. Just define a ClusterImportPolicy for OAM and you will be able to discover these clusters

[Iceber]

@Somefive Hi, https://github.com/clusterpedia-io/clusterpedia/pull/259 has completed the basic features, It will be released in 0.4

To get a preview of this new feature, you can deploy rbac, crds and controller manager deployment in an existing clusterpedia environment, and create a ClusterImportPolicy for kubevela/ocm, eg. https://github.com/clusterpedia-io/clusterpedia/pull/263

Welcome to contribute this policy to clusterpedia so that more people can use it!

[Somefive]

Great! I will try it in recent days. :)

[Iceber]

@Somefive 0.4 has been released, you can use helm to deploy directly, and the documentation of the interface to the multi-cloud platform has been perfected, you can refer to Interfacing to Multi-Cloud Platforms :grinning:

[Somefive]

@Iceber I tried the ClusterImportPolicy and found some potential problems that might make it hard for KubeVela cluster to be imported. I'm not sure if I'm understanding the problem correctly.

The Cluster in KubeVela, backended by ClusterGateway natively uses secrets directly to store the credential but do not have an actual CustomResource to represent the cluster. Instead, we also use AggregatedServer layer to make a ClusterGateway proxy to the underlying secrets.

For example, if users "create a cluster in KubeVela", we will store a secret with special labels in the vela-system namespace (by default). However, if I'm understanding it correctly, the ClusterImportPolicy in clusterpedia "watches" the creation and deletion event. But KubeVela clustergateway does not implement the watch interface, so it is impossible to watch clustergateway. Alternatively, we could directly watch secrets in the vela-system with special labels to filter out those valid cluster secret. But it seems the .spec.source in ClusterImportPolicy is unable to do so.

Is there any recommendation/advices for this situation?

P.S. KubeVela has two ways to manage clusters. One way is the "native secret" and the other one is the "OCM ManagedCluster". I think the ClusterImportPolicy is okay with the OCM case but most KubeVela users use the former one.

[Somefive]

Sorry for the late response :)

P.S. I was trying to go to the community meeting directly but seems Tencent meeting is not available for me.

[Iceber]

@Somefive Thank you very much for the feedback!

I think using secret is a simpler way for clusterpedia. I would add a sourceSelectorTemplate field to the ClusterImportPolicy to filter source resources (secrets), what do you think?

I was trying to go to the community meeting directly but seems Tencent meeting is not available for me.

We'll use zoom in our next meeting

[Somefive]

@Iceber Thanks you for your response.

I would add a sourceSelectorTemplate field to the ClusterImportPolicy to filter source resources (secrets), what do you think?

Great! Looking forward to this feature.

We'll use zoom in our next meeting

That works for me. Thanks :)

[Iceber]

@Somefive source.selectorTemplate feature has been merged.

You can update the controller-manager image tag to the latest, and then update the crd.

kubectl apply -f https://raw.githubusercontent.com/clusterpedia-io/clusterpedia/main/charts/clusterpedia/_crds/policy.clusterpedia.io_clusterimportpolicies.yaml

An example of the use of source.selectorTemplate is

apiVersion: policy.clusterpedia.io/v1alpha1
kind: ClusterImportPolicy
metadata:
  name: secret
spec:
  source:
    group: ""
    resource: secrets
    selectorTemplate: |
      {{ if eq .source.metadata.namespace "default" }}
        {{ hasKey .source.metadata.labels "clusterpedia" }}
      {{ end }}
  nameTemplate: 'secret-{{ .source.metadata.namespace }}-{{ .source.metadata.name }}'
  template: |
    spec:
      apiserver: "https://127.0.0.1"
      syncResources:
        - group: ""
          resources:
            - "pods"
            - "configmaps"
      syncResourcesRefName: ""
  creationCondition: "true"

[Somefive]

It is now okay to sync KubeVela clusters through the ClusterImportPolicy now, thanks for it.

apiVersion: policy.clusterpedia.io/v1alpha1
kind: ClusterImportPolicy
metadata:
  name: kubevela
spec:
  source:
    group: ""
    resource: secrets
    selectorTemplate: |
      {{ if eq .source.metadata.namespace "vela-system" }}
        {{ if .source.metadata.labels }}
          {{ eq (index .source.metadata.labels "cluster.core.oam.dev/cluster-credential-type") "X509Certificate" }}
        {{ end }}
      {{ end }}
  nameTemplate: '{{ .source.metadata.name }}'
  template: |
    spec:
      apiserver: {{ index .source.data "endpoint" | b64dec }}
      caData: {{ index .source.data "ca.crt" }}
      certData: {{ index .source.data "tls.crt" }}
      keyData: {{ index .source.data "tls.key" }}
      syncResources:
        - group: ""
          resources:
            - "pods"
            - "configmaps"
        - group: apps
          resources:
            - deployments
      syncResourcesRefName: ""
  creationCondition: "true"

BTW, there are multiple ways for KubeVela to use multi-clusters. The above ClusterImportPolicy is just one of it. I think for X509 and ServiceAccount cases the ClusterImportPolicy will works fine to generate PediaCluster. The OCM managed cluster on the other hand, cannot be directly accessed through Certificate or Token (the managed cluster apiserver is not accessible from the hub cluster), which means PediaCluster could not handle that by now. But that is an extra case.

There is still something trouble with using the use of ClusterPedia searching, but that should not belong to this issue. I will go into it and give some responses later. Thanks for your help.

[Iceber]

It is now okay to sync KubeVela clusters through the ClusterImportPolicy now, thanks for it.

Thank you very much for creating this policy yaml, I think it is a very important contribution to clusterpedia, would you like to create a pr to submit it? It looks like there could be more types of resources to synchronize, for example:

      syncResources:
        - group: ""
          resources:
            - "pods"
            - "services"
            - "configmaps"
            - "secrets"
            - "namespaces"
            - "nodes"
        - group: "apps"
          resources:
            - "*"

BTW, there are multiple ways for KubeVela to use multi-clusters. The above ClusterImportPolicy is just one of it. I think for X509 and ServiceAccount cases the ClusterImportPolicy will works fine to generate PediaCluster. The OCM managed cluster on the other hand, cannot be directly accessed through Certificate or Token (the managed cluster apiserver is not accessible from the hub cluster), which means PediaCluster could not handle that by now. But that is an extra case.

For this case, it looks like we need to wait until clusterpedia adds agent collection mode.

There is still something trouble with using the use of ClusterPedia searching, but that should not belong to this issue. I will go into it and give some responses later. Thanks for your help.

More questions are very welcome and hopefully the clusterpedia will work better for the kubevela ❤️