Expose a service in karmada controlplane

[oussexist]

Please provide an in-depth description of the question you have: i wonder if i want to expose the deployment from karmada control plane since the deployment is propagated to the 2 cluster members so as a service user i won't face any problem if one of the clusters goes down since the control plane still points on the other cluster's service.. to simplify things , putting multi-cluster federation apart , if am using just one cluster and create a service for a deployment either by cluster-ip or nodeport or loadbalancer type , i simply put type as loadbalancer for example and can access to the service through the loadbalancer's ip. I almost want to do this scenario but just having the advantage of multi-cluster feature so i really don't know exactly maybe something like : a service on the control plane that points on services deployed on member clusters ( since same deployment just propagated) and user access to the karmada's service.

What do you think about this question?: Well , I am not sure if the feature is already exists but as i searched in documentation i found this : https://karmada.io/docs/userguide/service/multi-cluster-service/ and this https://karmada.io/docs/next/tutorials/access-service-across-clusters/ , but am not sure if that's what am looking for .. because those actually assure that a member cluster can access to another member's service but am talking of exposing the service to user from control plane for example nginx deployment when i access to it's service i get Welcome to nginx.

Regards.

[XiShanYongYe-Chang]

Hi @oussexist, the two you mentioned are both implementations of multi-cluster services. They both enable multi-cluster service discovery, allowing you to access services from one cluster to another.

If you want to export the service from the Karmada control plane, I think you can use the method mentioned in the https://karmada.io/docs/next/tutorials/access-service-across-clusters/ , and include the LoadBalancer type in the MultiClusterService object. Like this:

apiVersion: networking.karmada.io/v1alpha1
kind: MultiClusterService
metadata:
   name: nginx
spec:
  types:
    - LoadBalancer
  providerClusters:
    - name: member1
    - name: member2

In addition, you can implement multicluster-cloud-provider to expose the service to the LB service.

[oussexist]

Hello @XiShanYongYe-Chang , Thanks for you reply. So i did the steps of enabling multicluster service and deployed wiht propagation policy deployment in each of my clusters and then tried to apply the service & multiclusterservice file as follows :

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: my-nginx
---
apiVersion: networking.karmada.io/v1alpha1
kind: MultiClusterService
metadata:
   name: my-nginx-cross-svc
spec:
  types:
    - LoadBalancer
  providerClusters:
    - name: localcluster
    - name: kubernetes

but as i can see , there is no services deployed on my member clusters , all i can see is a service and a multiclusterservice deployed on karmada controlplane :

ubuntu@master:~$ kubectl get svc --kubeconfig /etc/karmada/karmada-apiserver.config
NAME           TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
kubernetes     ClusterIP   10.96.0.1     <none>        443/TCP   42h
my-nginx-svc   ClusterIP   10.105.2.32   <none>        80/TCP    56m

ubuntu@master:~$ kubectl --kubeconfig /etc/karmada/karmada-apiserver.config get multiclusterservice
NAME                 AGE
my-nginx-cross-svc   57m

this is a describe for the multi cluster service :

ubuntu@master:~$ kubectl --kubeconfig /etc/karmada/karmada-apiserver.config describe multiclusterservice my-nginx-cross-svc
Name:         my-nginx-cross-svc
Namespace:    default
Labels:       multiclusterservice.karmada.io/permanent-id=cbc3c51e-77ca-4bff-9c95-0ff5eef7da51
Annotations:  <none>
API Version:  networking.karmada.io/v1alpha1
Kind:         MultiClusterService
Metadata:
  Creation Timestamp:  2024-07-24T09:48:50Z
  Generation:          1
  Resource Version:    115167
  UID:                 39a5680e-c7f2-4bfd-bbe3-de51dd109242
Spec:
  Provider Clusters:
    Name:  localcluster
    Name:  kubernetes
  Range:
  Types:
    LoadBalancer
Events:  <none>

Note : I am not sure if the 2 cluster are well connected through Submariner since i dont master this tool , so i did deployed broker on a cluster member and joined the other member to it. Although , i think Submariner's role is not here , so the problem that am facing is not due to submariner connectivity..

[XiShanYongYe-Chang]

Sorry, I forgot to mention that the name of the MultiClusterService object needs to be the same as the name of the Service.

If you want to access the Service across clusters at the same time, you can configure it like this:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: my-nginx
---
apiVersion: networking.karmada.io/v1alpha1
kind: MultiClusterService
metadata:
   name: my-nginx-svc
spec:
  types:
    - LoadBalancer
    - CrossCluster
  providerClusters:
    - name: localcluster
    - name: kubernetes

This should allow you to see the EndpointSlices from the other cluster on the Karmada control plane as well as on the other cluster.

[oussexist]

Sorry, I forgot to mention that the name of the MultiClusterService object needs to be the same as the name of the Service.

If you want to access the Service across clusters at the same time, you can configure it like this:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: my-nginx
---
apiVersion: networking.karmada.io/v1alpha1
kind: MultiClusterService
metadata:
   name: my-nginx-svc
spec:
  types:
    - LoadBalancer
    - CrossCluster
  providerClusters:
    - name: localcluster
    - name: kubernetes

This should allow you to see the EndpointSlices from the other cluster on the Karmada control plane as well as on the other cluster.

using both types at the same time gives an error :

service/my-nginx-svc created
Error from server (Forbidden): error when creating "cross-service.yaml": admission webhook "multiclusterservice.karmada.io" denied the request: spec.types: Invalid value: []v1alpha1.ExposureType{"LoadBalancer", "CrossCluster"}: MultiClusterService types should not contain more than one type

[XiShanYongYe-Chang]

Oh, Sorry. Now it looks like there is a limitation to this feature. User can't use both types at the same time.

[oussexist]

So the idea of exposing a service on controle plane that communicates with member services can't be done ?

[XiShanYongYe-Chang]

So the idea of exposing a service on controle plane that communicates with member services can't be done ?

Yes, you can use the LoadBalancer type.

[oussexist]

Ok , can you please tell me what exactly the type loadbalancer on cross-cluster service kind on this case do ? Because i searched a lot on the documentation and i didnt really find something or a test case to explain it's job on karmada control plane. Otherwise , we can close this issue and thanks.

[XiShanYongYe-Chang]

Hi @oussexis, Its function is to export the backends of the Service from different clusters out of the Karmada control plane, and then expose the service to the public cloud's LoadBalancer service or your own LoadBalancer server through implementing a multicluster-cloud-provider (this job is similar to the ingress-controller in Kubernetes). This feature is not out-of-the-box; you will need to undertake some development work.

[oussexist]

Hi @oussexis, Its function is to export the backends of the Service from different clusters out of the Karmada control plane, and then expose the service to the public cloud's LoadBalancer service or your own LoadBalancer server through implementing a multicluster-cloud-provider (this job is similar to the ingress-controller in Kubernetes). This feature is not out-of-the-box; you will need to undertake some development work.

Hello again , Sorry for inactivity cuz i was kind of busy with other stuff. Thanks @XiShanYongYe-Chang for this explain, anyway yes i kind of undertsand all of that , but I am still wondering why MulticlusterService dosen't even propagate services same as deployments. as i said before i simply after deploying deployments to member clusters i want to deploy services also, but it's not happenning yet, there is not services propagated to member clusters Here are some logs :

ubuntu@master:~/nginx$ kubectl --kubeconfig /etc/karmada/karmada-apiserver.config get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        4d1h
nginx-svc    NodePort    10.108.113.174   <none>        80:30000/TCP   4m5s
ubuntu@master:~/nginx$ kubectl --kubeconfig /etc/karmada/karmada-apiserver.config get multiclusterservice
NAME        AGE
nginx-svc   5m7s
ubuntu@master:~/nginx$ kubectl --kubeconfig /etc/karmada/karmada-apiserver.config describe multiclusterservice nginx-svc
Name:         nginx-svc
Namespace:    default
Labels:       multiclusterservice.karmada.io/permanent-id=61b5f246-8ba2-4922-b122-24d324043185
Annotations:  <none>
API Version:  networking.karmada.io/v1alpha1
Kind:         MultiClusterService
Metadata:
  Creation Timestamp:  2024-08-12T10:51:30Z
  Generation:          1
  Resource Version:    149485
  UID:                 ba14ff86-11ae-4e5e-9fd9-37e73f5cc00c
Spec:
  Provider Clusters:
    Name:  preprod
    Name:  prod
  Range:
  Types:
    LoadBalancer
Events:  <none>

and here are the manifests used :

ubuntu@master:~/nginx$ cat svc-propagation.yaml
apiVersion: networking.karmada.io/v1alpha1
kind: MultiClusterService
metadata:
   name: nginx-svc
spec:
  types:
    - LoadBalancer
  providerClusters:
    - name: preprod
    - name: prod
ubuntu@master:~/nginx$ cat nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  selector:
    app: nginx
  type: NodePort
  ports:
  - port: 80
    nodePort: 30000
    targetPort: 80

P.S: even tho i enabled the multiclusterservice , and i tried both types LoadBalancer and CrossCluster types , same thing @chaosi-zju Do you have any idea about that ?

Edited : Well , popagatin services shouldn't be like that i followed this : https://github.com/karmada-io/karmada/pull/3811#issuecomment-1670766563 and its all good. Regards.