fgiorgetti
commented
2 weeks ago Hello Alba,
You're exposing a service, but if your intention is to have access to each pod by name directly, I would recommend you adding the "--headless" flag to the "skupper expose" command, and, instead of exposing the "service", you have to expose the "statefulset" workload.
Thank you,
On Wed, Nov 6, 2024 at 9:56 AM Alba Cañete Garrucho < @.***> wrote:
Describe the bug I do not understand how the DNS resolution works between two clusters that execute statefulsets. When using a single cluster, I can access a Pod through its name (deploying a headless svc), but cannot do the same when using Skupper. Am I missing something?
How To Reproduce
Two clusters were created using kubeadm v1.29.5, containerd as CRI and Flannel as CNI.
Edge cluster
@.***:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION agx14 Ready
19d v1.29.5 agx15 Ready 19d v1.29.5 rpi42 Ready control-plane 19d v1.29.5 HPC cluster
@.***:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION nano1 Ready control-plane 19d v1.29.5 workstation Ready
19d v1.29.5 Create a namespace with the same name in each cluster.
Edge cluster
kubectl create ns compss
HPC cluster
kubectl create ns compss
Deploy Skupper in each namespace. Since I am deploying it on-prem and with private IP addresses, I am using NodePort.
Edge cluster. 192.168.50.15 is the IP address of the agx14 node.
skupper init -n compss --ingress nodeport --ingress-host 192.168.50.15
HPC cluster. 192.168.50.61 is the IP address of the workstation node.
skupper init -n compss --ingress nodeport --ingress-host 192.168.50.61
Link namespaces
Edge cluster
skupper -n compss token create edge.token
HPC cluster. The edge.token file was copied to a machine on the HPC cluster.
skupper -n compss link create edge.token
Output
@.***:~$ skupper -n compss link status
Links created from this site:
There are no links configured or connected
Current links from other sites that are connected:
Incoming link from site 472fdc04-1406-4281-bbf9-81f5e5ad3737 on namespace compss
@.***:~$ skupper -n compss link status
Links created from this site:
Link link1 is connected
Current links from other sites that are connected:
There are no connected links
Deploy test applications in both clusters
The YAML file for the StatefulSet that runs in the edge cluster is
apiVersion: v1 kind: Service metadata: name: compss-matmul-4fc9d6 namespace: compss spec: clusterIP: None # This makes it a headless service selector: app: compss wf_id: compss-matmul-4fc9d6 ports:
name: port-22 protocol: TCP port: 22 targetPort: ssh-port
apiVersion: apps/v1 kind: StatefulSet metadata: name: compss-matmul-4fc9d6-worker namespace: compss spec: selector: matchLabels: app: compss wf_id: compss-matmul-4fc9d6 pod-hostname: worker serviceName: compss-matmul-4fc9d6 replicas: 2 ordinals: start: 2 template: metadata: labels: app: compss wf_id: compss-matmul-4fc9d6 pod-hostname: worker spec: subdomain: compss-matmul-4fc9d6 dnsConfig: searches:
- compss-matmul-4fc9d6.compss.svc.cluster.local containers:
- name: worker image: albabsc/compss-matmul:verge-0.1.8 command: [ "/usr/sbin/sshd", "-D" ] resources: limits: memory: 2G cpu: 4 ports:
- containerPort: 22 name: ssh-port
The YAML file for the StatefulSet that runs in the HPC cluster is
apiVersion: v1 kind: Service metadata: name: compss-matmul-4fc9d6 namespace: compss spec: clusterIP: None # This makes it a headless service selector: app: compss wf_id: compss-matmul-4fc9d6 ports:
name: port-22 protocol: TCP port: 22 targetPort: ssh-port
apiVersion: apps/v1 kind: StatefulSet metadata: name: compss-matmul-4fc9d6-worker namespace: compss spec: selector: matchLabels: app: compss wf_id: compss-matmul-4fc9d6 pod-hostname: worker serviceName: compss-matmul-4fc9d6 replicas: 2 template: metadata: labels: app: compss wf_id: compss-matmul-4fc9d6 pod-hostname: worker spec: subdomain: compss-matmul-4fc9d6 dnsConfig: searches:
- compss-matmul-4fc9d6.compss.svc.cluster.local containers:
- name: worker image: albabsc/compss-matmul:verge-0.1.8 command: [ "/usr/sbin/sshd", "-D" ] resources: limits: memory: 2G cpu: 4 ports:
- containerPort: 22 name: ssh-port
Ensure connection among pods of the same cluster
Edge cluster
@.***:~$ kubectl -n compss get pods NAME READY STATUS RESTARTS AGE compss-matmul-4fc9d6-worker-0 1/1 Running 0 4m19s compss-matmul-4fc9d6-worker-1 1/1 Running 0 4m18s skupper-router-6895bb6f95-88hnj 2/2 Running 0 55m skupper-service-controller-559ddbdd56-wnsvh 1/1 Running 0 55m
@.:~$ kubectl -n compss exec -it compss-matmul-4fc9d6-worker-0 -- bash @.:/# ssh compss-matmul-4fc9d6-worker-1 Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 6.8.0-47-generic x86_64)
- Documentation: https://help.ubuntu.com
- Management: https://landscape.canonical.com
- Support: https://ubuntu.com/pro
This system has been minimized by removing packages and content that are not required on a system that users do not log into.
To restore this content, you can run the 'unminimize' command. Last login: Wed Nov 6 12:33:29 2024 from 10.244.1.160 @.***:~#
HPC cluster
@.***:~$ kubectl -n compss get pods NAME READY STATUS RESTARTS AGE compss-matmul-4fc9d6-worker-2 1/1 Running 0 4m48s compss-matmul-4fc9d6-worker-3 1/1 Running 0 4m47s skupper-router-748c487879-gvpxg 2/2 Running 0 56m skupper-service-controller-6f69b974bd-grgzc 1/1 Running 0 56m
@.:~$ kubectl -n compss exec -ti compss-matmul-4fc9d6-worker-2 -- bash @.:/# ssh compss-matmul-4fc9d6-worker-3 Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.10.192-tegra aarch64)
- Documentation: https://help.ubuntu.com
- Management: https://landscape.canonical.com
- Support: https://ubuntu.com/pro
This system has been minimized by removing packages and content that are not required on a system that users do not log into.
To restore this content, you can run the 'unminimize' command. Last login: Wed Nov 6 12:49:41 2024 from 10.244.2.97 @.***:~#
Expose service with Skupper
Command executed in the edge cluster
skupper -n compss expose service compss-matmul-4fc9d6 --port 22 --address compss-matmul-4fc9d6
Check the service is correctly created
@.***:~$ skupper -n compss service status Services exposed through Skupper: ╰─ compss-matmul-4fc9d6:22 (tcp)
@.***:~$ skupper -n compss service status Services exposed through Skupper: ╰─ compss-matmul-4fc9d6:22 (tcp)
DNS resolution no longer works
When trying to ssh between two pods of the same cluster (e.g edge)
@.***:~$ kubectl -n compss get pods NAME READY STATUS RESTARTS AGE compss-matmul-4fc9d6-worker-0 1/1 Running 0 4m19s compss-matmul-4fc9d6-worker-1 1/1 Running 0 4m18s skupper-router-6895bb6f95-88hnj 2/2 Running 0 55m skupper-service-controller-559ddbdd56-wnsvh 1/1 Running 0 55m
@.:~$ kubectl -n compss exec -it compss-matmul-4fc9d6-worker-0 -- bash @.:/# ssh compss-matmul-4fc9d6-worker-1 ssh: Could not resolve hostname compss-matmul-4fc9d6-worker-1: No address associated with hostname @.:/# ssh compss-matmul-4fc9d6-worker-2 ssh: Could not resolve hostname compss-matmul-4fc9d6-worker-2: No address associated with hostname @.:/#
Expected behavior I would like for every Pod of a StatefulSet to be accessed through their Pod names, or to know the name I have to use. And to know if the name is different when a Pod in cluster 1 want to access a Pod in cluster 2.
Environment details
- Skupper CLI: 1.8.1
- Skupper Operator (if applicable): none
- Platform: kubernetes
Additional context Pods have the the following /etc/resolv.conf file
search compss.svc.cluster.local svc.cluster.local cluster.local lan compss-matmul-4fc9d6.compss.svc.cluster.local nameserver 10.96.0.10 options ndots:5
— Reply to this email directly, view it on GitHub https://github.com/skupperproject/skupper/issues/1772, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYML4SDQ7QQS2RJUTFB7CTZ7IGXFAVCNFSM6AAAAABRIYZVSWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGYZTQMBRHEZTKOI . You are receiving this because you are subscribed to this thread.Message ID: @.***>
albacanete
Describe the bug I do not understand how the DNS resolution works between two clusters that execute statefulsets. When using a single cluster, I can access a Pod through its name (deploying a headless svc), but cannot do the same when using Skupper. Am I missing something?
How To Reproduce
Two clusters were created using kubeadm v1.29.5, containerd as CRI and Flannel as CNI.
*Edge cluster* ``` acanete@rpi42:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION agx14 ReadyCreate a namespace with the same name in each cluster.
*Edge cluster* ``` kubectl create ns compss ``` *HPC cluster* ``` kubectl create ns compss ```Deploy Skupper in each namespace. Since I am deploying it on-prem and with private IP addresses, I am using NodePort.
*Edge cluster*. 192.168.50.15 is the IP address of the agx14 node. ``` skupper init -n compss --ingress nodeport --ingress-host 192.168.50.15 ``` *HPC cluster*. 192.168.50.61 is the IP address of the workstation node. ``` skupper init -n compss --ingress nodeport --ingress-host 192.168.50.61 ```Link namespaces
*Edge cluster* ``` skupper -n compss token create edge.token ``` *HPC cluster*. The edge.token file was copied to a machine on the HPC cluster. ``` skupper -n compss link create edge.token ``` Output ``` acanete@rpi42:~$ skupper -n compss link status Links created from this site: There are no links configured or connected Current links from other sites that are connected: Incoming link from site 472fdc04-1406-4281-bbf9-81f5e5ad3737 on namespace compss ``` ``` acanete@nano1:~$ skupper -n compss link status Links created from this site: Link link1 is connected Current links from other sites that are connected: There are no connected links ```Deploy test applications in both clusters
The YAML file for the StatefulSet that runs in the edge cluster is ``` apiVersion: v1 kind: Service metadata: name: compss-matmul-4fc9d6 namespace: compss spec: clusterIP: None # This makes it a headless service selector: app: compss wf_id: compss-matmul-4fc9d6 ports: - name: port-22 protocol: TCP port: 22 targetPort: ssh-port --- apiVersion: apps/v1 kind: StatefulSet metadata: name: compss-matmul-4fc9d6-worker namespace: compss spec: selector: matchLabels: app: compss wf_id: compss-matmul-4fc9d6 pod-hostname: worker serviceName: compss-matmul-4fc9d6 replicas: 2 ordinals: start: 2 template: metadata: labels: app: compss wf_id: compss-matmul-4fc9d6 pod-hostname: worker spec: subdomain: compss-matmul-4fc9d6 dnsConfig: searches: - compss-matmul-4fc9d6.compss.svc.cluster.local containers: - name: worker image: albabsc/compss-matmul:verge-0.1.8 command: [ "/usr/sbin/sshd", "-D" ] resources: limits: memory: 2G cpu: 4 ports: - containerPort: 22 name: ssh-port ``` The YAML file for the StatefulSet that runs in the HPC cluster is ``` apiVersion: v1 kind: Service metadata: name: compss-matmul-4fc9d6 namespace: compss spec: clusterIP: None # This makes it a headless service selector: app: compss wf_id: compss-matmul-4fc9d6 ports: - name: port-22 protocol: TCP port: 22 targetPort: ssh-port --- apiVersion: apps/v1 kind: StatefulSet metadata: name: compss-matmul-4fc9d6-worker namespace: compss spec: selector: matchLabels: app: compss wf_id: compss-matmul-4fc9d6 pod-hostname: worker serviceName: compss-matmul-4fc9d6 replicas: 2 template: metadata: labels: app: compss wf_id: compss-matmul-4fc9d6 pod-hostname: worker spec: subdomain: compss-matmul-4fc9d6 dnsConfig: searches: - compss-matmul-4fc9d6.compss.svc.cluster.local containers: - name: worker image: albabsc/compss-matmul:verge-0.1.8 command: [ "/usr/sbin/sshd", "-D" ] resources: limits: memory: 2G cpu: 4 ports: - containerPort: 22 name: ssh-port ```Ensure connection among pods of the same cluster
*Edge cluster* ``` acanete@nano1:~$ kubectl -n compss get pods NAME READY STATUS RESTARTS AGE compss-matmul-4fc9d6-worker-0 1/1 Running 0 4m19s compss-matmul-4fc9d6-worker-1 1/1 Running 0 4m18s skupper-router-6895bb6f95-88hnj 2/2 Running 0 55m skupper-service-controller-559ddbdd56-wnsvh 1/1 Running 0 55m ``` ``` acanete@nano1:~$ kubectl -n compss exec -it compss-matmul-4fc9d6-worker-0 -- bash root@compss-matmul-4fc9d6-worker-0:/# ssh compss-matmul-4fc9d6-worker-1 Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 6.8.0-47-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/pro This system has been minimized by removing packages and content that are not required on a system that users do not log into. To restore this content, you can run the 'unminimize' command. Last login: Wed Nov 6 12:33:29 2024 from 10.244.1.160 root@compss-matmul-4fc9d6-worker-1:~# ``` *HPC cluster* ``` acanete@rpi42:~$ kubectl -n compss get pods NAME READY STATUS RESTARTS AGE compss-matmul-4fc9d6-worker-2 1/1 Running 0 4m48s compss-matmul-4fc9d6-worker-3 1/1 Running 0 4m47s skupper-router-748c487879-gvpxg 2/2 Running 0 56m skupper-service-controller-6f69b974bd-grgzc 1/1 Running 0 56m ``` ``` acanete@rpi42:~$ kubectl -n compss exec -ti compss-matmul-4fc9d6-worker-2 -- bash root@compss-matmul-4fc9d6-worker-2:/# ssh compss-matmul-4fc9d6-worker-3 Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.10.192-tegra aarch64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/pro This system has been minimized by removing packages and content that are not required on a system that users do not log into. To restore this content, you can run the 'unminimize' command. Last login: Wed Nov 6 12:49:41 2024 from 10.244.2.97 root@compss-matmul-4fc9d6-worker-3:~# ```Expose service with Skupper
Command executed in the *edge cluster* ``` skupper -n compss expose service compss-matmul-4fc9d6 --port 22 --address compss-matmul-4fc9d6 ``` Check the service is correctly created ``` acanete@rpi42:~$ skupper -n compss service status Services exposed through Skupper: ╰─ compss-matmul-4fc9d6:22 (tcp) ``` ``` acanete@nano1:~$ skupper -n compss service status Services exposed through Skupper: ╰─ compss-matmul-4fc9d6:22 (tcp) ```DNS resolution no longer works
When trying to ssh between two pods of the same cluster (e.g edge) ``` acanete@nano1:~$ kubectl -n compss get pods NAME READY STATUS RESTARTS AGE compss-matmul-4fc9d6-worker-0 1/1 Running 0 4m19s compss-matmul-4fc9d6-worker-1 1/1 Running 0 4m18s skupper-router-6895bb6f95-88hnj 2/2 Running 0 55m skupper-service-controller-559ddbdd56-wnsvh 1/1 Running 0 55m ``` ``` acanete@nano1:~$ kubectl -n compss exec -it compss-matmul-4fc9d6-worker-0 -- bash root@compss-matmul-4fc9d6-worker-0:/# ssh compss-matmul-4fc9d6-worker-1 ssh: Could not resolve hostname compss-matmul-4fc9d6-worker-1: No address associated with hostname root@compss-matmul-4fc9d6-worker-0:/# ssh compss-matmul-4fc9d6-worker-2 ssh: Could not resolve hostname compss-matmul-4fc9d6-worker-2: No address associated with hostname root@compss-matmul-4fc9d6-worker-0:/# ```Expected behavior I would like for every Pod of a StatefulSet to be accessed through their Pod names, or to know the name I have to use. And to know if the name is different when a Pod in cluster 1 want to access a Pod in cluster 2.
Environment details
Additional context Pods have the the following
/etc/resolv.conffile