multus-service not working - service endpoint always get default network IP address

[yockgen]

Running this demo https://github.com/redhat-nfvpe/multus-service-demo/blob/main/multus-service-demo1.yaml

Multus-Service: kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-service/main/deploy.yml

Can see all pods - nginx and fedora came with two network interfaces, and managed to ping each other.

However, the service has no response from ping on fedora pod: [root@fedora-net1 /]# ping multus-nginx-macvlan PING multus-nginx-macvlan.default.svc.cluster.local (10.233.5.253) 56(84) bytes of data.

The multus-nginx-macvlan's endpoints are eth0 IP instead of net1 IP:

=================================================

root@focal01:~# kubectl describe svc multus-nginx-macvlan Name: multus-nginx-macvlan Namespace: default Labels: service.kubernetes.io/service-proxy-name=multus-proxy Annotations: k8s.v1.cni.cncf.io/service-network: macvlan1 Selector: app=multus-nginx-macvlan Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 10.233.5.253 IPs: 10.233.5.253 Port: 80/TCP TargetPort: 80/TCP Endpoints: 10.233.65.165:80,10.233.66.235:80 <-this is eth0 IP Session Affinity: None Events:

===================================================

Please help.

[s1061123]

First, thank you for the issue to ask us about multus-service.

Currently multus-service uses endpointslice to manage endpoint and it does not use endpoints, so could you please check endpointslice by kubectl get endpointslices.discovery.k8s.io and find your IP. Endpoints are added by Kubernetes as default but these endpoints are not used by service.kubernetes.io/service-proxy-name=multus-proxy. If you use kube-proxy for servicew forwarding, then you cannot find these endpoints iptables-rule in container host.

Regading verification by ping, as Kubernetes service does, multus-service uses iptables to load-balancing and forwarding the request, so ping does not work. Please use curl to get http contents from nginx.

You can find the demo resources in https://github.com/redhat-nfvpe/multus-service-demo and video in the blog, https://cloud.redhat.com/blog/how-to-use-kubernetes-services-on-secondary-networks-with-multus-cni and I hope that it helps you to try multus-servicve.

[yockgen]

Thanks for the quick response! Yes, I've followed the https://cloud.redhat.com/blog/how-to-use-kubernetes-services-on-secondary-networks-with-multus-cni since few days back.

The endpointslices looks like below:

==================================

root@focal01:~# kubectl get endpointslices.discovery.k8s.io NAME ADDRESSTYPE PORTS ENDPOINTS AGE kubernetes IPv4 6443 192.168.222.88 32d multus-nginx-macvlan-multus-cpqlz IPv4 80 10.2.128.3,10.2.128.2 39s <- this is net1 IPs multus-nginx-macvlan-sjncf IPv4 80 10.233.66.239,10.233.65.198 39s <- this is eth0 IPs

=====================================

However, the issue still the same, CURL just run forever on the multus service multus-nginx-macvlan:

=====================================

root@focal01:~# kubectl exec -it fedora-net1 -- bash [root@fedora-net1 /]# curl multus-nginx-macvlan.default.svc.cluster.local

======================================

Appreciate your further guidance.

Thanks!

[yockgen]

I'm further checking , noticed that all multus-proxy in CrashLoopBackOff status, checked the log, it is mentioned about failed to crio socket, however, I'm using containerd as runtime, is it related?

==============================================

root@focal01:~# kubectl logs -f multus-proxy-ds-amd64-x2qlm -n kube-system E0520 08:22:34.224192 1 pod.go:388] failed to get cri client: failed to connect: failed to connect to unix:///host/run/crio/crio.sock, make sure you are running as root and the runtime has been started: context deadline exceeded F0520 08:22:34.224738 1 main.go:61] cannot create pod change tracker

=============================================================

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node1 Ready control-plane,master 32d v1.23.4 192.168.222.88 Ubuntu 20.04.4 LTS 5.4.0-1063-kvm containerd://1.6.1 node2 Ready 32d v1.23.4 192.168.222.81 Ubuntu 20.04.4 LTS 5.4.0-1063-kvm containerd://1.6.1 node3 Ready 32d v1.23.4 192.168.222.227 Ubuntu 20.04.4 LTS 5.4.0-1063-kvm containerd://1.6.1

==================================================================

Please comment.

Thanks!

[s1061123]

Hi, so as your output, endpointslice contains multus endpoint. Using the endpointslice, multus-proxy configures the forwarding rules for the service, but as you mentioned multus-proxy is not worked yet. So please fix that as error message tells.

As we described in 'Current Status of Repository', currently the repository is too-early phase in development, so currently we do not supported how-to-deploy question because we lack these information yet, but thank you for the feedback. We will add some information (how to troubleshoot) in future.