I have a deployment setup running across multiple Proxmox VMs each running a K3S cluster. One of the VMs has a Docker container running a simulator for sending data to the clusters. Each Cluster consists of a single node with 6 pods and 6 services as viewed in the attached image. One of the pods (pod 6) needs to communicate with it's counterparts on the other nodes via multicast (represented by the blue line).
I am using Flannel and Multus to accomplish this. I created the following Network Attachment Definition
and I added this annotation to my pods, with different IPs and MAC addresses on each pod of course.
With this setup I can exec into one of the pods and ping the other pods and VMs, but using IPERF to test Multicast fails.
Here is the command and output for starting the Multicast subscriber:
and the publisher :
It appears that the problem is that the publisher is defaulting to the internal Flannel network (10.42.0.186) instead of the network I attached. I do not know how to fix this.
Note: I did try doing this without Multus and just using "hostNetwork: true" in my pod which did allow me to send multicast data between the pods. But the side effect was that the comms from the Docker container to the attached SVC 6 no longer works. If I could figure out how to get the data directly to the pod without the service, this might also solve my problem, though I would prefer to go the Multus route.
I have a deployment setup running across multiple Proxmox VMs each running a K3S cluster. One of the VMs has a Docker container running a simulator for sending data to the clusters. Each Cluster consists of a single node with 6 pods and 6 services as viewed in the attached image. One of the pods (pod 6) needs to communicate with it's counterparts on the other nodes via multicast (represented by the blue line).
I am using Flannel and Multus to accomplish this. I created the following Network Attachment Definition
and I added this annotation to my pods, with different IPs and MAC addresses on each pod of course.
With this setup I can exec into one of the pods and ping the other pods and VMs, but using IPERF to test Multicast fails.
Here is the command and output for starting the Multicast subscriber:
and the publisher :
It appears that the problem is that the publisher is defaulting to the internal Flannel network (10.42.0.186) instead of the network I attached. I do not know how to fix this.
Note: I did try doing this without Multus and just using "hostNetwork: true" in my pod which did allow me to send multicast data between the pods. But the side effect was that the comms from the Docker container to the attached SVC 6 no longer works. If I could figure out how to get the data directly to the pod without the service, this might also solve my problem, though I would prefer to go the Multus route.
Any assistance would be appreciated.