Multi-Host Network Across Worker Nodes

infinitydon commented 5 years ago

Hello,

I am trying to create a set of network subnets that will be available to all the worker nodes. An example:

net1 with IP 10.10.10.1/24 in POD1 (running in worker 1) should be able to ping net1 with IP 10.10.10.2/24 in POD1 (running in worker 2).

net2 with IP 10.10.20.1/24 in POD1 (running in worker 1) should be able to ping net2 with IP 10.10.20.2/24 in POD1 (running in worker 2).

N.B -- I will not be using SR-IOV.

A sample architecture (from the knitter project):

Levovar commented 5 years ago

not sure about the background of your query, but I will try and answer it from the perspective of what DANM can do for you :) DanmNets are namespaced resources, and also store the IP related information in the same API. As a result IP management in DANM -for dynamic backends- is not tied to any host. Taking your use-case as a base:

if Pod1 and Pod2 belong to the same K8s namespace, they simply connect to the same DanmNet, and ask for dynamic IPs from the same subnet. No need to create a CIDR/Node. Assuming your underlying, host-level network fabric is properly configured, the two Pods will be able to ping each other
If they don't belong to the same Namespace, you need to create one DanmNet in both of their namespaces, provide the same CIDR, but provision different allocation_pools into each DanmNet. This will effectively partition your CIDR per network - but still not per Node!
If you are adamant of separating your networks into multiple CIDRs, ofc you can also do that. In this case you might need to provision IP routes into your containers, but fortunately DANM also supports this (both static, and dynamic IP routes) Hope this clarifies!

Reg your earlier query about VLANs: I assume you meant the "normal" VLAN CNI? Because DANM does not use that, it takes care of VLAN provisioning on its own. As a result, it definitely allows you to connect multiple Pods to the same VLAN on the same host - to the VLAN configured into their DanmNet.

infinitydon commented 5 years ago

Thanks @Levovar ..

Will try the first suggestion:

if Pod1 and Pod2 belong to the same K8s namespace, they simply connect to the same DanmNet, and ask for dynamic IPs from the same subnet. No need to create a CIDR/Node.
Assuming your underlying, host-level network fabric is properly configured, the two Pods will be able to ping each other

Levovar commented 5 years ago

I see you have opened the same Issue on the Multus project as well, so I guess this is the reason behind the somewhat generic nature of the query :) Highlighting the differences between the two projects for your reference:

in Multus you will need to use a separate, CNI compatible IPAM module supporting this host-independent IP management use-case; and then configure into the CNI spec of all your networks. AFAIK there aren't many available, though something was done recently with Whereabouts (not sure ow prod ready it is in its current format). With DANM you don't depend on such an external plugin, as DANM provides this for you out of the box, integrated into the network management API
yeah reg VLAN as I stated above. If you want to place your container interfaces over VLANs with Multus your probably need to chain together multiple CNIs to achieve the effect: they might work, might not. With DANM your VLANs are automatically managed by Netwatcher component again based on the same API, and your MACVLAN /IPVLAN/SR-IOV interfaces are even connected to the right VLAN. Same if you would decide to use VxLANs with MACVLAN/IPVLAN backends. SR-IOV I think would work out-of-box even with Multus, as tagging the VFs is taken care of by the sriov-cni. Therefore there is no need to chain in this case

infinitydon commented 5 years ago

The background of my request is learning how to deploy GPRS nodes like GGSN,SGSN,MME,PGW,SGW etc inside kubernetes..

So I want to be able to create network partitions like Gn, Gx, S11,S5 etc..

But currently I don't have access to sr-iov capable network card.

Levovar commented 5 years ago

Ah, I see! DANM is used inside my company to deploy MME, UPF, TAS, CSF, HLR&HSS in different environments, so I'm hopeful it will be able to satisfy your use-case as well!

One additional info you might find useful if you are looking into TelCo specific use-cases: lots of ETSI standards require the provisioning of static IPs on the aforementioned VNF's external interfaces, and do not allow dynamic service discovery to exist between them. To serve these use-case DANM also supports the static allocation of the desired IPs to some interfaces. Of course this will make the requesting micro-service unscalable, but properly partitioning a VNF into micro-services (loadbalancers + workers) can mitigate the issue.

infinitydon commented 5 years ago

So I was able to install DANM (though I will suggest a simpler approach, I think the current learning curve is a bit high :) )..

1.) Now the IP allocation is unique across the worker nodes (just as you explained that damnets of the same namespace will provide unique IP irrespective of the worker node).

2.) Connectivity -- I played with the example applications (svcwatcher_demo) but still PODs in different worker nodes still can't ping each other. IPVLAN was used.

What exactly do you mean by "Assuming your underlying, host-level network fabric is properly configured, the two Pods will be able to ping each other". Are there any further things configure to make the connectivity to work?

Also I think Canal makes things a bit cumbersome, I will also try to use Flannel as the base CNI.

Levovar commented 5 years ago

we have an auto-installer project ongoing, that should ease the curve. I agree it takes some effort, but hopefully the feature set worth the effort :)

Regarding connectivity: usual stuff for a flat L3 network needs to be set-up in your switch e.g. VLANs for the ports, proper IP routes between subnets (if,any) etc. You also need to be familiar with how IPVLAN works:

IPVLAN slaves have their own IPs directly allocated therefore the master interface shall not have an IP on its own, otherwise switching won't work
IPVLAN slaves are not routable from the same host's host network namespace, therefore the clusterIP routing part of the Kubernetes Services do not work with them IPVLANs are mostly used as datapath interfaces, they are not really suitable for OAM kind of access.

infinitydon commented 5 years ago

The feature set is very promising to be honest (I like to tag danm as multus on steroids :) )...

I think am almost through.. With IPVLAN, can I allocate static IP to the POD manifest? If yes, please provide reference on how to do this with the annotation (also danmnet configuration)..

infinitydon commented 5 years ago

Tried to set a static IP but I got the following error:

vents:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               6s    default-scheduler  Successfully assigned nextepc/nextepc-hss-deployment-fcf6f7576-n8s8c to k8s-3
  Warning  FailedCreatePodSandBox  3s    kubelet, k8s-3     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "eeddbb4866ad266f0a47905f3ac40beffb8a3f389c411ec41b5f055dedbd9981" network for pod "nextepc-hss-deployment-fcf6f7576-n8s8c": NetworkPlugin cni failed to set up pod "nextepc-hss-deployment-fcf6f7576-n8s8c_nextepc" network: CNI network could not be set up: CNI operation for network:s6a failed with:IP address reservation failed for network:s6a with error:failed to allocate IP address for network:s6a with error:static IP cannot be allocated for a L2 network!

My annotation config:

Annotations: danm.k8s.io/interfaces: [ {"network":"management", "ip":"dynamic"}, {"network":"s6a", "ip":"192.168.20.2/24"} ]

Levovar commented 5 years ago

CIDR is missing from your DanmNet. IPs can be only allocated from the defined allocation range

infinitydon commented 5 years ago

Thanks.. All pods are running now the way they should!

nokia / danm

Multi-Host Network Across Worker Nodes #92