search

futurewei-cloud / Distrinet

Distributed Network emulator, based on Mininet
MIT License
3 stars 6 forks source link

Deploy Alcor k8s cluster and connect microservices to ACA containers #24

Open cj-chung opened 2 years ago

cj-chung commented 2 years ago

This task provides an alternative for Alcor lxc containers in Distrinet. If Alcor lxc containers are hard to reach, we can implement Alcor k8s cluster and try to connect these microservices to ACAs containers in Distrinet.

Requirements:

  1. Allocate 1 or 2 physical machines.
  2. Setup a Kubernetes (k8s) Cluster using Ansible script (will provide later).
  3. Deploy Alcor services pods in the k8s cluster
  4. Test connection from Alcor services to ACAs
gethurb commented 2 years ago

Just entered this project a few days ago. During this time, I have learned the deployment of K8S and the use of Distrinet. I am preparing to deploy Alcor Controller independently to provide guarantee for the subsequent work.

gethurb commented 2 years ago

the deployment and test of controller cluster is still in progress

gethurb commented 2 years ago

The current test shows that the traffic from k8s can reach aca in lxd. We can simply forward port on lxd and then let kubenetes take over the rest. But since the current version of dpm can only connect to one port on the corresponding aca node, we cannot deploy multiple lxd containers on one node. There are two options I can think of at the moment：

  1. Modify the code of the relevant parts of dpm so that dpm can establish multiple grpc connections with the same aca node
  2. We implement a kube-proxy like module ourselves, register the ip of the lxd container into alcor, and then configure the corresponding iptables rules on every aca node. One problem with this solution is that we are not sure whether this will conflict iptables rules configured by kube-proxy.
cj-chung commented 2 years ago

I think the easiest way to connect alcor services in k8s and aca in lxc container is a gateway or router between two different network segment.

gethurb commented 2 years ago

I think the easiest way to connect alcor services in k8s and aca in lxc container is a gateway or router between two different network segment.

  • For the connection from aca containers to alcor services, you just need to allow lxc container connecting to the Alcor's service IP.
  • For the grpc connection from networkconfigmanager in Alcor to aca containers, you probably need to setup a gateway between them. You can refer to this link for the gateway configuration: https://unix.stackexchange.com/questions/222054/how-can-i-use-linux-as-a-gateway

Sorry, I had read the page you supported and don't totally understand what do you mean. According to my understanding, we use aca as an internal service, and the node where aca is located as a gateway, but this approach requires dpm or ncm to be able to establish grpc connections to multiple ports. The current code shows grpc connections can only be made on port 50001.

cj-chung commented 2 years ago

You don't need to assign public IP for each ACA, you just need to create a gateway and make this gateway two network interfaces, one for public ip and the other one for lxc container's network ip. Then make a static routing rule in each Networkconfigmanager in Alcor, if they need to reach aca's network ip, forward the packet to the gateway container. The link I provided is how to setup the iptables to forward the packet in the gateway.