search

gaopeiliang / questions

0 stars 0 forks source link

calico service cidr 的新玩法 #1

Open gaopeiliang opened 4 years ago

gaopeiliang commented 4 years ago

k8s 的 Services 通过 kube-proxy 组件处理，只能在集群内部访问，如果想在外部访问，只能通过NodePort和load balancer， calico 通过BGP将 Services CIDR广播出去，通过 ECMP 达到负载的效果，同时使用local services 还能防止SNAT丢失原IP

gaopeiliang commented 4 years ago

使用的方法只需要在calico-node 上设置 

CALICO_ADVERTISE_CLUSTER_IPS=<Service CIDR>
gaopeiliang commented 4 years ago

image

如上图片所示，设置CIDR 为 10.96.0.0/12 外部学习到的路由如下：

node-external~$ ip r
default via 10.192.0.1 dev eth0
10.96.0.0/12 proto bird
  nexthop via 10.192.0.2  dev eth0 weight 1
  nexthop via 10.192.0.3  dev eth0 weight 1
  nexthop via 10.192.0.4  dev eth0 weight 1
10.192.0.0/24 dev eth0 proto kernel scope link src 10.192.0.5
192.168.135.128/26 via 10.192.0.3 dev eth0 proto bird
192.168.169.128/26 via 10.192.0.4 dev eth0 proto bird
192.168.221.192/26 via 10.192.0.2 dev eth0 proto bird

如果创建某个服务，同时设置local services：

kube-master~$ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
kube-master~$ kubectl create service nodeport nginx --tcp 80:80
service/nginx created

kube-master~$ kubectl patch service nginx -p '{"spec":{"externalTrafficPolicy":"Local"}}'
service/nginx patched

我们会得到一条/32的精确路由：

node-external~$ ip r
default via 10.192.0.1 dev eth0
10.96.0.0/12 proto bird
  nexthop via 10.192.0.2  dev eth0 weight 1
  nexthop via 10.192.0.3  dev eth0 weight 1
  nexthop via 10.192.0.4  dev eth0 weight 1
10.96.0.1 via 10.192.0.3 dev eth0 proto bird
10.192.0.0/24 dev eth0 proto kernel scope link src 10.192.0.5
192.168.135.128/26 via 10.192.0.3 dev eth0 proto bird
192.168.169.128/26 via 10.192.0.4 dev eth0 proto bird
192.168.221.192/26 via 10.192.0.2 dev eth0 proto bird

如果服务多副本，那么这条精确路由同样遵循ECMP：
kube-master~$ kubectl scale --replicas=2 deployment/nginx
deployment.extensions/nginx scaled

node-external~$ ip r
default via 10.192.0.1 dev eth0
10.96.0.0/12 proto bird
  nexthop via 10.192.0.2  dev eth0 weight 1
  nexthop via 10.192.0.3  dev eth0 weight 1
  nexthop via 10.192.0.4  dev eth0 weight 1
10.96.0.1 proto bird
  nexthop via 10.192.0.3  dev eth0 weight 1
  nexthop via 10.192.0.4  dev eth0 weight 1
10.192.0.0/24 dev eth0 proto kernel scope link src 10.192.0.5
192.168.135.128/26 via 10.192.0.3 dev eth0 proto bird
192.168.169.128/26 via 10.192.0.4 dev eth0 proto bird
192.168.221.192/26 via 10.192.0.2 dev eth0 proto bird