search

Azure / azure-container-networking

Azure Container Networking Solutions for Linux and Windows Containers
MIT License
377 stars 240 forks source link

need help to configure azure cni on a onpremise/baremetal kubernetes installation in azure #812

Closed lightmans2 closed 2 years ago

lightmans2 commented 3 years ago

Hello together,

I am currently involved into a dev project in azure for kubernetes. We have build there 3VMs with kubernetes (master) and 3VMs with kubernetes (worker). We dont use azure aks cluster because we want to lear and get more experience with kubernetes. I already installed two onpremise kuberentes installations with cni flannel. But on azure its different because i need to manage the azure network with the azure cni plugin

I already initaliazed the kubernetes cluster on one host. We installed azure cli on all kubernetes master nodes and activated it. I installed succesfully the azure cni plugin as described in the manual. We can see under /var/log/azure-vmnet.log that the plugin tries to communicate with azure. In azure i already added a subnet 192.168.1.0/24 specially for the pod cidr.

How i need to configure OR where exactly i need to configure the azure cni plugin? Where or how i tell the plugin that he can use this subnet created? Do i need to bridge or create a bridge interface manually for the plugin or does the cni plugin configuration automation process this for me, like in flannel or others cni plugins?

Here some infos:

[root@kub-master-01 jose]# kubectl get po,svc,nodes --all-namespaces
NAMESPACE     NAME                                        READY   STATUS              RESTARTS   AGE
kube-system   pod/coredns-74ff55c5b-6mq9c                 0/1     ContainerCreating   0          18h
kube-system   pod/coredns-74ff55c5b-qvt7j                 0/1     ContainerCreating   0          18h
kube-system   pod/etcd-kub-master-01                      1/1     Running             4          25h
kube-system   pod/kube-apiserver-kub-master-01            1/1     Running             4          25h
kube-system   pod/kube-controller-manager-kub-master-01   1/1     Running             5          25h
kube-system   pod/kube-proxy-8bzdt                        1/1     Running             4          25h
kube-system   pod/kube-proxy-n996m                        1/1     Running             1          20h
kube-system   pod/kube-scheduler-kub-master-01            1/1     Running             5          25h

NAMESPACE     NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes                    ClusterIP   10.96.0.1        <none>        443/TCP                  25h
kube-system   service/kube-dns                      ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP   25h
kube-system   service/npm-metrics-cluster-service   ClusterIP   10.104.173.183   <none>        9000/TCP                 22h

NAMESPACE   NAME                 STATUS   ROLES                  AGE   VERSION
            node/kub-master-01   Ready    control-plane,master   25h   v1.20.4
            node/kub-work-01     Ready    <none>                 20h   v1.20.4

[root@kub-master-01 jose]# cat /etc/cni/net.d/10-azure.conflist  
{
   "cniVersion":"0.3.0",
   "name":"azure",
   "plugins":[
      {
         "type":"azure-vnet",
         "mode":"transparent",
         "ipsToRouteViaHost":["169.254.20.10"],
         "ipam":{
            "type":"azure-vnet-ipam"
         }
      },
      {
         "type":"portmap",
         "capabilities":{
            "portMappings":true
         },
         "snat":true
      }
   ]
}

[root@kub-master-01 jose]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0d:3a:fe:9c:7d brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.5/24 brd 192.168.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master eth0 state UP group default qlen 1000
    link/ether 00:0d:3a:fe:9c:7d brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:4b:41:3f:f8 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

[root@kub-master-01 jose]# brctl show
bridge name bridge id       STP enabled interfaces
docker0     8000.02424b413ff8   no  

[root@kub-master-01 jose]# tail -f /var/log/azure-vnet.log
2021/03/04 11:36:00 [48691] [Azure-Utils] ebtables --version
2021/03/04 11:36:00 [48691] [cni-net] ebtable version ebtables v2.0.10-4 (December 2011), err:<nil>
2021/03/04 11:36:00 [48691] [net] Network interface: {Index:1 MTU:65536 Name:lo HardwareAddr: Flags:up|loopback} with IP: [127.0.0.1/8]
2021/03/04 11:36:00 [48691] [net] Network interface: {Index:2 MTU:1500 Name:eth0 HardwareAddr:00:0d:3a:fe:9c:7d Flags:up|broadcast|multicast} with IP: [192.168.0.5/24]
2021/03/04 11:36:00 [48691] [net] Network interface: {Index:3 MTU:1500 Name:eth1 HardwareAddr:00:0d:3a:fe:9c:7d Flags:up|broadcast|multicast} with IP: []
2021/03/04 11:36:00 [48691] [net] Network interface: {Index:4 MTU:1500 Name:docker0 HardwareAddr:02:42:4b:41:3f:f8 Flags:up|broadcast|multicast} with IP: [172.17.0.1/16]
2021/03/04 11:36:00 [48691] [net] network store key not found
2021/03/04 11:36:00 [48691] [cni-net] Plugin started.
2021/03/04 11:36:00 [48691] CNI_COMMAND environment variable set to VERSION
2021/03/04 11:36:00 [48691] [cni-net] Plugin stopped.
2021/03/04 11:36:05 [48721] [cni] reboot time 2021-03-04 08:34:11 +0000 UTC
2021/03/04 11:36:05 [48721] Connected to telemetry service
2021/03/04 11:36:05 [48721] [cni-net] Plugin azure-vnet version v1.2.6.
2021/03/04 11:36:05 [48721] [cni-net] Running on Linux version 3.10.0-1160.15.2.el7.x86_64 (mockbuild@x86-034.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Thu Jan 21 16:15:07 EST 2021
2021/03/04 11:36:05 [48721] [Azure-Utils] iptables --version
2021/03/04 11:36:05 [48721] [cni-net] iptable version:iptables v1.4.21, err:<nil>
2021/03/04 11:36:05 [48721] [Azure-Utils] ebtables --version
2021/03/04 11:36:05 [48721] [cni-net] ebtable version ebtables v2.0.10-4 (December 2011), err:<nil>
2021/03/04 11:36:05 [48721] [net] Network interface: {Index:1 MTU:65536 Name:lo HardwareAddr: Flags:up|loopback} with IP: [127.0.0.1/8]
2021/03/04 11:36:05 [48721] [net] Network interface: {Index:2 MTU:1500 Name:eth0 HardwareAddr:00:0d:3a:fe:9c:7d Flags:up|broadcast|multicast} with IP: [192.168.0.5/24]
2021/03/04 11:36:05 [48721] [net] Network interface: {Index:3 MTU:1500 Name:eth1 HardwareAddr:00:0d:3a:fe:9c:7d Flags:up|broadcast|multicast} with IP: []
2021/03/04 11:36:05 [48721] [net] Network interface: {Index:4 MTU:1500 Name:docker0 HardwareAddr:02:42:4b:41:3f:f8 Flags:up|broadcast|multicast} with IP: [172.17.0.1/16]
2021/03/04 11:36:05 [48721] [net] network store key not found
2021/03/04 11:36:05 [48721] [cni-net] Plugin started.
2021/03/04 11:36:05 [48721] CNI_COMMAND environment variable set to VERSION
2021/03/04 11:36:05 [48721] [cni-net] Plugin stopped.

[root@kub-master-01 log]# tail -f azure-vnet-ipam.log
2021/03/03 16:53:44 [6985] [ipam] Refreshing address source.
2021/03/03 16:53:44 [6985] [Utils] Initializing HTTP client with connection timeout: 10, response header timeout: 10
2021/03/03 16:53:44 [6985] [ipam] Wireserver call http://168.63.129.16/machine/plugins?comp=nmagent&type=getinterfaceinfov1 to retrieve IP List
2021/03/03 16:53:44 [6985] [ipam] got 0 addresses from interface eth0, subnet 192.168.0.0/24
2021/03/03 16:53:44 [6985] [ipam] merging address space
2021/03/03 16:53:44 [6985] [ipam] saving ipam state.
2021/03/03 16:53:44 [6985] [ipam] Save succeeded.
2021/03/03 16:53:44 [6985] [azure-vnet-ipam] Failed to release address: Pool id  not found :Invalid address pool.
2021/03/03 16:53:44 [6985] [cni-ipam] DEL command completed with err:Failed to release address: Pool id  not found :Invalid address pool.
2021/03/03 16:53:44 [6985] [cni-ipam] Plugin stopped

thx in advance for all help and answers Jose

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

github-actions[bot] commented 2 years ago

Issue closed due to inactivity.