search

flannel-io / flannel

flannel is a network fabric for containers, designed for Kubernetes
Apache License 2.0
8.82k stars 2.87k forks source link

Got duplicate ip on flannel 0.7.0 #721

Closed sysit closed 7 years ago

sysit commented 7 years ago

Expected Behavior

when I use kubectl get pods --all-namespace -o wide , find some duplicate ip for some pods

`xqy-finance   xqy-front-w4cd9                         1/1       Running   5          2h        10.2.55.10      10.199.134.36
xqy-boss      xqy-front-7dv9v                         1/1       Running   7          2h        10.2.55.12      10.199.134.36
xqy-finance   xqy-gateway-nf4gk                       1/1       Running   1          5d        10.2.55.12      10.199.134.36
xqy-qa3       xqy-manage-service-c6wnp                1/1       Running   0          2h        10.2.55.16      10.199.134.36
xqy-ptax      xqy-finance-service-ghqxz               1/1       Running   0          1h        10.2.55.18      10.199.134.36
xxb-data      xxb-cxtj-tfbft                          1/1       Running   0          7m        10.2.55.19      10.199.134.36
xqy-boss      xqy-middleware-b5qsk                    3/3       Running   0          4d        10.2.55.27      10.199.134.36
xqy-qa3       xqy-middleware-d085z                    3/3       Running   0          4d        10.2.55.39      10.199.134.36
xqy-manage    xqy-middleware-cdhzp                    3/3       Running   0          4d        10.2.55.46      10.199.134.36
xqy-manage    xqy-boss-service-7s7kk                  1/1       Running   0          4d        10.2.55.47      10.199.134.36
xqy-manage    xqy-gateway-rwv4d                       1/1       Running   0          4d        10.2.55.48      10.199.134.36
xqy-manage    xqy-tax-service-dh3z2                   1/1       Running   0          4d        10.2.55.49      10.199.134.36
xqy-finance   xqy-finance-service-cc363               1/1       Running   0          2h        10.2.55.5       10.199.134.36
xqy-finance   xqy-manage-service-r8w1l                1/1       Running   1          4d        10.2.55.5       10.199.134.36
xqy-boss      xqy-gateway-fj3d2                       1/1       Running   0          4d        10.2.55.52      10.199.134.36
xqy-boss      xqy-support-publish-66kqd               1/1       Running   3          7d        10.2.55.6       10.199.134.36
xqy-tax       xqy-gateway-f4n85                       1/1       Running   0          2h        10.2.55.6       10.199.134.36
xqy-qa3       xqy-boss-service-chmc4                  1/1       Running   0          2h        10.2.55.8       10.199.134.36
xqy-qa3       xqy-tax-service-5rzs7                   1/1       Running   0          2h        10.2.55.9       10.199.134.36
xqy-boss      xqy-boss-service-735cs                  1/1       Running   0          9m        10.2.94.5       10.199.134.34
kube-system   nginx-ingress-controller-3v7g8          1/1       Running   7          33d       10.199.134.35   10.199.134.35
kube-system   kube-dns-autoscaler-2813114833-4k0z4    1/1       Running   3          19d       10.2.2.13       10.199.134.35
px-17win      news-17win-rc-rql8p                     1/1       Running   3          26d       10.2.48.13      10.199.134.32
kube-system   default-http-backend-mj532              1/1       Running   15         90d       10.2.48.17      10.199.134.32
test          frontend-lnmhr                          1/1       Running   2          18d       10.2.55.10      10.199.134.36
kube-system   default-http-backend-58x1k              1/1       Running   42         19d       10.2.55.13      10.199.134.36
test          redis-slave-1httf                       1/1       Running   2          18d       10.2.55.21      10.199.134.36
kube-system   kube-dns-782804071-0ml9l                4/4       Running   14         19d       10.2.55.7       10.199.134.36
xqy-tax       xqy-front-qqb51                         1/1       Running   1          15m       10.2.94.3       10.199.134.34`

Current Behavior

`kube-node6 ~ # journalctl -u flanneld
-- Logs begin at Sat 2017-05-13 12:03:15 CST, end at Mon 2017-05-15 17:34:26 CST. --
May 13 18:58:51 kube-node6 flannel-wrapper[1360]: I0513 10:58:51.182160    1360 network.go:160] Lease renewed, new expiration: 2017-05-14 10:58:51.183067508 
May 14 17:58:51 kube-node6 flannel-wrapper[1360]: I0514 09:58:51.188299    1360 network.go:160] Lease renewed, new expiration: 2017-05-15 09:58:51.183912306 
May 15 16:58:51 kube-node6 flannel-wrapper[1360]: I0515 08:58:51.191772    1360 network.go:160] Lease renewed, new expiration: 2017-05-16 08:58:51.185494848 
kube-node6 ~ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:23:31:36 brd ff:ff:ff:ff:ff:ff
    inet 10.199.134.36/24 brd 10.199.134.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe23:3136/64 scope link 
       valid_lft forever preferred_lft forever
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 76:56:62:77:6b:f0 brd ff:ff:ff:ff:ff:ff
    inet 10.2.55.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::7456:62ff:fe77:6bf0/64 scope link 
       valid_lft forever preferred_lft forever`

the flanneld start service :

`kube-node6 ~ # systemctl cat flanneld
# /usr/lib/systemd/system/flanneld.service
[Unit]
Description=flannel - Network fabric for containers (System Application Container)
Documentation=https://github.com/coreos/flannel
After=etcd.service etcd2.service etcd-member.service
Requires=flannel-docker-opts.service

[Service]
Type=notify
Restart=always
RestartSec=10s
LimitNOFILE=40000
LimitNPROC=1048576

Environment="FLANNEL_IMAGE_TAG=v0.7.0"
Environment="FLANNEL_OPTS=--ip-masq=true"
Environment="RKT_RUN_ARGS=--uuid-file-save=/var/lib/coreos/flannel-wrapper.uuid"
EnvironmentFile=-/run/flannel/options.env

ExecStartPre=/sbin/modprobe ip_tables
ExecStartPre=/usr/bin/mkdir --parents /var/lib/coreos /run/flannel
ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/lib/coreos/flannel-wrapper.uuid
ExecStart=/usr/lib/coreos/flannel-wrapper $FLANNEL_OPTS
ExecStop=-/usr/bin/rkt stop --uuid-file=/var/lib/coreos/flannel-wrapper.uuid

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/flanneld.service.d/40-ExecStartPre-symlink.conf.conf
[Service]
ExecStartPre=/usr/bin/ln -sf /etc/flannel/options.env /run/flannel/options.env
# /etc/systemd/system/flanneld.service.d/flannel_opt.conf
Environment="FLANNEL_OPTS=-v=1"`
kube-node6 ~ # cat /etc/flannel/options.env 
FLANNELD_IFACE=10.199.134.36
FLANNELD_ETCD_ENDPOINTS=http://10.199.134.31:2379,http://10.199.134.33:2379,http://10.199.134.35:2379
kube-node6 ~ # etcdctl get /coreos.com/network/config
{"Network":"10.2.0.0/16","Backend":{"Type":"vxlan"}}
kube-node6 ~ # cat /etc/kubernetes/cni/net.d/10-flannel.conf 
{
    "name": "podnet",
    "type": "flannel",
    "delegate": {
        "isDefaultGateway": true
    }
}

Steps to Reproduce (for bugs)

The cluster have been runned under flannel v0.6.2 , when the container linux update to version 1353. the flanneld update automated. now some app abnormal.

Your Environment

tomdee commented 7 years ago

Flannel doesn't assign IP addresses to pods. Maybe you have a problem with host-local IPAM plugin - in which case you should raise it in the containernetworking repo.