search

google / seesaw

Seesaw v2 is a Linux Virtual Server (LVS) based load balancing platform.
Apache License 2.0
5.63k stars 511 forks source link

read: connection reset by peer #111

Closed salamanderrex closed 2 years ago

salamanderrex commented 3 years ago

Hi, i am trying to put my docker private regsitry service behind VIP so i can pull docker images from it. It looks like seesaw -> haproxy in a contaienr -> registry conatiner

10.162.11.103 is the VIP. It works for normal curl for web page or pulling small images. However, if i pull big image, i will see errors from time to time.

logs for docker clicent pulling image.

14 11:42:47 xxxxxx dockerd[605]: time="2021-06-14T11:42:47.197009668+08:00" level=error msg="Download failed, retrying: read tcp 10.160.8.47:51772->10.162.11.103:443: read: connection reset by peer"

I used ss to monitor the ports, so my client tries to use 51772 port to pull from my vip. 

root@qingyu-prober:~# ss -tiepm | grep docker
ESTAB  3995947    0               10.160.8.47:51772       10.162.11.103:https    users:(("dockerd",pid=605,fd=24)) timer:(keepalive,2.128ms,0) ino:23494 sk:25 <->
ESTAB  3257168    0               10.160.8.47:51770       10.162.11.103:https    users:(("dockerd",pid=605,fd=22)) timer:(keepalive,29sec,0) ino:23493 sk:26 <->

However, on my seesaw nodes, i do not see the ports from my client ip for that port. ( i made ipvsadm timeout 2 hour so i expect all connections should be there. but i do not see the port 51772 )

root@seesaw-01:/usr/local/seesaw# ipvsadm -lcn  | grep '.47:'
TCP 110:32 FIN_WAIT    10.160.8.47:51624  10.162.11.103:443  10.162.11.113:443
TCP 110:41 FIN_WAIT    10.160.8.47:51644  10.162.11.103:443  10.162.11.113:443
TCP 110:46 FIN_WAIT    10.160.8.47:51658  10.162.11.103:443  10.162.11.113:443
TCP 110:31 FIN_WAIT    10.160.8.47:51614  10.162.11.103:443  10.162.11.113:443
TCP 110:41 FIN_WAIT    10.160.8.47:51636  10.162.11.103:443  10.162.11.113:443
TCP 110:41 FIN_WAIT    10.160.8.47:51648  10.162.11.103:443  10.162.11.113:443
TCP 114:43 ESTABLISHED 10.160.8.47:51660  10.162.11.103:443  10.162.11.113:443
TCP 110:32 FIN_WAIT    10.160.8.47:51618  10.162.11.103:443  10.162.11.113:443
TCP 110:46 FIN_WAIT    10.160.8.47:51656  10.162.11.103:443  10.162.11.113:443
TCP 110:32 FIN_WAIT    10.160.8.47:51632  10.162.11.103:443  10.162.11.113:443
TCP 110:48 FIN_WAIT    10.160.8.47:51650  10.162.11.103:443  10.162.11.113:443
TCP 109:43 FIN_WAIT    10.160.8.47:51612  10.162.11.103:443  10.162.11.113:443
TCP 110:41 FIN_WAIT    10.160.8.47:51646  10.162.11.103:443  10.162.11.113:443
TCP 110:32 FIN_WAIT    10.160.8.47:51634  10.162.11.103:443  10.162.11.113:443
TCP 110:57 FIN_WAIT    10.160.8.47:51652  10.162.11.103:443  10.162.11.113:443
TCP 114:43 ESTABLISHED 10.160.8.47:51662  10.162.11.103:443  10.162.11.113:443
TCP 110:32 FIN_WAIT    10.160.8.47:51626  10.162.11.103:443  10.162.11.113:443
TCP 110:57 FIN_WAIT    10.160.8.47:51664  10.162.11.103:443  10.162.11.113:443
TCP 110:23 FIN_WAIT    10.160.8.47:51610  10.162.11.103:443  10.162.11.113:443
TCP 110:45 FIN_WAIT    10.160.8.47:51638  10.162.11.103:443  10.162.11.113:443
TCP 110:32 FIN_WAIT    10.160.8.47:51628  10.162.11.103:443  10.162.11.113:443
TCP 110:42 FIN_WAIT    10.160.8.47:51640  10.162.11.103:443  10.162.11.113:443
TCP 110:45 FIN_WAIT    10.160.8.47:51654  10.162.11.103:443  10.162.11.113:443
TCP 110:31 FIN_WAIT    10.160.8.47:51616  10.162.11.103:443  10.162.11.113:443
TCP 110:32 FIN_WAIT    10.160.8.47:51630  10.162.11.103:443  10.162.11.113:443
TCP 110:32 FIN_WAIT    10.160.8.47:51620  10.162.11.103:443  10.162.11.113:443
TCP 110:33 FIN_WAIT    10.160.8.47:51622  10.162.11.103:443  10.162.11.113:443
TCP 110:41 FIN_WAIT    10.160.8.47:51642  10.162.11.103:443  10.162.11.113:443

My seesaw is running on ubuntu18, uname 4.19.57. I use DR, but i tried other schedulers like NAT/TUN. All of them have this issue.

I tried these settings on seesaw, but not helpful

sysctl -qw net.ipv4.vs.sloppy_tcp=1
sysctl -qw net.ipv4.vs.expire_nodest_conn=0
sysctl -qw net.ipv4.vs.expire_quiescent_template=0
sysctl -qw net.ipv4.vs.pmtu_disc=0
ipvsadm --set 7200 7000 300
echo "0" >/proc/sys/net/ipv4/ip_forward

other info on the seesaw node

root@seesaw-01:/usr/local/seesaw# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 72:d4:e7:1f:cc:58 brd ff:ff:ff:ff:ff:ff
    inet 10.162.11.90/23 brd 10.162.11.255 scope global dynamic ens18
       valid_lft 477900sec preferred_lft 477900sec
    inet6 fe80::70d4:e7ff:fe1f:cc58/64 scope link
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:00:xx:00:??:80 brd ff:ff:ff:ff:ff:ff
    inet 10.162.11.101/23 brd 10.162.11.255 scope global ens19
       valid_lft forever preferred_lft forever
    inet 10.162.11.103/23 brd 10.162.11.255 scope global secondary ens19
       valid_lft forever preferred_lft forever
    inet6 fe80::200:5eff:fe00:180/64 scope link
       valid_lft forever preferred_lft forever
4: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 5a:9b:78:8b:bc:2a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::589b:78ff:fe8b:bc2a/64 scope link
       valid_lft forever preferred_lft forever
5: ip+net: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether da:2a:8c:e5:d9:96 brd ff:ff:ff:ff:ff:ff
root@seesaw-01:/usr/local/seesaw# lsmod | grep nf_con
nf_conntrack          143360  5 nf_nat,nf_nat_ipv4,xt_nat,xt_CT,ip_vs
nf_defrag_ipv6         20480  2 nf_conntrack,ip_vs
nf_defrag_ipv4         16384  1 nf_conntrack
libcrc32c              16384  3 nf_conntrack,nf_nat,ip_vs
root@seesaw-01:/usr/local/seesaw#

Really appreciate your help 🙏

salamanderrex commented 3 years ago

I found the issue. it is because router caches the ARP. so traffic routes to my RS and seesaw randomly.

# https://docs.huihoo.com/hpc-cluster/linux-virtual-server/HOWTO/LVS-HOWTO.arp_problem.html
# Make sure you don't bring up the ethernet device (say at bootup) before arp_ignore/arp_announce have been setup, or you will get a round of arp broadcasts from the NIC.

do this on seesaw this to update router 

 arping -S  vip router
zhangbo1882 commented 2 years ago

So it seems that this issue can be closed.