search

iqiyi / dpvs

DPVS is a high performance Layer-4 load balancer based on DPDK.
Other
3k stars 723 forks source link

DPVS didn't process packet send to it and connection stay in SYN_RECV status. #296

Open haosdent opened 5 years ago

haosdent commented 5 years ago

I use ECMP + FULLNAT to set up my DPVS. My DPVS config is

# ipvsadm -Ln
IP Virtual Server version 0.0.0 (size=0)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.65.222.55:80 rr
  -> 10.65.233.14:7070            FullNat 1      0          0

and the local IP is 10.65.222.130

# ipvsadm -G
VIP:VPORT            TOTAL    SNAT_IP              CONFLICTS  CONNS
10.65.222.55:80      1
                              10.65.222.130        0          0

The route table in my DPVS is

# dpip route show
inet 10.65.222.55/32 via 0.0.0.0 src 0.0.0.0 dev dpdk1.1148 mtu 1500 tos 0 scope host metric 0 proto auto
inet 10.65.222.129/32 via 0.0.0.0 src 0.0.0.0 dev dpdk1.1148 mtu 1500 tos 0 scope host metric 0 proto auto
inet 10.65.222.130/32 via 0.0.0.0 src 0.0.0.0 dev dpdk1.1148 mtu 1500 tos 0 scope host metric 0 proto auto
inet 10.65.222.130/27 via 0.0.0.0 src 0.0.0.0 dev dpdk1.1148 mtu 1500 tos 0 scope link metric 0 proto auto
inet 10.0.0.0/8 via 10.65.222.132 src 0.0.0.0 dev dpdk1.1148 mtu 1500 tos 0 scope global metric 0 proto auto

and addr info is

# dpip addr show
inet 10.65.222.130/32 scope global dpdk1.1148
     valid_lft forever preferred_lft forever sa_used 0 sa_free 1032176 sa_miss 0
inet 10.65.222.55/32 scope global dpdk1.1148
     valid_lft forever preferred_lft forever
inet 10.65.222.129/27 scope global dpdk1.1148
     valid_lft forever preferred_lft forever

When I curl 10.65.222.55 from other servers, sometimes it works, sometimes it doesn't work. Most time it doesn't work.

When it doesn't work, I could see the connection state is SYN_RECV

[4]tcp 30s    SYN_RECV    10.65.233.21:57526 10.65.222.55:80    10.65.222.130:1051 10.65.233.14:7070

in the DPVS.

I also try to dump the packet in the gateway. It looks like DPVS didn't send SYN+ACK after it receives SYN+ACK from the real server.

# 10.65.233.21 is the client ip, 10.65.222.55 is the DPVS VIP. The client send a SYN packet to the DPVS.

23:20:22.298942 Out f8:f2:1e:18:5a:c0 ethertype IPv4 (0x0800), length 68: 10.65.233.21.57526 > 10.65.222.55.80: Flags [S], seq 1813846078, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

# 10.65.222.130 is the DPVS local IP. DPVS send a SYN packet to the real server.
23:20:22.299042  In 24:6e:96:c0:46:fc ethertype IPv4 (0x0800), length 76: 10.65.222.130.1051 > 10.65.233.14.7070: Flags [S], seq 1896887987, win 29200, options [exp-e0b6,mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
23:20:22.299072 Out f8:f2:1e:18:5a:c0 ethertype IPv4 (0x0800), length 76: 10.65.222.130.1051 > 10.65.233.14.7070: Flags [S], seq 1896887987, win 29200, options [exp-e0b6,mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

# The real server reply SYN+ACK to the DPVS, however, the DPVS didn't reply.
23:20:22.299211  In b4:96:91:15:c5:64 ethertype IPv4 (0x0800), length 68: 10.65.233.14.7070 > 10.65.222.130.1051: Flags [S.], seq 1358437063, ack 1896887988, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
23:20:22.299241 Out f8:f2:1e:18:61:a0 ethertype IPv4 (0x0800), length 68: 10.65.233.14.7070 > 10.65.222.130.1051: Flags [S.], seq 1358437063, ack 1896887988, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

# The DPVS didn't reply, so cause retransmission, the real server reply SYN+ACK again.
23:20:23.299065  In b4:96:91:15:c5:64 ethertype IPv4 (0x0800), length 68: 10.65.233.14.7070 > 10.65.222.130.1051: Flags [S.], seq 1358437063, ack 1896887988, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
23:20:23.299089 Out f8:f2:1e:18:61:a0 ethertype IPv4 (0x0800), length 68: 10.65.233.14.7070 > 10.65.222.130.1051: Flags [S.], seq 1358437063, ack 1896887988, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

From the gateway, I could ping the DPVS VIP, DPVS local IP and any other DPVS IPs without packet loss.

@beacer @mscbg Any reason would cause this? I could modify the code to provide logs to you if need. Highly appreicated your help in advance since I have stucked at here more than 2 months. Orz

ywc689 commented 5 years ago

Does it work when with one worker only? If so, you can turn on the compile debug macro DPVS_CFG_PARSER_DEBUG defined in src/config.mk, and change the dpvs log level to DEBUG in /etc/dpvs.conf. Rebuild and run dpvs to see if there exists session miss.

haosdent commented 5 years ago

@ywc689 , thanks a lot!!! OrzOrzOrz

Use 1 worker indeed don't have such issues and indeed session miss. Do you know any reason would cause this?

haosdent commented 5 years ago

@ywc689 is it because of VLAN + FULLNAT? I just check our nic, it is X710(i40e) which support fdir

image

ywc689 commented 5 years ago

The dpdk 17.11.2 driver of i40e does not support Flow Director. We are considering to upgrade dpdk version.

senlinzhan commented 5 years ago

@ywc689 When will the version of dpdk be updated?