search

alibaba / LVS

A distribution of Linux Virtual Server with some advanced features. It introduces a new packet forwarding method - FULLNAT other than NAT/Tunneling/DirectRouting, and defense mechanism against synflooding attack - SYNPROXY.
2k stars 682 forks source link

v3 beta fail to pass server response skb to client #6

Open vincentmli opened 10 years ago

vincentmli commented 10 years ago

I haven't got v3 beta release working so far, the syncookie issue disappeared, but it always seems that LVS failed to pass the response from server to client, the debugging log as below with my added syncookie debug noise:

Jan 10 20:43:46 alilvs kernel: [ 6379.609084] device eth0 entered promiscuous mode Jan 10 20:43:56 alilvs kernel: [ 6389.510018] device eth1 entered promiscuous mode Jan 10 20:44:02 alilvs kernel: [ 6395.506678] syn_cookie check cookie=81872209 Jan 10 20:44:02 alilvs kernel: [ 6395.506681] syn_cookie check vs syn seq=2095846056 Jan 10 20:44:02 alilvs kernel: [ 6395.506684] syn_cookie check count=71684 Jan 10 20:44:02 alilvs kernel: [ 6395.506686] syn_cookie check diff=0 Jan 10 20:44:02 alilvs kernel: [ 6395.506687] syn_cookie check excuted second hash Jan 10 20:44:02 alilvs kernel: [ 6395.506690] syn_cookie check res=2125824 Jan 10 20:44:02 alilvs kernel: [ 6395.506692] syn_cookie check mssind=7 Jan 10 20:44:02 alilvs kernel: [ 6395.506694] syn_cookie check NUM_MSS=10 Jan 10 20:44:02 alilvs kernel: [ 6395.506697] IPVS: ip_vs_rr_schedule(): Scheduling... Jan 10 20:44:02 alilvs kernel: [ 6395.506701] IPVS: RR: server 192.168.3.3:80 activeconns 0 refcnt 1 weight 1 Jan 10 20:44:02 alilvs kernel: [ 6395.506712] IPVS: Bind-dest TCP c:192.168.1.7:43826 v:192.168.1.169:80 d:192.168.3.3:80 fwd:F s:0 conn->flags:185 conn->refcnt:1 dest->refcnt:2 Jan 10 20:44:02 alilvs kernel: [ 6395.506719] IPVS: Schedule fwd:F c:192.168.1.7:43826 v:192.168.1.169:80 d:192.168.3.3:80 conn->flags:81C5 conn->refcnt:2 Jan 10 20:44:02 alilvs kernel: [ 6395.506726] syn_proxy_send_rs_syn netdevice:eth0 Jan 10 20:44:02 alilvs kernel: [ 6395.506729] IPVS: save_xmit_info, skb->dev is NULL. Jan 10 20:44:02 alilvs kernel: [ 6395.507263] in syn_proxy_synack_rcv, seq = 1230729923 ack_seq = 1294869472 SA- cp->is_synproxy = 32768 cp->state = 2 Jan 10 20:44:02 alilvs kernel: [ 6395.507267] synproxy_save_fast_xmit netdevice:eth1 Jan 10 20:44:02 alilvs kernel: [ 6395.507270] IPVS: save_xmit_info, netdevice:eth0 Jan 10 20:44:02 alilvs kernel: [ 6395.507273] tcp_dnat_handler: tcph->ack_seq 1312875283 => 1230729924, delta = 82145359 Jan 10 20:44:02 alilvs kernel: [ 6395.507276] IPVS: ip_vs_fast_xmit: send skb to RS! Jan 10 20:44:02 alilvs kernel: [ 6395.507284] IPVS: save_xmit_info, netdevice:eth0 Jan 10 20:44:02 alilvs kernel: [ 6395.507287] tcp_dnat_handler: tcph->ack_seq 1312875283 => 1230729924, delta = 82145359 Jan 10 20:44:02 alilvs kernel: [ 6395.507289] IPVS: ip_vs_fast_xmit: send skb to RS! Jan 10 20:44:02 alilvs kernel: [ 6395.507718] in syn_proxy_synack_rcv, seq = 1230729924 ack_seq = 1294869637 -A- cp->is_synproxy = 32768 cp->state = 1 Jan 10 20:44:02 alilvs kernel: [ 6395.507722] IPVS: eth1(): netdevice:ip_vs_save_xmit_inside_info Jan 10 20:44:02 alilvs kernel: [ 6395.507726] tcp_snat_handler: tcph->seq 1230729924 => 1312875283, delta = 82145359 Jan 10 20:44:02 alilvs kernel: [ 6395.507729] IPVS: ip_vs_fast_response_xmit: send skb to client! Jan 10 20:44:02 alilvs kernel: [ 6395.508013] IPVS: Unbind-dest TCP c:192.168.1.7:43826 v:192.168.1.169:80 d:192.168.3.3:80 fwd:F s:1 conn->flags:8005 conn->refcnt:1 dest->refcnt:2 Jan 10 20:44:02 alilvs kernel: [ 6395.508021] IPVS: Unbind-laddr TCP c:192.168.1.7:43826 v:192.168.1.169:80 l:192.168.3.168:5015 d:192.168.3.3:80 fwd:F s:1 conn->flags:8005 conn->refcnt:1 local->refcnt:2

I initially thought this might be caused by the ip_vs_fast_response_xmit in v3 release, but even disabling fast_response_xmit has no effect

vincentmli commented 10 years ago

the tcpdump capture on server side ( inside 2 outside) shows weird icmp error, V2 release does not have this icmp error. it looks to me LVS unable to pass the HTTP/1.1 200 OK to client, failed to ack the 200 ok from server, server keeps retransmitting the 200 ok and each time triggers the ICMP Destination unreachable error

6 14:32:48.483960 0.000745s AsustekC_51:2b:cd 3com_ca:cb:17 192.168.3.2 192.168.3.168 80 5006 HTTP 1293989791 45442026 HTTP/1.1 200 OK (text/html)

7 14:32:48.484006 0.000046s 3com_ca:cb:17 AsustekC_51:2b:cd 192.168.3.168 192.168.3.2 80 5006 ICMP 1293989791 45442026 Destination unreachable (Host administratively prohibited)

I looked at the LVS centos default iptables rules /etc/sysconfig/iptables

:INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited <----------------------------- -A FORWARD -j REJECT --reject-with icmp-host-prohibited<---------------------------- COMMIT

comment out above 2 rules, tcpdump not capturing any ICMP error any more, but still LVS unable to pass the 200 ok to client and fail to ack the 200 ok from server

jlijian3 commented 9 years ago

depend on 10GE NIC with flow director features, I push a request to fix it with rps