Open nomyself opened 3 years ago
ywc689
commented
3 years ago defence_tcp_drop 打开的时候， 目标 IP 是 vip 但端口不是 vport 的包会直接丢弃掉。defence_tcp_drop 关闭会把这种包转发到 KNI。如果在非安全环境，建议打开这个配置。
从你给出的抓包截图上看，KNI 收到的包都是来自 client 的 RST 包，这有点像异常攻击。我用你上面给出的复现方法没有复现这个问题，KNI 接口流量只有 10kpps 左右。
nomyself
commented
3 years ago
defence_tcp_drop打开的时候， 目标 IP 是 vip 但端口不是 vport 的包会直接丢弃掉。defence_tcp_drop关闭会把这种包转发到 KNI。如果在非安全环境，建议打开这个配置。从你给出的抓包截图上看，KNI 收到的包都是来自 client 的 RST 包，这有点像异常攻击。我用你上面给出的复现方法没有复现这个问题，KNI 接口流量只有 10kpps 左右。
defence_tcp_drop 从1.7开始默认关闭这个配置不合适，建议改为打开
10kpps是正常的，调整-i参数更改发包频率。
我模拟环境打了400kpss，也只是出现了kni_send2kern_loop的异常，没有出现当时no memory的报错。以下是当时场景的一些错误日志
2021-02-28T19:43:14+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:14+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:14+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:16+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:16+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:16+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:17+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:18+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:18+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:18+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:18+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:18+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:18+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:18+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:18+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T19:43:19+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_synack_rcv: got ack_mbuf NULL pointer: ack-saved = 0
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_filter_ack: no memory
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_conn_new: no memory
2021-02-28T20:06:38+08:00 lvs warning dpvs[132777]: IPVS: dp_vs_synproxy_ack_rcv: ip_vs_schedule failed
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_conn_new: no memory
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_ack_rcv: ip_vs_schedule failed
2021-02-28T20:06:38+08:00 lvs warning dpvs[132777]: IPVS: dp_vs_synproxy_ack_rcv: ip_vs_schedule failed
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_filter_ack: no memory
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_conn_new: no memory
2021-02-28T20:06:38+08:00 lvs warning dpvs[132777]: IPVS: dp_vs_synproxy_ack_rcv: ip_vs_schedule failed
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_filter_ack: no memory
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_filter_ack: no memory
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_conn_new: no memory
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_filter_ack: no memory
2021-02-28T20:06:38+08:00 lvs warning dpvs[132777]: IPVS: dp_vs_synproxy_ack_rcv: ip_vs_schedule failed
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_filter_ack: no memory
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_filter_ack: no memory
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_filter_ack: no memory
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_filter_ack: no memory
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_filter_ack: no memory
2021-02-28T20:06:38+08:00 lvs err dpvs[132777]: IPVS: dp_vs_synproxy_filter_ack: no memory这个场景是rs没响应syn包导致SYNPROXY_ACK_MBUFPOOL消耗完了？
nomyself
commented
3 years ago 
defence_tcp_drop关闭，后端服务异常，丢包，重传不响应syn包建连的时候容易把ack_mbufpool打满，导致dpvs异常无法处理。上图是为了测试故意把ack_mbufpool默认值100万改为了2万，用一个客户端wrk打流量
场景描述
fullnat模式syn flood的包，部分包全部透传到kni接口，大概300k pps。导致BGP中断，健康检查出现中断的现象。从代码看到vip:vport的流量不应该走到kni接口才对复现方法
dpvs.conf关闭defence_tcp_drop，测试1.6.1和1.8.4都可以复现。1.7版本开始默认关闭，这个集群使用的是1.8.4版本synproxyserver 端外网口kni抓包
10:36:48.854348 IP (tos 0x0, ttl 62, id 21851, offset 0, flags [DF], proto TCP (6), length 40) c_ip.20000 > vip.80: Flags [R], cksum 0xfdf1 (correct), seq 695027805, win 0, length 0 10:36:48.896353 IP (tos 0x0, ttl 62, id 21852, offset 0, flags [DF], proto TCP (6), length 40) c_ip.20000 > vip.80: Flags [R], cksum 0x2f3c (correct), seq 1195112772, win 0, length 0 10:36:48.954346 IP (tos 0x0, ttl 62, id 21853, offset 0, flags [DF], proto TCP (6), length 40) c_ip.20000 > vip.80: Flags [R], cksum 0xd931 (correct), seq 1623995838, win 0, length 0 10:36:48.996348 IP (tos 0x0, ttl 62, id 21854, offset 0, flags [DF], proto TCP (6), length 40) c_ip.20000 > vip.80: Flags [R], cksum 0x6614 (correct), seq 1559669937, win 0, length 0