search

F-Stack / f-stack

F-Stack is an user space network development kit with high performance based on DPDK, FreeBSD TCP/IP stack and coroutine API.
http://www.f-stack.org
Other
3.85k stars 896 forks source link

Disable TX ip checksum offload in VMware ESXi 5.5.0 Update 2 #520

Open vincentmli opened 4 years ago

vincentmli commented 4 years ago

Just file an issue to track this, while running F-Stack Nginx in VM in VMware ESXi 5.5.0 Update 2, the TX ip checksum offload is enabled by default, but the the ip checksum is not calculated in the hardware, result in ip checksum value 0

# tcpdump -nn  -v -r /tmp/cpu0_0.pcap  host 10.1.72.28 and tcp | head -4

SYN from client 10.1.72.28  to  F-Stack Nginx server 10.1.72.168

23:06:37.104023 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.72.28.1025 > 10.1.72.168.80: Flags [S], cksum 0xd9ce (correct), seq 1012484, win 14600, options [mss 1460,sackOK,TS val 2692380804 ecr 0,nop,wscale 7], length 0

SYN+ACK from F-Stack Nginx server 10.1.72.168 to client, the ip checksum is 0, and dropped by client.

23:06:37.104023 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->95f6)!)
    10.1.72.168.80 > 10.1.72.28.1025: Flags [S.], cksum 0xa4f4 (incorrect -> 0x656f), seq 2977315146, ack 1012485, win 8192, options [mss 1460,nop,wscale 9,sackOK,TS val 237605994 ecr 2692380804], length 0

After digging through the code and found tx_csum_offoad_skip config in [dpdk], and https://github.com/F-Stack/f-stack/issues/317 


            if (ff_global_cfg.dpdk.tx_csum_offoad_skip == 0) {
                if ((dev_info.tx_offload_capa & DEV_TX_OFFLOAD_IPV4_CKSUM)) {
                    rte_log(RTE_LOG_INFO, RTE_LOGTYPE_USER1, "TX ip checksum offload supported\n");
                    port_conf.txmode.offloads |= DEV_TX_OFFLOAD_IPV4_CKSUM;
                    pconf->hw_features.tx_csum_ip = 1;
                }

                if ((dev_info.tx_offload_capa & DEV_TX_OFFLOAD_UDP_CKSUM) &&
                    (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_TCP_CKSUM)) {
                    rte_log(RTE_LOG_INFO, RTE_LOGTYPE_USER1, "TX TCP&UDP checksum offload supported\n");
                    port_conf.txmode.offloads |= DEV_TX_OFFLOAD_UDP_CKSUM | DEV_TX_OFFLOAD_TCP_CKSUM;
                    pconf->hw_features.tx_csum_l4 = 1;
                }

disable ip checksum offload resolves the tcp connection issue, this also affect icmp ping since it is ip packet.

tx_csum_offoad_skip=1
vincentmli commented 4 years ago

I wonder if we can introduce another configuration option to disable TX ip checksum offload only, and keep tcp/udp checksum offload enabled, I did a test with that and it works in VMware ESXi 5.5.0 Update 2 environment.

the SYN+ACK from F-Stack Nginx, no more "bad cksum 0" in the ip header, but keep tcp checksum offload enabled

# tcpdump -n -v -r /tmp/cpu0_0.pcap host 10.1.72.28

15:53:16.597155 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.72.168.80 > 10.1.72.28.1025: Flags [S.], cksum 0xa4f4 (incorrect -> 0x2b07), seq 1851342229, ack 1012485, win 8192, options [mss 1460,nop,wscale 9,sackOK,TS val 824592216 ecr 2752780346], length 0
jfb8856606 commented 4 years ago

Yes, you can create a pull requeset to F-Stack.