search

F-Stack / f-stack

F-Stack is an user space network development kit with high performance based on DPDK, FreeBSD TCP/IP stack and coroutine API.
http://www.f-stack.org
Other
3.76k stars 879 forks source link

my application after high bandwidth doesn't process any incoming connection(no reply to SYN) nor replies pings for couple of minutes #753

Closed rajeshwarp2002 closed 1 year ago

rajeshwarp2002 commented 1 year ago

Hello, I am building forward proxy. When I run speedtest.net, lots of data is sent through my f-stack application. Speedtest download gets over, but sometimes upload never starts and after that my f-stack application doesn't process any new incoming connection for 2-5 minutes.

symptoms

  1. ff_netstat shows no packets being read or written(RecvQ and SendQ stays same for some time), I do have event added to kevent for read and write. Atleast new connection should be accepted, event ping doesnt work.

  2. happens only when I run speed test, for normal internet traffic no issues. The socket Qs are drained fast.

3.I can confirm that thread is not deadlocked or sleeping somewhere. I can see from gdb it continuously calls various methods of dpdk and also calls my kevent.

Any Ideas would be helpful.

[root@localhost src]# ff_netstat Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 142.250.179.130.https ESTABLISHED tcp4 0 0 myip ESTABLISHED tcp4 0 0 172.217.168.227.https ESTABLISHED tcp4 0 0 myip.62674 ESTABLISHED tcp4 0 0 87.121.0.44.webcache ESTABLISHED tcp4 0 28612 myip.62673 ESTABLISHED tcp4 0 0 185.206.145.41.webcach ESTABLISHED tcp4 0 29772 myip.62672 ESTABLISHED tcp4 0 0 213.183.62.34.webcache ESTABLISHED tcp4 0 0 185.97.72.7.webcache ESTABLISHED tcp4 0 0 87.121.0.44.webcache ESTABLISHED tcp4 0 0 185.206.145.41.webcach ESTABLISHED tcp4 0 38868 myip.62671 ESTABLISHED tcp4 0 64340 myip.62668 ESTABLISHED tcp4 0 90804 myip.62670 ESTABLISHED tcp4 0 77616 myip.62669 ESTABLISHED tcp4 773472 0 185.97.72.7.webcache ESTABLISHED tcp4 0 164724 myip.62666 ESTABLISHED tcp4 648160 0 213.183.62.34.webcache ESTABLISHED tcp4 0 442332 myip.62667 ESTABLISHED tcp4 981952 0 213.183.62.34.webcache ESTABLISHED tcp4 1536760 0 185.97.72.7.webcache ESTABLISHED tcp4 0 556276 myip.62664 ESTABLISHED tcp4 0 574396 myip.62665 ESTABLISHED tcp4 0 0 87.121.0.44.webcache ESTABLISHED tcp4 0 52652 myip.62663 ESTABLISHED tcp4 0 0 185.206.145.41.webcach ESTABLISHED tcp4 0 43820 myip.62662 ESTABLISHED tcp4 1630224 0 185.97.72.7.webcache ESTABLISHED tcp4 0 606972 myip.62661 ESTABLISHED tcp4 1030568 0 213.183.62.34.webcache ESTABLISHED tcp4 0 721796 myip.62660 ESTABLISHED tcp4 0 0 87.121.0.44.webcache ESTABLISHED tcp4 0 84316 myip.62657 ESTABLISHED tcp4 0 0 87.121.0.44.webcache ESTABLISHED tcp4 0 0 185.206.145.41.webcach ESTABLISHED tcp4 0 0 185.206.145.41.webcach ESTABLISHED tcp4 0 0 185.97.72.7.webcache ESTABLISHED tcp4 0 75292 myip.62656 ESTABLISHED tcp4 0 86948 myip.62655 ESTABLISHED tcp4 0 70520 myip.62654 ESTABLISHED tcp4 0 0 213.183.62.34.webcache ESTABLISHED tcp4 0 67968 myip.62659 ESTABLISHED tcp4 0 114108 myip.62658 ESTABLISHED tcp4 13123456 213.183.62.34.webcache ESTABLISHED tcp4 0 0 87.121.0.44.webcache ESTABLISHED tcp4 2829000 0 185.97.72.7.webcache ESTABLISHED tcp4 0 26379882 myip.62652 ESTABLISHED tcp4 0 85860 myip.62651 ESTABLISHED tcp4 0 26381482 myip.62653 ESTABLISHED tcp4 0 0 185.206.145.41.webcach ESTABLISHED tcp4 0 88732 myip.62650 ESTABLISHED tcp4 0 0 185.206.145.41.webcach ESTABLISHED

[dpdk] lcore_mask=0x01 channel=4 promiscuous=1 numa_on=1 tx_csum_offoad_skip=0 tso=0 vlan_strip=1 idle_sleep=0 pkt_tx_delay=0 symmetric_rss=1 port_list=0 nb_vdev=0 nb_bond=0 fd_reserve=10240 kern.ipc.maxsockets=262144 net.inet.tcp.syncache.hashsize=4096 net.inet.tcp.syncache.bucketlimit=100 net.inet.tcp.tcbhashsize=65536 kern.ncallout=262144 kern.features.inet6=1 [freebsd.sysctl] kern.ipc.somaxconn=32768 kern.ipc.maxsockbuf=16777216 net.link.ether.inet.maxhold=5 net.inet.tcp.fast_finwait2_recycle=1 net.inet.tcp.sendspace=16384 net.inet.tcp.recvspace=8192 net.inet.tcp.cc.algorithm=cubic net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendbuf_auto=1 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.sack.enable=1 net.inet.tcp.blackhole=1 net.inet.tcp.msl=2000 net.inet.tcp.delayed_ack=1 net.inet.tcp.rfc1323=1 net.inet.udp.blackhole=1 net.inet.ip.redirect=0 net.inet.ip.forwarding=0 net.inet6.ip6.auto_linklocal=1 net.inet6.ip6.accept_rtadv=2 net.inet6.icmp6.rediraccept=1 net.inet6.ip6.forwarding=0 net.inet.tcp.functions_default=freebsd net.inet.tcp.hpts.skip_swi=1 net.inet.tcp.hpts.minsleep=250 net.inet.tcp.hpts.maxsleep=51200

rajeshwarp2002 commented 1 year ago

I found the issue, I ran out of mbufs.