search

F-Stack / f-stack

F-Stack is an user space network development kit with high performance based on DPDK, FreeBSD TCP/IP stack and coroutine API.
http://www.f-stack.org
Other
3.88k stars 901 forks source link

fstack nginx: Connection timed out #290

Open dgbo opened 6 years ago

dgbo commented 6 years ago

Hi,

I try to make f_stack work with XL710 40Gb/s NIC. The dpdk, ff_stack, nginx are complied based on README.md.

The dpdk-devbind.py --status output: 0000:03:00.0 'Ethernet Controller XL710 for 40GbE QSFP+ 1584' drv=igb_uio unused=vfio-pci,uio_pci_generic

nginx installed: [root@master dpdk]# ls /usr/local/nginx_fstack/ client_body_temp conf fastcgi_temp html logs proxy_temp sbin scgi_temp uwsgi_temp

But the nginx fails to start: [root@master f-stack]# /usr/local/nginx_fstack/sbin/nginx [root@master f-stack]# cat /usr/local/nginx_fstack/logs/error.log EAL: Detected 96 lcore(s) 2018/10/18 00:43:26 [error] 39396#0: primary worker process failed to initialize (110: Connection timed out)

The XL710 NIC is directly connect (without a gateway) to another XL710 port on another server. The ethtool info: [root@master dpdk]# ifconfig eth12 eth12: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.100.26 netmask 255.255.255.0 broadcast 192.168.100.255 inet6 fe80::3efd:feff:fea1:ea10 prefixlen 64 scopeid 0x20 ether 3c:fd:fe:a1:ea:10 txqueuelen 1000 (Ethernet) RX packets 3 bytes 218 (218.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 29 bytes 2110 (2.0 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

The port0 configuration has been modified in f-stack.conf: [root@master nginx_fstack]# cat conf/f-stack.conf [dpdk]

lcore_mask=1 channel=4 promiscuous=1 numa_on=1

tso=0

vlan_strip=1

port_list=0

[port0] addr=192.168.100.26 netmask=255.255.255.0 broadcast=192.168.100.255 gateway=

pcap=./a.pcap

[kni]

enable=1

method=reject

tcp_port=80,443

udp_port=53

[freebsd.boot] hz=100

fd_reserve=1024

kern.ipc.maxsockets=262144

net.inet.tcp.syncache.hashsize=4096 net.inet.tcp.syncache.bucketlimit=100

net.inet.tcp.tcbhashsize=65536

kern.ncallout=262144

[freebsd.sysctl] kern.ipc.somaxconn=32768 kern.ipc.maxsockbuf=16777216

net.link.ether.inet.maxhold=5

net.inet.tcp.fast_finwait2_recycle=1 net.inet.tcp.sendspace=16384 net.inet.tcp.recvspace=8192 net.inet.tcp.nolocaltimewait=1 net.inet.tcp.cc.algorithm=cubic net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendbuf_auto=1 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_inc=524288 net.inet.tcp.sack.enable=1 net.inet.tcp.blackhole=1 net.inet.tcp.msl=2000 net.inet.tcp.delayed_ack=0

net.inet.udp.blackhole=1 net.inet.ip.redirect=0

whl739 commented 6 years ago

It seems that primary-process(worker process 0) hasn't been rightly started after 15s. The code is in src/os/unix/ngx_process_cycle.c:ngx_start_worker_processes.