F-Stack / f-stack

F-Stack is an user space network development kit with high performance based on DPDK, FreeBSD TCP/IP stack and coroutine API.
http://www.f-stack.org
Other
3.76k stars 879 forks source link

Failed to launch multiple nginx workers using vmxnet3 driver #792

Closed coolkingh closed 7 months ago

coolkingh commented 7 months ago

I want to run 8 Nginx workers as I have 16 CPU

Below is the NGINX configuration :-

# root account is necessary.
user  root;
# should be equal to the lcore count of `dpdk.lcore_mask` in f-stack.conf.
worker_processes  8;

Below is the f-stack configuration :-

[dpdk]
# Hexadecimal bitmask of cores to run on.
lcore_mask=8

# Number of memory channels.
channel=4

Below is the NIC driver details:-

Network devices using DPDK-compatible driver
============================================
0000:0b:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=vmxnet3

Network devices using kernel driver
===================================
0000:13:00.0 'VMXNET3 Ethernet Controller 07b0' if=ens224 drv=vmxnet3 unused=igb_uio *Active*

Below is the number of worker running in NGINX :-

[root@ f-stack]# ps -ax | grep nginx
19154 pts/0    S+     0:00 grep --color=auto nginx
29068 ?        Rs   640:35 nginx: master process /usr/local/nginx_fstack/sbin/nginx
29069 ?        Rl   757:06 nginx: worker process

Below is the nginx logs:-

[root@f-stack]# cat /var/log/nginx/error.log
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: probing started: [auxiliary]
EAL: probing started: [dpaa_bus]
EAL: probing started: [fslmc]
EAL: probing started: [ifpga]
EAL: probing started: [pci]
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 net_vmxnet3
EAL: Probe PCI driver: net_vmxnet3 (15ad:7b0) device: 0000:0b:00.0 (socket -1)
EAL: PCI device 0000:13:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 net_vmxnet3
EAL: probing started: [vmbus]
EAL: probing started: [dsa]
EAL: probing vbus 0
TELEMETRY: No legacy callbacks, legacy socket not created
2023/12/01 10:29:58 [alert] 26028#0: close() channel failed (9: Bad file descriptor)
2023/12/01 10:29:59 [alert] 26028#0: close() channel failed (9: Bad file descriptor)
2023/12/01 10:29:59 [alert] 26028#0: close() channel failed (9: Bad file descriptor)
2023/12/01 10:29:59 [alert] 26028#0: close() channel failed (9: Bad file descriptor)
2023/12/01 10:29:59 [alert] 26028#0: close() channel failed (9: Bad file descriptor)

It is working if worker_processes is set to 1 without any error.

coolkingh commented 7 months ago

working after changing the configuration to ff..