dpdk+netwrap+nginx can be ok?

[dongbin1024]

i want to run the latest nginx with netwrap which is implemented with odp-dpdk, i tried 1、download dpdk stable 17.11; 2、download odp-dpdk-1.19.0.2_DPDK_17.11； 3、download ofp-3.0.0; before compile with scripts/devbuild_ofp_odp_dpdk.sh, i modify the ofp ./configure，below: ./configure --with-odp=$REPOS/odp-dpdk/install --enable-cunit --prefix=$REPOS/install/ --with-config-flv=netwrap-webserver --enable-sp=no and something is wrong: 1、libofp.so.0.0.0 not found，i try to change to libofp.so.3.0.0 in ofp_netwrap.sh； 2、after that:

odp_crypto.c:556:odp_crypto_capability():No crypto devices available
E -1223208614 0:1643843968 ofp_ipsec.c:180] odp_ipsec_capability failed
E -1223203080 0:1643843968 ofp_ipsec.c:181] Setting maximum number of IPsec SAs to zero
I -1223178086 0:1643843968 ofp_init.c:434] Slow path threads on core 0
odp_packet_dpdk.c:383:setup_pkt_dpdk():Interface name should only contain numbers!: eth0
../linux-generic/odp_packet_io.c:233:setup_pktio_entry():Unable to init any I/O type.
../linux-generic/odp_packet_io.c:295:odp_pktio_open():interface: eth0, driver: bad handle
E -1223036270 0:1643843968 ofp_ifnet.c:23] odp_pktio_open failed
Error: OFP global init failed.
scripts/ofp_netwrap.sh: line 8:   348 Segmentation fault      LD_PRELOAD=libofp_netwrap_crt.so.0.0.0:libofp.so.3.0.0:libofp_netwrap_proc.so.0.0.0 $@

is something i did wrong?thx

[MatiasElo]

When using ODP-DPDK you have to first bind the NIC ports to DPDK. You can follow instructions in https://github.com/OpenDataPlane/odp-dpdk/blob/master/platform/linux-dpdk/README : 4. Prepare DPDK for running ODP-DPDK examples

After this has been done you have to change the interface name to port index (e.g. eth1 -> 0) in ofp_netwrap.sh script.

[dongbin1024]

ok, i have bind the nic port to dpdk, and also find the issue to modify eth1 -> 0 in ofp_netwrap.sh and it's running. ./scripts/ofp_netwrap.sh /usr/local/nginx-1.9.8/sbin/nginx howerver, it also run the destructor function ofp_netwrap_main_dtor ,which i think shoudn't run now. here is the output:

odp_packet_dpdk.c:206:_dpdk_print_port_mac():Port 0, MAC address: FA:16:3E:C8:49:EB
../linux-generic/odp_packet_io.c:295:odp_pktio_open():interface: 0, driver: odp-dpdk
I -1584043924 1:224633216 ofp_ifnet.c:215] Device '0' addr  fa:16:3e:c8:49:eb
I -1584030892 1:224633216 ofp_ifnet.c:237] Device '0' MTU=1500
../linux-generic/odp_packet_io.c:522:odp_pktio_start():interface: 0, input queues: 1, output queues: 1
odp_thread.c:165:odp_thread_init_local():There is a thread already running on core 0
I 0 0:67106560 ofp_cli.c:1628] CLI server started on core 0

CLI: debug 0

CLI: loglevel set debug

CLI: ifconfig fp0 192.168.0.235/24

I -1580297874 0:67106560 ofp_rt_mtrie_lookup.c:280] ofp_rt_rule_add inserted new rule vrf 0 prefix 192.168.0.235/32
I -1580259456 0:67106560 ofp_rt_mtrie_lookup.c:280] ofp_rt_rule_add inserted new rule vrf 0 prefix 192.168.0.0/24

I 1419276428 1:224633216 app_main.c:216] End Netwrap processing constructor()

I 129893088 1:224633216 ofp_init.c:539] Cleaning device '0' addr  fa:16:3e:c8:49:eb
../linux-generic/odp_packet_io.c:574:odp_pktio_stop():interface: 0
../linux-generic/odp_packet_io.c:411:odp_pktio_close():interface: 0
W 130137190 1:224633216 ofp_timer.c:476] Timeout already expired or inactive
W 131446130 1:224633216 ofp_timer.c:476] Timeout already expired or inactive
W 270002958 1:224633216 ofp_timer.c:476] Timeout already expired or inactive
W 270155502 1:224633216 ofp_timer.c:476] Timeout already expired or inactive
W 270227354 1:224633216 ofp_timer.c:476] Timeout already expired or inactive

[MatiasElo]

The ofp_netwrap_main_dtor () destructor should be run when the libofp_netwrap_proc library is unloaded. So, perhaps nginx is failing to start properly and hence exiting. Capturing some nginx logs could be helpful.

[dongbin1024]

nginx log level is setted to be debug and no warning or error log; i thought nginx is running with daremon on, and i tried modify the nginx.conf,add two line:

daemon off;
master_process off; (for debug easily)

and it works! The ofp_netwrap_main_dtor () not run again, hahaha

[MatiasElo]

nginx log level is setted to be debug and no warning or error log; i thought nginx is running with daremon on, and i tried modify the nginx.conf,add two line:

daemon off;
master_process off; (for debug easily)

and it works! The ofp_netwrap_main_dtor () not run again, hahaha

Makes sense. In daemon mode the nginx master process exits, which triggers the netwrap destructor.

[bogdanPricope]

And it actually worked?? Last time I checked, netwrap was not working with dpdk... was something about dpdk using epoll inside. Basically, I was using odp + linux-generic Note:

1. 'netwrap' was 'experimental'
2. 'netwrap' example was tested with (some version of) nginx (with select , etc.)

[dongbin1024]

And it actually worked?? Last time I checked, netwrap was not working with dpdk... was something about dpdk using epoll inside. Basically, I was using odp + linux-generic Note:

1. 'netwrap' was 'experimental'
2. 'netwrap' example was tested with (some version of) nginx (with select , etc.)

actually it has not worked yet, 1 I modified the orig nginx to be a multithread tcp server, 2 it can run up with dpdk+netwrap,
3 it can epoll and process tcp connection request,
4 the it add acceptfd to epollfd with readableevent; 5 it can't get readable event with epoll_wait when client send some cmd after connection startup; i'm not sure is epoll of netwrap is ok totally;

[dongbin1024]

does netwrap epoll support both level-triggerded and edge-triggered ?

[bogdanPricope]

Is not about netwrap epoll support but is more about ofp_epoll implementation: only EPOLLIN (OFP_EPOLLIN) event type can be set and only Level Triggered behaviour is supported.