search

sdnfv / openNetVM

A high performance container-based NFV platform from GW and UCR.
http://sdnfv.github.io/onvm/
Other
263 stars 136 forks source link

onvm/go.sh: line 80: 227467 Segmentation fault #150

Closed chenxiang2019 closed 5 years ago

chenxiang2019 commented 5 years ago

onvm/go.sh: line 80: 227467 Segmentation fault

Hi all, I'm trying to use openNetVM to establish a NF chain: 

aes_encrypt => basic_monitor => firewall => simple_forward

I modified the source codes of basic monitor to support the parameter -d. The source codes are available here.

Everything seems to be ok so I started MoonGen on another machine to inject 40 Gbps traffic to openNetVM. However, the manager goes down as soon as I started MoonGen.

Current Behavior The output of the NF manager:

NF TAG         IID / SID / CORE    rx_pps  /  tx_pps        rx_drop  /  tx_drop           out   /    tonf     /   drop
----------------------------------------------------------------------------------------------------------------------
aes_encrypt     1  /  1  /  3            0 / 0                     0 / 0                      0 / 0           / 0           
basic_monitor   2  /  2  /  4            0 / 0                     0 / 0                      0 / 0           / 0           
firewall        3  /  3  /  5            0 / 0                     0 / 0                      0 / 0           / 0           
simple_forward  4  /  4  /  6            0 / 0                     0 / 0                      0 / 0           / 0           
onvm/go.sh: line 80: 227467 Segmentation fault      (core dumped) sudo $SCRIPTPATH/onvm_mgr/$RTE_TARGET/onvm_mgr -l $cpu -n 4 --proc-type=primary ${virt_addr} -- -p ${ports} -n ${nf_cores} ${num_srvc} ${def_srvc} ${stats} ${stats_sleep_time} ${verbosity_level} ${ttl} ${packet_limit} ${shared_cpu_flag}

Expected behavior/code I expect that everything goes well.

Steps to reproduce I opened five terminals (T1-T5) for openNetVM:

In T1:

onvm/go.sh 0,1,2,3,4 1 0x3F8 -s stdout

In T2:

./examples/start_nf.sh aes_encrypt 1 -d 2

In T3:

./examples/start_nf.sh basic_monitor 2 -d 3

In T4:

./examples/start_nf.sh firewall 3 -d 4 -f examples/firewall/rules.json

In T5:

./examples/start_nf.sh simple_forward 4 -d 4

Environment

Possible Solution

Additional context/Screenshots

Any possible solutions to fix this? Thank you!

koolzz commented 5 years ago

Looks like you're trying to send msgs from the basic monitor to destination but with the action NF OUT which will try to send them out of the NIC port. This would cause a segfault as you don't have a port with ID 3. Simply switch the action to the same as in the simple forward NF(TONF)

chenxiang2019 commented 5 years ago

@koolzz It works. Thanks for your help!

chenxiang2019 commented 5 years ago

@koolzz Hello, sorry for bothering you again. I have another question: How can I make the simple_forwarder forward the traffic to the egress NIC port? Specifically, I have changed the action of simple_forwarder in "packet_handler" function to ONVM_NF_ACTION_OUT. I have checked that other NFs are normally forwarded packets to the next one. But it seems to not work well since the ONVM manager crashes again. The error information is the same as I mentioned above. Any help would be appreciated.

koolzz commented 5 years ago

Make sure the port you're sending to is enabled in the onvm mgr ( from your example only port 1 is enabled)

chenxiang2019 commented 5 years ago

@koolzz Thanks for your reply. I have read go.sh under the folder onvm/. I noticed that the second parameter of go.sh is to assign port mask. Therefore, I changed my command to:

onvm/go.sh 0,1,2,3,4 3 0x3F8 -s stdout

I try to use one port for RX and one port for TX. However, the onvm manager is still crashed when I injected traffic into openNetVM.

koolzz commented 5 years ago

@Wasdns Which port are you trying to send out from(meta destination argument)? Can you show me what onvm_mgr prints? The port needs to be bound to dpdk

chenxiang2019 commented 5 years ago

@koolzz Sorry for my late reply. The output of onvm_mgr:

Command: onvm/go.sh 0,1,2,3 3 0xFF -s stdout

EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:1a:00.0 on NUMA socket 0
EAL:   probe driver: 8086:37d0 net_i40e
EAL: PCI device 0000:1a:00.1 on NUMA socket 0
EAL:   probe driver: 8086:37d0 net_i40e
EAL: PCI device 0000:1a:00.2 on NUMA socket 0
EAL:   probe driver: 8086:37d1 net_i40e
EAL: PCI device 0000:1a:00.3 on NUMA socket 0
EAL:   probe driver: 8086:37d1 net_i40e
EAL: PCI device 0000:3b:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1583 net_i40e
EAL: PCI device 0000:3b:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1583 net_i40e
Registered 8 cores for NFs: 0, 1, 2, 3, 4, 5, 6, 7
Creating mbuf pool 'MProc_pktmbuf_pool' [32767 mbufs] ...
Creating mbuf pool 'NF_INFO_MEMPOOL' ...
Creating mbuf pool 'NF_MSG_MEMPOOL' ...
Port 0 init ... 
Port 0 socket id 0 ... 
Port 0 Rx rings 1 ... 
Port 0 Tx rings 2 ... 
Port 0 modified RSS hash function based on hardware support,requested:0x3efbc configured:0x6eb8
done: 
Port 1 init ... 
Port 1 socket id 0 ... 
Port 1 Rx rings 1 ... 
Port 1 Tx rings 2 ... 
Port 1 modified RSS hash function based on hardware support,requested:0x3efbc configured:0x6eb8
done: 

Checking link statusdone
Port 0 Link Up - speed 40000 Mbps - full-duplex
Port 1 Link Up - speed 40000 Mbps - full-duplex
Default service chain: send to sdn NF
cur_index:1, action:2, destination:1

APP: Finished Process Init.
APP: 4 cores available in total
APP: 1 cores available for handling manager RX queues
APP: 2 cores available for handling TX queues
APP: 1 cores available for handling stats
APP: Core 1: Running TX thread for NFs 1 to 64
APP: Core 2: Running TX thread for NFs 65 to 127
APP: Core 0: Running master thread
APP: Stats verbosity level = 1
APP: Core 3: Running RX thread for RX queue 0

PORTS
-----
Port 0: '9c:69:b4:60:35:24' Port 1: '9c:69:b4:60:35:25' 

Port 0 - rx:         0  (        0 pps) tx:         0  (        0 pps)
Port 1 - rx:         0  (        0 pps) tx:         0  (        0 pps)

NF TAG         IID / SID / CORE    rx_pps  /  tx_pps        rx_drop  /  tx_drop           out   /    tonf     /   drop
----------------------------------------------------------------------------------------------------------------------
aes_encrypt     1  /  1  /  0            0 / 0                     0 / 0                      0 / 0           / 0           
aes_decrypt     2  /  2  /  1            0 / 0                     0 / 0                      0 / 0           / 0           
basic_monitor   3  /  3  /  2            0 / 0                     0 / 0                      0 / 0           / 0           
firewall        4  /  4  /  3            0 / 0                     0 / 0                      0 / 0           / 0           
simple_forward  6  /  5  /  4            0 / 0                     0 / 0                      0 / 0           / 0

I have two ports connected to IGB UIO driver. I have changed the action of the last NF "simple_forward" to ONVM_NF_ACTION_OUT.

koolzz commented 5 years ago

Yes that looks okay to me, what is the meta->destination argument of the packet? The one you set in simple forward. It can either be 0 or 1 based on your port setup

chenxiang2019 commented 5 years ago

@koolzz According the example here, I use the command ./examples/start_nf.sh simple_forward 5 -d 5 to start "simple_forwarder".

koolzz commented 5 years ago

If you're trying to send it out of the nic use either -d 0 or -d 1(0 might not work if we arg check dest I'd, but you can edit that)

koolzz commented 5 years ago

The -d 5 tries to send it out of port 5 which segfault s

chenxiang2019 commented 5 years ago

@koolzz The problem has been resolved based on your instructions. Thanks a lot!