namhyung / uftrace

Function graph tracer for C/C++/Rust/Python
https://uftrace.github.io/slide/
GNU General Public License v2.0
2.93k stars 419 forks source link

segfault on DPDK #878

Open DanielTimLee opened 4 years ago

DanielTimLee commented 4 years ago

It seems uftrace isn't running properly with DPDK. Actually I do not know well about DPDK, so I'm trying to figure out more.

danieltl@NR0:~/git/dpdk/app/test-pmd$ sudo uftrace record ./testpmd -c 0xF -n 4 -- --portmask=0x1 --nb-cores=2
EAL: Detected 8 lcore(s)
...

Configuring Port 0 (socket 0)
Port 0: 00:1E:67:DB:D6:F2
Checking link statuses...
Done
...
Logical Core 1 (socket 0) forwards packets on 1 streams:
...
Press enter to exit

Port 0: link state change event

Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 37143          RX-dropped: 0             RX-total: 37143
  TX-packets: 37812          TX-dropped: 168           TX-total: 37980
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 37143          RX-dropped: 0             RX-total: 37143
  TX-packets: 37812          TX-dropped: 168           TX-total: 37980
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.
WARN: child terminated by signal: 11: Segmentation fault
danieltl@NR0:~/git/dpdk/app/test-pmd$ sudo uftrace record --no-libcall ./testpmd -c 0xF -n 4 -- --portmask=0x1 --nb-cores=2
EAL: Detected 8 lcore(s)
...
Configuring Port 0 (socket 0)
Port 0: 00:1E:67:DB:D6:F2
Checking link statuses...
Done
...
Logical Core 1 (socket 0) forwards packets on 1 streams:
...
Press enter to exit

Port 0: link state change event
Signal 2 received, preparing to exit...
Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 168424         RX-dropped: 0             RX-total: 168424
  TX-packets: 168420         TX-dropped: 0             TX-total: 168420
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 168424         RX-dropped: 0             RX-total: 168424
  TX-packets: 168420         TX-dropped: 0             TX-total: 168420
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Stopping port 0...
Stopping ports...
Done

Shutting down port 0...
Closing ports...
Bye...
WARN: Segmentation fault: address not mapped (addr: 0x2)
WARN: Backtrace from uftrace:
WARN: =====================================
WARN: [3] (pmd_test_exit[55fa2e29ac43] <= signal_handler[55fa2e29afba])
WARN: [2] (signal_handler[55fa2e29af7f] <= <7ffb03169f60>[7ffb03169f60])
WARN: [1] (mbuf_pool_create[55fa2e0d50ee] <= main.cold.19[55fa2e0d5856])
WARN: [0] (main[55fa2e1f5cf7] <= __libc_start_main[7ffb0314cb6b])  
Steps to reproduce #### Steps to reproduce ```sh # clone DPDK git clone git://dpdk.org/dpdk cd dpdk # set env variable TARGET=x86_64-native-linuxapp-gcc export RTE_SDK=`pwd` export RTE_TARGET=$TARGET # make and install targets # make install -j 8 T=$RTE_TARGET cd $RTE_SDK/$RTE_TARGET make -j 8 cd $RTE_SDK/app/test-pmd # open makefile and add -pg option to Makefile make -j 8 # bind NIC to dpdk sudo modprobe uio sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio.ko sudo $RTE_SDK/usertools/dpdk-devbind.py --status # danieltl@NR0:~/git/dpdk/usertools$ ./dpdk-devbind.py --status # Network devices using kernel driver # =================================== # 0000:01:00.0 'Device 4000' if=enp1s0np1,enp1s0np0 drv=nfp unused= *Active* # 0000:04:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb unused= *Active* # 0000:05:00.0 'I210 Gigabit Network Connection 1533' if=eno2 drv=igb unused= *Active* # 0000:06:00.0 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller 8168' if=enp6s0 drv=r8169 unused= *Active* sudo ip link set down dev eno1 sudo $RTE_SDK/usertools/dpdk-devbind.py -b igb_uio 0000:04:00.0 sudo $RTE_SDK/usertools/dpdk-devbind.py --status # danieltl@NR0:~/git/dpdk/usertools$ ./dpdk-devbind.py --status # Network devices using DPDK-compatible driver # ============================================ # 0000:04:00.0 'I210 Gigabit Network Connection 1533' drv=igb_uio unused=igb # Network devices using kernel driver # =================================== # 0000:01:00.0 'Device 4000' if=enp1s0np1,enp1s0np0 drv=nfp unused=igb_uio *Active* # 0000:05:00.0 'I210 Gigabit Network Connection 1533' if=eno2 drv=igb unused=igb_uio *Active* # 0000:06:00.0 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller 8168' if=enp6s0 drv=r8169 unused=igb_uio *Active* # Set hugepage 2M * 1024 = 2G sudo mkdir -p /mnt/huge sudo bash -c 'echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages' sudo mount -t hugetlbfs nodev /mnt/huge # Trace test-pmd cd $RTE_SDK/app/test-pmd sudo uftrace record ./testpmd -c 0xF -n 4 -- --portmask=0x1 --nb-cores=2 sudo uftrace record --no-libcall ./testpmd -c 0xF -n 4 -- --portmask=0x1 --nb-cores=2 # Crash on both case ```
honggyukim commented 4 years ago

Hmm.. I don't know what DPDK is and it looks it doesn't have clear reproducible steps for the crash.