OpenFastPath / ofp

OpenFastPath project
BSD 3-Clause "New" or "Revised" License
351 stars 126 forks source link

BUG: ofp with dpdk example will run failed when static linked #226

Closed InJoins closed 5 years ago

InJoins commented 5 years ago

compiling script

`#!/bin/sh

prepare

apt -y install libelf-dev apt -y install automake apt -y install autoconf apt -y install libtool apt -y install pkg-config apt -y install libconfig-dev apt -y install libssl-dev apt -y install libnuma-dev

dpdk - clone & make & install

export RTE_SDK=/root/dpdk export RTE_TARGET=x86_64-native-linuxapp-gcc export RTE_DESTDIR=$RTE_SDK/install git clone https://github.com/DPDK/dpdk.git $RTE_SDK --branch=v17.11 --depth 1

sed -ri 's,(CONFIG_RTE_LIBRTE_PMD_OPENSSL=).*,\1y,' $RTE_SDK/$RTE_TARGET/.config

cd $RTE_SDK make config T=$RTE_TARGET make -j8 install T=$RTE_TARGET DESTDIR=$RTE_DESTDIR

setup driver & bind nic

modprobe uio insmod $RTE_DESTDIR/lib/modules/$(uname -r)/extra/dpdk/igb_uio.ko

config huge page 1/4 of total memory

mkdir -p /mnt/huge umount -a -t hugetlbfs mount -t hugetlbfs nodev /mnt/huge total_mem=cat /proc/meminfo | grep MemTotal | awk '{print $2}' hugepages_mem=$(($total_mem/80000001024)) echo $hugepages_mem > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages if [[ -d /sys/devices/system/node/node1 ]]; then hugepages_mem=$(($total_mem/80000001024/2)) echo $hugepages_mem > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages echo $hugepages_mem > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages else hugepages_mem=$(($total_mem/8000000*1024)) echo $hugepages_mem > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages fi

ifconfig eth1 down $RTE_DESTDIR/share/dpdk/usertools/dpdk-devbind.py --bind=igb_uio eth1 $RTE_DESTDIR/share/dpdk/usertools/dpdk-devbind.py --status

odp - clone & make & install

export ODP_SDK=/root/odp export ODP_DESTDIR=$ODP_SDK/install git clone https://github.com/OpenDataPlane/odp-dpdk.git $ODP_SDK --branch=v1.19.0.2_DPDK_17.11 --depth 1

cd $ODP_SDK ./bootstrap ./configure --disable-shared --with-dpdk-path=$RTE_DESTDIR --prefix=$ODP_DESTDIR make -j8 install

ofp - clone & make & install

export OFP_SDK=/root/ofp export OFP_DESTDIR=$OFP_SDK/install git clone https://github.com/OpenFastPath/ofp.git $OFP_SDK --branch=master --depth 1

cd $OFP_SDK ./bootstrap ./configure --disable-shared --enable-debug --enable-debug-print --prefix=$OFP_DESTDIR --with-odp=$ODP_DESTDIR make -j8 install

InJoins commented 5 years ago

minimal ofp example fpm run failed

os: ubuntu 18.04 nic: virtio in a cloud btw: odp examples can run successfully

root@ubuntu18.04:~/ofp/example/fpm# sudo ./fpm -i 0 EAL: Detected 8 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! Queue config: queue_basic.max_queue_size: 8192 queue_basic.default_queue_size: 4096

Using scheduler 'basic' Scheduler config: sched_basic.prio_spread: 4

PKTIO: initialized loop interface. PKTIO: initialized null interface. No crypto devices available odp_thread.c:165:odp_thread_init_local():There is a thread already running on core 0

ODP system info

ODP API version: 1.19.0 CPU model: Intel(R) Xeon(R) Gold 61xx CPU

CPU freq (hz): 2394374000 Cache line size: 64 Core count: 8

Running ODP appl: "fpm"

IF-count: 1 Using IFs: 0

D 0 0:530788672 ofp_init.c:199] Using configuration file: /root/ofp/install/etc/ofp.conf

E 0 0:530788672 ofp_init.c:203] (null)(0): file I/O error

Num worker threads: 6 first CPU: 2 cpu mask: 0xFC I -1656795248 0:530788672 ofp_uma.c:45] Creating pool 'udp_inpcb', nitems=1024 size=904 total=925696 I -1655618674 0:530788672 ofp_uma.c:45] Creating pool 'tcp_inpcb', nitems=2048 size=904 total=1851392 I -1653930752 0:530788672 ofp_uma.c:45] Creating pool 'tcpcb', nitems=2048 size=784 total=1605632 I -1652374400 0:530788672 ofp_uma.c:45] Creating pool 'tcptw', nitems=409 size=80 total=32720 I -1652153706 0:530788672 ofp_uma.c:45] Creating pool 'syncache', nitems=30720 size=168 total=5160960 I -1643970970 0:530788672 ofp_uma.c:45] Creating pool 'tcpreass', nitems=320 size=48 total=15360 I -1643289108 0:530788672 ofp_uma.c:45] Creating pool 'sackhole', nitems=65536 size=40 total=2621440 odp_crypto.c:556:odp_crypto_capability():No crypto devices available E -1627998904 0:530788672 ofp_ipsec.c:180] odp_ipsec_capability failed E -1627993334 0:530788672 ofp_ipsec.c:181] Setting maximum number of IPsec SAs to zero I -1627981386 0:530788672 ofp_init.c:434] Slow path threads on core 0 D -1627920728 0:530788672 ofp_ifnet.c:302] Interface '0' becomes 'fp0', port 0 ../linux-generic/odp_packet_io.c:233:setup_pktio_entry():Unable to init any I/O type. E -1627872632 0:530788672 ofp_ifnet.c:23] odp_pktio_open failed Error: OFP global init failed.

InJoins commented 5 years ago

Finally, I get compiled & run succeed. Everything must be compiled/linked with share lib, below is the new compile script.

!/bin/sh

prepare

apt -y install libelf-dev apt -y install automake apt -y install autoconf apt -y install libtool apt -y install pkg-config apt -y install libconfig-dev apt -y install libssl-dev apt -y install libnuma-dev

dpdk - clone & make & install

export RTE_SDK=/root/dpdk export RTE_TARGET=x86_64-native-linuxapp-gcc export RTE_DESTDIR=$RTE_SDK/install git clone https://github.com/DPDK/dpdk.git $RTE_SDK --branch=v17.11 --depth 1

sed -i 's/CONFIG_RTE_BUILD_SHARED_LIB=y/CONFIG_RTE_BUILD_SHARED_LIB=n/' $RTE_SDK/config/common_base sed -ri 's,(CONFIG_RTE_LIBRTE_PMD_OPENSSL=).*,\1y,' $RTE_SDK/$RTE_TARGET/.config

cd $RTE_SDK make config T=$RTE_TARGET make -j8 install T=$RTE_TARGET DESTDIR=$RTE_DESTDIR EXTRA_CFLAGS="-fPIC"

setup driver & bind nic

modprobe uio insmod $RTE_DESTDIR/lib/modules/$(uname -r)/extra/dpdk/igb_uio.ko ifconfig eth1 down $RTE_DESTDIR/share/dpdk/usertools/dpdk-devbind.py --bind=igb_uio eth1 $RTE_DESTDIR/share/dpdk/usertools/dpdk-devbind.py --status

config huge page 1/4 of total memory

mkdir -p /mnt/huge umount -a -t hugetlbfs mount -t hugetlbfs nodev /mnt/huge total_mem=cat /proc/meminfo | grep MemTotal | awk '{print $2}' hugepages_mem=$(($total_mem/8000000*1024)) echo $hugepages_mem > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

odp - clone & make & install

export ODP_SDK=/root/odp export ODP_DESTDIR=$ODP_SDK/install git clone https://github.com/OpenDataPlane/odp-dpdk.git $ODP_SDK --branch=v1.19.0.2_DPDK_17.11 --depth 1

cd $ODP_SDK ./bootstrap ./configure --enable-dpdk --with-dpdk-path=$RTE_DESTDIR --prefix=$ODP_DESTDIR make -j8 install

ofp - clone & make & install

export OFP_SDK=/root/ofp export OFP_DESTDIR=$OFP_SDK/install git clone https://github.com/OpenFastPath/ofp.git $OFP_SDK --branch=master --depth 1

cd $OFP_SDK ./bootstrap ./configure --prefix=$OFP_DESTDIR --with-odp=$ODP_DESTDIR make -j8 install

InJoins commented 5 years ago

If static linked, dpdk get device wil return NULL