search

F-Stack / f-stack

F-Stack is an user space network development kit with high performance based on DPDK, FreeBSD TCP/IP stack and coroutine API.
http://www.f-stack.org
Other
3.76k stars 879 forks source link

Can't send request to running helloworld program #741

Open adribr53 opened 1 year ago

adribr53 commented 1 year ago

Hello, I have tried to get started with f-stack for a few days now. I managed to build the lib/tools/helloworld/nginx programs, but could not manage to actually use them.

I have 2 scripts : one to build, one for seting up the system and then running the hello world program.

Executin 'bash build.sh', and then 'bash set_up.sh eno2', I have a helloworld program up and running but I cannot reach it. Here a few outputs :

Stdout for starting the helloworld application :

./example/helloworld --conf config.ini --proc-type=primary --proc-id=0 
[dpdk]: lcore_mask=1
[dpdk]: channel=4
[dpdk]: promiscuous=1
[dpdk]: numa_on=1
[dpdk]: tx_csum_offoad_skip=0
[dpdk]: tso=0
[dpdk]: vlan_strip=1
[dpdk]: idle_sleep=0
[dpdk]: pkt_tx_delay=100
[dpdk]: symmetric_rss=0
[dpdk]: port_list=0
[dpdk]: nb_vdev=0
[dpdk]: nb_bond=0
[pcap]: enable=0
[pcap]: snaplen=96
[pcap]: savelen=16777216
[pcap]: savepath=.
[port0]: addr=x.x.97.114
[port0]: netmask=255.255.240.0
[port0]: broadcast=x.x.111.255
[port0]: gateway=x.x.111.254
[kni]: enable=1
[kni]: method=reject
[kni]: tcp_port=80,443
[freebsd.boot]: hz=100
[freebsd.boot]: fd_reserve=1024
[freebsd.boot]: kern.ipc.maxsockets=262144
[freebsd.boot]: net.inet.tcp.syncache.hashsize=4096
[freebsd.boot]: net.inet.tcp.syncache.bucketlimit=100
[freebsd.boot]: net.inet.tcp.tcbhashsize=65536
[freebsd.boot]: kern.ncallout=262144
[freebsd.boot]: kern.features.inet6=1
[freebsd.boot]: net.inet6.ip6.auto_linklocal=1
[freebsd.boot]: net.inet6.ip6.accept_rtadv=2
[freebsd.boot]: net.inet6.icmp6.rediraccept=1
[freebsd.boot]: net.inet6.ip6.forwarding=0
[freebsd.sysctl]: kern.ipc.somaxconn=32768
[freebsd.sysctl]: kern.ipc.maxsockbuf=16777216
[freebsd.sysctl]: net.link.ether.inet.maxhold=5
[freebsd.sysctl]: net.inet.tcp.fast_finwait2_recycle=1
[freebsd.sysctl]: net.inet.tcp.sendspace=16384
[freebsd.sysctl]: net.inet.tcp.recvspace=8192
[freebsd.sysctl]: net.inet.tcp.cc.algorithm=cubic
[freebsd.sysctl]: net.inet.tcp.sendbuf_max=16777216
[freebsd.sysctl]: net.inet.tcp.recvbuf_max=16777216
[freebsd.sysctl]: net.inet.tcp.sendbuf_auto=1
[freebsd.sysctl]: net.inet.tcp.recvbuf_auto=1
[freebsd.sysctl]: net.inet.tcp.sendbuf_inc=16384
[freebsd.sysctl]: net.inet.tcp.sack.enable=1
[freebsd.sysctl]: net.inet.tcp.blackhole=1
[freebsd.sysctl]: net.inet.tcp.msl=2000
[freebsd.sysctl]: net.inet.tcp.delayed_ack=1
[freebsd.sysctl]: net.inet.tcp.rfc1323=1
[freebsd.sysctl]: net.inet.udp.blackhole=1
[freebsd.sysctl]: net.inet.ip.redirect=0
[freebsd.sysctl]: net.inet.ip.forwarding=0
[freebsd.sysctl]: net.inet.tcp.functions_default=freebsd
[freebsd.sysctl]: net.inet.tcp.hpts.skip_swi=1
EAL: Detected 32 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:01:00.1 (socket 0)
EAL: No legacy callbacks, legacy socket not created
f-stack -c1 -n4 --proc-type=primary lcore: 0, port: 0, queue: 0
create mbuf pool on socket 0
create ring:dispatch_ring_p0_q0 success, 2047 ring entries are now free!
create kni on port 0 success!
create kni ring success, 2047 ring entries are now free!
Port 0 MAC: ec f4 bb d1 05 92
Port 0 modified RSS hash function based on hardware support,requested:0x3ffffc configured:0x38d34
RX checksum offload supported
TX ip checksum offload supported
TX TCP&UDP checksum offload supported
TSO is disabled
port[0]: rss table size: 128
set port 0 to promiscuous mode ok

Checking link status............................done
Port 0 Link Up - speed 10000 Mbps - full-duplex
link_elf_lookup_symbol: missing symbol hash table
link_elf_lookup_symbol: missing symbol hash table
Timecounters tick every 10.000 msec
WARNING: Adding ifaddrs to all fibs has been turned off by default. Consider tuning net.add_addr_allfibs if needed
Attempting to load tcp_bbr
tcp_bbr is now available
TCP Hpts created 1 swi interrupt threads and bound 0 to cpus
Timecounter "ff_clock" frequency 100 Hz quality 1
TCP_ratelimit: Is now initialized
f-stack-0: No addr6 config found.
f-stack-0: Ethernet address: ec:f4:bb:d1:05:92
x.x.97.114
Configure network interface of 0 up
    Address:EC:F4:BB:D1:05:92

ifconfig :

veth0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet x.x.97.114  netmask 255.255.240.0  broadcast x.x.111.255
        ether ec:f4:bb:d1:05:92  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ff_ifconfig :

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1 prefixlen 64 scopeid 0x1
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
f-stack-0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=3<RXCSUM,TXCSUM>
    ether ec:f4:bb:d1:5:92
    inet x.x.97.114 netmask 0xfffff000 broadcast x.x.111.255
    inet6 fe80::eef4:bbff:fed1:592 prefixlen 64 scopeid 0x2
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

ff_arp -an

? (x.x.97.114) at ec:f4:bb:d1:5:92 on f-stack-0 permanent [ethernet]

top

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
492634 root      20   0  128.1g  43148  15284 R  99.7   0.0  11:36.58 helloworld  
...

ff_top

|---------|---------|---------|---------------|
|     idle|      sys|      usr|           loop|
|---------|---------|---------|---------------|
|   99.80%|    0.00%|    0.20%|        7233913|
|   99.80%|    0.00%|    0.20%|        7194194|
|   99.80%|    0.00%|    0.20%|        7189648|
|   99.80%|    0.00%|    0.20%|        7192832|
| .... (and so one)                           |

when trying to reach the helloworld server from another machine

curl http://x.x.97.114
curl: (7) Failed to connect to x.x.97.114 port 80: Connection refused

(netcat works fine). Also, ff_traffic shows no traffic.

Any hint on what the problem might be ? Thanks for your time

Machine :

Scripts : set_env.sh : 

#!/bin/bash

cd ..

if [ $# -eq 0 ]
then
  echo "need an argument: set_env.sh dev_name"
  exit 0
fi

DEV=$1

install_dependencies() {  
  apt update
  apt -y install openssl bc libnuma1 libpcre3 python  
}

link_executables() {
  mkdir /data/f-stack/dpdk/build/kmod
  cp /data/f-stack/dpdk/build/kernel/linux/igb_uio/igb_uio.ko /data/f-stack/dpdk/build/kmod/igb_uio.ko
  cp /data/f-stack/dpdk/build/kernel/linux/kni/rte_kni.ko /data/f-stack/dpdk/build/kmod/rte_kni.ko
}

install_nginx() {
  if [ -d "/usr/local/nginx_fstack" ]
  then
    echo "/usr/local/nginx_fstack already exists"
    return
  else
    cp -r /data/f-stack/app/nginx-1.16.1/nginx_fstack /usr/local/nginx_fstack
    cp /data/f-stack/config.ini /usr/local/nginx_fstack/conf/f-stack.conf
  fi
}

set_hugepages() {
  echo "set hugepages ..."
  echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
  if [ ! -d "/mnt/huge" ]
  then
    mkdir /mnt/huge
  fi
  mount -t hugetlbfs nodev /mnt/huge
  echo 0 > /proc/sys/kernel/randomize_va_space
}

install_mod() {
  echo "install mod uio and kni ..."
  modprobe uio
  insmod /data/f-stack/dpdk/build/kmod/igb_uio.ko
  insmod /data/f-stack/dpdk/build/kmod/rte_kni.ko
}

get_ifconfig() {
  get_ifconfig_by_iproute2
}

get_ifconfig_by_iproute2() {
  echo "save if info"
  ip_mask=`ip a show $DEV | grep inet | head -n 1 | awk -F ' ' '{print $2}'`
  ip=${ip_mask%%/*}
  mask=${ip_mask##*/}
  bc=`ip a show $DEV | grep inet | head -n 1 | awk -F ' ' '{print $4}'`
  hw=`ip a show $DEV | grep ether | awk -F ' ' '{print $2}'`
  gw=`ip route | grep br0 | grep default | awk -F ' ' '{print $3}'`

  M=$(( 0xffffffff ^ ((1 << (32-mask)) -1) ))
  mask=$(( (M>>24) & 0xff )).$(( (M>>16) & 0xff )).$(( (M>>8) & 0xff )).$(( M & 0xff ))
}

set_config_ini() {
  sed "s/addr=192.168.1.2/addr=${ip}/" -i /data/f-stack/config.ini
  sed "s/netmask=255.255.255.0/netmask=${mask}/" -i /data/f-stack/config.ini
  sed "s/broadcast=192.168.1.255/broadcast=${bc}/" -i /data/f-stack/config.ini
  sed "s/gateway=192.168.1.1/gateway=${gw}/" -i /data/f-stack/config.ini

  sed "s/#\[kni\]/\[kni\]/" -i /data/f-stack/config.ini
  sed "s/#enable=1/enable=1/" -i /data/f-stack/config.ini
  sed "s/#method=reject/method=reject/" -i /data/f-stack/config.ini
  sed "s/#tcp_port=80/tcp_port=80/" -i /data/f-stack/config.ini
  sed "s/#vlanstrip=1/vlanstrip=1/" -i /data/f-stack/config.ini
}

bind_dev() {
  ip link set dev $DEV down
  python3 /data/f-stack/dpdk/usertools/dpdk-devbind.py --bind=igb_uio $DEV
}

install_dependencies
link_executables
get_ifconfig
set_config_ini
install_nginx
set_hugepages
install_mod
bind_dev

echo "run helloworld"
cd /data/f-stack
/data/f-stack/start.sh

sleep 10
echo "${ip}"
ifconfig veth0 ${ip}  netmask ${mask}  broadcast ${bc} hw ether ${hw}

echo "set_env completed"

build.sh

apt update

# dependencies
apt -y install net-tools git gcc openssl libssl-dev linux-headers-$(uname -r) bc libnuma1 libnuma-dev libpcre3 libpcre3-dev zlib1g-dev meson  python3-pip unzip python-is-python3 netcat

# debian specific fix for when building dpdk
wget -O /usr/src/linux-headers-5.10.0-20-common/scripts/pahole-flags.sh https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/plain/scripts/pahole-flags.sh?h=v5.10.162
sudo chmod +x /usr/src/linux-headers-5.10.0-20-common/scripts/pahole-flags.sh

# check interfaces
ifconfig | tee ifconfig_res
route -n | tee route_res

mkdir /data
cd /data/ 

# build fstack
wget https://github.com/F-Stack/f-stack/archive/refs/tags/v1.22.zip
unzip v1.22.zip
mv f-stack-1.22 f-stack
pip3 install pyelftools
cd /data/f-stack/dpdk/
meson -Denable_kmods=true build
ninja -C build
ninja -C build install

cd /data/
# pkg config version
wget https://pkg-config.freedesktop.org/releases/pkg-config-0.29.2.tar.gz
tar xzvf pkg-config-0.29.2.tar.gz
cd pkg-config-0.29.2/
./configure --with-internal-glib
make
make install
ln -s /usr/local/bin/pkg-config /usr/bin/pkg-config

cd /data/
export FF_PATH=/data/f-stack
export PKG_CONFIG_PATH=/usr/lib64/pkgconfig:/usr/local/lib64/pkgconfig:/usr/lib/pkgconfig:/data/f-stack/dpdk/build/meson-private:/usr/lib/x86_64-linux-gnu/pkgconfig

# link dpdk lib
pkg-config --cflags libdpdk
pkg-config --libs libdpdk

# install api
cd /data/f-stack/lib/
make
make install 

# install nginx app
cd /data/f-stack/app/nginx-1.16.1
sed -i '548,$d' ./src/event/modules/ngx_ff_module.c
echo 'int
gettimeofday(struct timeval *tv, void *__restrict tzp)
{
    struct timezone *tz = (struct timezone *) tzp;
    if (unlikely(inited == 0)) {
        return SYSCALL(gettimeofday)(tv, tz);
    }

    return ff_gettimeofday(tv, tz);
}' >> ./src/event/modules/ngx_ff_module.c
./configure --prefix=/usr/local/nginx_fstack --with-ff_module
make
make install

# install tools such as ifconfig
cd /data/f-stack/tools/
make
make install

cd /data/f-stack/example
make
jfb8856606 commented 1 year ago

ifconfig :

veth0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 NO RUNNING inet x.x.97.114 netmask 255.255.240.0 broadcast x.x.111.255 ether ec:f4:bb:d1:05:92 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

insmod /data/f-stack/dpdk/build/kmod/rte_kni.ko

I see your veth0 not RUNNING, and you need add argument carrier=on , otherwise need to be up veth0 via echo 1 > /sys/class/net/veth0/carrier.