Receive UDP packets out of sequence from FPGA board over fiber

[dragonnull]

Hi, F-Stack! We transmit data via UDP. Since the Linux socket does not perform well at the 10 Gigabit fiber rate and often causes packet loss, we try to use F-Stack. We receive UDP data from the FPGA board, and the serial number increases from 1 to 65536, but the serial number received and parsed by F-Stack is out of order. I am a newbie to F-Stack. I tried using the helloworld program and the Makefile from the example, and the config.ini in the root directory. Modify them to complete my receive UDP program. So it's not clear to me if I'm using the F-Stack correctly, or if it's misconfigured. I will show my code and config.ini, hope someone can help me find the problem. Thank you so much for taking the time to read my question.

main.cpp

#include <array>
#include <cassert>
#include <cstring>
#include <iostream>
#include "sys/epoll.h"
#include <ff_epoll.h>
#include <unistd.h>

#include "ff_config.h"
#include "ff_api.h"

#define epoll

#ifdef kqueue

//int kq;
//kevent kv;
//kevent kevents[512];
#endif
#ifdef epoll

int epfd;
epoll_event ev;
epoll_event events[10];
#endif

int sockfd;
sockaddr_in server_addr{};
sockaddr_in remote_addr{};
socklen_t addr_len = sizeof(sockaddr_in);

std::array<char, 2000> buff{};
std::array<unsigned short, (1030 - 2) / 2> data{};
unsigned short before = 0;
unsigned short now = 0;

int udpSocket_build() {
    memset(&server_addr, 0, addr_len);
    server_addr.sin_family = AF_INET;
    server_addr.sin_addr.s_addr = htonl(INADDR_ANY);
    server_addr.sin_port = htons(41984);

    sockfd = ff_socket(PF_INET, SOCK_DGRAM, 0);
    if (sockfd < 0) {
        printf("ff_socket failed\n");
        return -1;
    }

    int ret = ff_bind(sockfd, (struct linux_sockaddr *) &server_addr, sizeof(server_addr));
    if (ret < 0) {
        printf("ff_bind failed\n");
        return -1;
    }

    return sockfd;
}

int recvLoop(void *arg) {
#ifdef epoll
    int nevents = ff_epoll_wait(epfd, events, 100, -1);
#endif

#ifdef kqueue
    //    int nevents = ff_kevent(kq, nullptr, 0, kevents, 512, nullptr);
#endif

    for (int i = 0; i < nevents; ++i) {
        if (events[i].data.fd == sockfd) {

#ifdef kqueue
            //        kevent kev = kevents[i];
            //        int clientfd = (int) kev.ident;
            //        if (kev.flags & EV_EOF) {
            //            ff_close(clientfd);
            //        } else if (clientfd == sockfd) {
#endif
            memset(buff.begin(), 0, buff.size());
            int n = ff_recvfrom(sockfd, buff.begin(), 1030, MSG_WAITALL, (linux_sockaddr *) &remote_addr, &addr_len);
            if (n < 0) {
                printf("ff_recvfrom error.\n");
            } else {
                memcpy(&now, buff.begin(), 2);
                memcpy(data.begin(), buff.begin(), 1030 - 2);
                printf("%d\n%d    %d    %d    %d\n", now - before,
                       data.at(0), data.at(1), data.at(2), data.at(3));
//                before = now;
            }

#ifdef kqueue
            //            EV_SET(&kv, n, EVFILT_READ, EV_ADD, 0, 0, nullptr);
#endif
        }
    }
}

int main(int argc, char *argv[]) {
    if (ff_init(argc, argv) < 0) {
        perror("init error.\n");
        return -1;
    }

#ifdef epoll
    assert((epfd = ff_epoll_create(10)) > 0);
#endif

#ifdef kqueue
    //    assert((kq = ff_kqueue()) > 0);
#endif

    udpSocket_build();

#ifdef epoll
    ev.data.fd = sockfd;
    ev.events = EPOLLIN;
    ff_epoll_ctl(epfd, EPOLL_CTL_ADD, sockfd, &ev);
#endif

#ifdef kqueue
    //    EV_SET(&kv, sockfd, EVFILT_READ, EV_ADD, 0, 512, nullptr);
    //    ff_kevent(kq, &kv, 1, nullptr, 0, nullptr);
#endif

    ff_run(recvLoop, nullptr);
    return 0;
}

config.ini

[dpdk]
# Hexadecimal bitmask of cores to run on.
lcore_mask=3

# Number of memory channels.
channel=8

# Specify base virtual address to map.
#base_virtaddr=0x7f0000000000

# Promiscuous mode of nic, defualt: enabled.
promiscuous=1
numa_on=1

# TX checksum offload skip, default: disabled.
# We need this switch enabled in the following cases:
# -> The application want to enforce wrong checksum for testing purposes
# -> Some cards advertize the offload capability. However, doesn't calculate checksum.
tx_csum_offoad_skip=0

# TCP segment offload, default: disabled.
tso=0

# HW vlan strip, default: enabled.
vlan_strip=1

# sleep when no pkts incomming
# unit: microseconds
idle_sleep=0

# sent packet delay time(0-100) while send less than 32 pkts.
# default 100 us.
# if set 0, means send pkts immediately.
# if set >100, will dealy 100 us.
# unit: microseconds
pkt_tx_delay=100

# use symmetric Receive-side Scaling(RSS) key, default: disabled.
symmetric_rss=0

# enabled port list
#
# EBNF grammar:
#
#    exp      ::= num_list {"," num_list}
#    num_list ::= <num> | <range>
#    range    ::= <num>"-"<num>
#    num      ::= '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9'
#
# examples
#    0-3       ports 0, 1,2,3 are enabled
#    1-3,4,7   ports 1,2,3,4,7 are enabled
#
# If use bonding, shoule config the bonding port id in port_list
# and not config slave port id in port_list
# such as, port 0 and port 1 trank to a bonding port 2,
# should set `port_list=2` and config `[port2]` section

port_list=0

# Number of vdev.
nb_vdev=0

# Number of bond.
nb_bond=0

# Each core write into own pcap file, which is open one time, close one time if enough.
# Support dump the first snaplen bytes of each packet.
# if pcap file is lager than savelen bytes, it will be closed and next file was dumped into.
[pcap]
enable = 0
snaplen= 16777216
savelen= 16777216

# Port config section
# Correspond to dpdk.port_list's index: port0, port1...
[port0]
addr=192.168.10.86
netmask=255.255.225.0
broadcast=192.168.10.255
gateway=192.168.10.1
# IPv6 net addr
# Optional parameters
#addr6=ff::02
#prefix_len=64
#gateway6=ff::01

# lcore list used to handle this port
# the format is same as port_list
lcore_list=0,1

# bonding slave port list used to handle this port
# need to config while this port is a bonding port
# the format is same as port_list
#slave_port_list=0,1

# Packet capture path, this will hurt performance
#pcap=./a.pcap

# Vdev config section
# orrespond to dpdk.nb_vdev's index: vdev0, vdev1...
#    iface : Shouldn't set always.
#    path : The vuser device path in container. Required.
#    queues : The max queues of vuser. Optional, default 1, greater or equal to the number of processes.
#    queue_size : Queue size.Optional, default 256.
#    mac : The mac address of vuser. Optional, default random, if vhost use phy NIC, it should be set to the phy NIC's mac.
#    cq : Optional, if queues = 1, default 0; if queues > 1 default 1.
#[vdev0]
##iface=/usr/local/var/run/openvswitch/vhost-user0
#path=/var/run/openvswitch/vhost-user0
#queues=1
#queue_size=256
#mac=00:00:00:00:00:01
#cq=0

# bond config section
# See http://doc.dpdk.org/guides/prog_guide/link_bonding_poll_mode_drv_lib.html
#[bond0]
#mode=4
#slave=0000:0a:00.0,slave=0000:0a:00.1
#primary=0000:0a:00.0
#mac=f0:98:38:xx:xx:xx
## opt argument
#socket_id=0
#xmit_policy=l23
#lsc_poll_period_ms=100
#up_delay=10
#down_delay=50

# Kni config: if enabled and method=reject,
# all packets that do not belong to the following tcp_port and udp_port
# will transmit to kernel; if method=accept, all packets that belong to
# the following tcp_port and udp_port will transmit to kernel.
[kni]
enable=1
method=reject
# The format is same as port_list
tcp_port=80,443
udp_port=53,41984

# FreeBSD network performance tuning configurations.
# Most native FreeBSD configurations are supported.
[freebsd.boot]
hz=100

# Block out a range of descriptors to avoid overlap
# with the kernel's descriptor space.
# You can increase this value according to your app.
fd_reserve=1024

kern.ipc.maxsockets=262144

net.inet.tcp.syncache.hashsize=4096
net.inet.tcp.syncache.bucketlimit=100

net.inet.tcp.tcbhashsize=65536

kern.ncallout=262144

kern.features.inet6=1
net.inet6.ip6.auto_linklocal=1
net.inet6.ip6.accept_rtadv=2
net.inet6.icmp6.rediraccept=1
net.inet6.ip6.forwarding=0

[freebsd.sysctl]
kern.ipc.somaxconn=32768
kern.ipc.maxsockbuf=16777216

net.link.ether.inet.maxhold=5

net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.sendspace=16384
net.inet.tcp.recvspace=8192
#net.inet.tcp.nolocaltimewait=1
net.inet.tcp.cc.algorithm=cubic
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.sack.enable=1
net.inet.tcp.blackhole=1
net.inet.tcp.msl=2000
net.inet.tcp.delayed_ack=0

net.inet.udp.recvspace=8388608
net.inet.udp.blackhole=1
net.inet.ip.redirect=0
net.inet.ip.forwarding=0

Makefile

TOPDIR=..

ifeq ($(FF_PATH),)
    FF_PATH=${TOPDIR}
endif

ifeq ($(FF_DPDK),)
    FF_DPDK=${TOPDIR}/dpdk/x86_64-native-linuxapp-gcc
endif

LIBS+= -L${FF_PATH}/lib -Wl,--whole-archive,-lfstack,--no-whole-archive
LIBS+= -L${FF_DPDK}/lib -Wl,--whole-archive,-ldpdk,--no-whole-archive
LIBS+= -Wl,--no-whole-archive -lrt -lm -ldl -lcrypto -pthread -lnuma

TARGET="helloworld"
all:
    g++ -std=c++11 -O -gdwarf-2  -I../lib -o ${TARGET} main.cpp ${LIBS}

.PHONY: clean
clean:
    rm -f *.o ${TARGET}

[jfb8856606]

There may be multiple reasons for the out-of-order problem，suah as multiple processes use different queues, and you can try modify lcore_mask=3 to lcore_mask=2 and retry it.

In fact, we usually do not use the protocol stack when processing UDP packets，but the application itself process it, for example through ff_regist_packet_dispatcher.

[dragonnull]

Thank you for your reply. Sorry to get back to you on Monday, for some reason I can't use the server on weekends. I have simplified the environment. The FPGA board is directly connected to the network card using F-Stack, and the network card is only used to receive data from the FPGA. About what you mentioned "multiple processes use different queues", when testing, I've made sure that the server is only running my program. I have tried lcore_mask=2 and it didn't work. Next I will try ff_regist_packet_dispatcher and hope it works.