search

F-Stack / f-stack

F-Stack is an user space network development kit with high performance based on DPDK, FreeBSD TCP/IP stack and coroutine API.
http://www.f-stack.org
Other
3.81k stars 886 forks source link

Documents on how to run f-stack with vdev in container with OVS-DPDK ? #501

Open vincentmli opened 4 years ago

vincentmli commented 4 years ago

Hi,

I searched through the issues report, there are a few tickets asking running f-stack with vdev, for example: https://github.com/F-Stack/f-stack/issues/419, https://github.com/F-Stack/f-stack/issues/494..

Could f-stack provide a basic howto/steps documents to run f-stack in container with vdev ? it appears tested working for f-stack nginx in container with virtio-user and ovs-dpdk vhost-user from https://doc.dpdk.org/guides/howto/virtio_user_for_container_networking.html. I think f-stack with similar documents would be helpful.

I could contribute the document but I need some guidance, I am kind of familiar with OVS-DPDK side, but I am not familiar with building f-stack nginx container and f-stack vdev side of configuration.

jfb8856606 commented 4 years ago

We will add the doc the later, or if you has some questions you can send email to me

eweyulu commented 3 years ago

Hi there,

I am learning how to use F-stack in a project that requires a userspace TCP stack. I installed and compiled everything successfully but when I run the helloworld example, this is all I get:

#:~/f-stack$ sudo ./example/helloworld 
invalid proc_id:-1, use default 0
[dpdk]: lcore_mask=1
[dpdk]: channel=4
[dpdk]: promiscuous=1
[dpdk]: numa_on=1
[dpdk]: tx_csum_offoad_skip=0
[dpdk]: tso=0
[dpdk]: vlan_strip=1
[dpdk]: idle_sleep=0
[dpdk]: pkt_tx_delay=100
[dpdk]: symmetric_rss=0
[dpdk]: port_list=0
[dpdk]: nb_vdev=0
[dpdk]: nb_bond=0
[pcap]: enable=0
[pcap]: snaplen=96
[pcap]: savelen=16777216
[port0]: addr=10.8.100.2
[port0]: netmask=255.255.225.0
[port0]: broadcast=10.8.100.255
[port0]: gateway=10.8.100.1
[port0]: pcap=./a.pcap
[freebsd.boot]: hz=100
[freebsd.boot]: fd_reserve=1024
[freebsd.boot]: kern.ipc.maxsockets=262144
[freebsd.boot]: net.inet.tcp.syncache.hashsize=4096
[freebsd.boot]: net.inet.tcp.syncache.bucketlimit=100
[freebsd.boot]: net.inet.tcp.tcbhashsize=65536
[freebsd.boot]: kern.ncallout=262144
[freebsd.boot]: kern.features.inet6=1
[freebsd.boot]: net.inet6.ip6.auto_linklocal=1
[freebsd.boot]: net.inet6.ip6.accept_rtadv=2
[freebsd.boot]: net.inet6.icmp6.rediraccept=1
[freebsd.boot]: net.inet6.ip6.forwarding=0
[freebsd.sysctl]: kern.ipc.somaxconn=32768
[freebsd.sysctl]: kern.ipc.maxsockbuf=16777216
[freebsd.sysctl]: net.link.ether.inet.maxhold=5
[freebsd.sysctl]: net.inet.tcp.fast_finwait2_recycle=1
[freebsd.sysctl]: net.inet.tcp.sendspace=16384
[freebsd.sysctl]: net.inet.tcp.recvspace=8192
[freebsd.sysctl]: net.inet.tcp.cc.algorithm=cubic
[freebsd.sysctl]: net.inet.tcp.sendbuf_max=16777216
[freebsd.sysctl]: net.inet.tcp.recvbuf_max=16777216
[freebsd.sysctl]: net.inet.tcp.sendbuf_auto=1
[freebsd.sysctl]: net.inet.tcp.recvbuf_auto=1
[freebsd.sysctl]: net.inet.tcp.sendbuf_inc=16384
[freebsd.sysctl]: net.inet.tcp.recvbuf_inc=524288
[freebsd.sysctl]: net.inet.tcp.sack.enable=1
[freebsd.sysctl]: net.inet.tcp.blackhole=1
[freebsd.sysctl]: net.inet.tcp.msl=2000
[freebsd.sysctl]: net.inet.tcp.delayed_ack=0
[freebsd.sysctl]: net.inet.udp.blackhole=1
[freebsd.sysctl]: net.inet.ip.redirect=0
[freebsd.sysctl]: net.inet.ip.forwarding=0
EAL: Detected 16 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
f-stack -c1 -n4 --proc-type=auto EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL:   using IOMMU type 1 (Type 1)
EAL: PCI device 0000:02:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL: PCI device 0000:c4:00.0 on NUMA socket 0
EAL:   probe driver: 1077:8070 net_qede
EAL: PCI device 0000:c4:00.1 on NUMA socket 0
EAL:   probe driver: 1077:8070 net_qede
lcore: 0, port: 0, queue: 0
create mbuf pool on socket 0
create ring:dispatch_ring_p0_q0 success, 2047 ring entries are now free!
Port 0 MAC: bc 97 e1 28 b5 20
Port 0 modified RSS hash function based on hardware support,requested:0x3ffffc configured:0xd34
RX checksum offload supported
TX ip checksum offload supported
TX TCP&UDP checksum offload supported
TSO is disabled
port[0]: rss table size: 128
set port 0 to promiscuous mode ok

Checking link statusdone
Port 0 Link Up - speed 25000 Mbps - full-duplex
link_elf_lookup_symbol: missing symbol hash table
link_elf_lookup_symbol: missing symbol hash table
Timecounters tick every 10.000 msec
Timecounter "ff_clock" frequency 100 Hz quality 1
f-stack-0: Ethernet address: bc:97:e1:28:b5:20

^C

and then it basically hangs. I am a beginner and I am not really sure if I am running the example correctly. Do you have some simple examples I can use to test my setup? I can see that my nginx server comes up when I do top:

#:~/f-stack$ top
top - 23:50:50 up 1 day, 11:12,  1 user,  load average: 4.00, 3.94, 3.05
Tasks: 263 total,   5 running, 258 sleeping,   0 stopped,   0 zombie
%Cpu(s): 24.9 us,  0.4 sy,  0.0 ni, 74.3 id,  0.0 wa,  0.0 hi,  0.4 si,  0.0 st
MiB Mem : 128372.4 total, 119836.9 free,   3201.1 used,   5334.3 buff/cache
MiB Swap:   2000.0 total,   2000.0 free,      0.0 used. 123923.4 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                               
25131 root      20   0   64.2g  80916   8540 R 100.0   0.1  17:56.73 nginx                                                 
25117 root      20   0   64.2g  81608   8956 R  93.8   0.1  17:17.25 nginx                                                 
25130 root      20   0   64.2g  80916   8540 R  93.8   0.1  17:56.70 nginx                                                 
25132 root      20   0   64.2g  80916   8540 R  93.8   0.1  17:56.73 nginx`

And that it is listening for incoming connections:

#:~/f-stack$ sudo ff_netstat -P 0 -a
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4       0      0 *.http                 *.*                    LISTEN
udp4       0      0 *.*                    *.*

Do you have any pointers as to how I can access this server from the outside? Just doing curl to the ip configured in f-stack.conf does not work.

This is my f-stack.conf file:

[dpdk]
# Hexadecimal bitmask of cores to run on.
lcore_mask=f

# Number of memory channels.
channel=4

# Specify base virtual address to map.
#base_virtaddr=0x7f0000000000

# Promiscuous mode of nic, defualt: enabled.
promiscuous=1
numa_on=1

# TX checksum offload skip, default: disabled.
# We need this switch enabled in the following cases:
# -> The application want to enforce wrong checksum for testing purposes
# -> Some cards advertize the offload capability. However, doesn't calculate checksum.
tx_csum_offoad_skip=0

# TCP segment offload, default: disabled.
tso=0

# HW vlan strip, default: enabled.
vlan_strip=1

# sleep when no pkts incomming
# unit: microseconds
idle_sleep=0

# sent packet delay time(0-100) while send less than 32 pkts.
# default 100 us.
# if set 0, means send pkts immediately.
# if set >100, will dealy 100 us.
# unit: microseconds
pkt_tx_delay=100

# enabled port list
#
# EBNF grammar:
#
#    exp      ::= num_list {"," num_list}
#    num_list ::= <num> | <range>
#    range    ::= <num>"-"<num>
#    num      ::= '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9'
#
# examples
#    0-3       ports 0, 1,2,3 are enabled
#    1-3,4,7   ports 1,2,3,4,7 are enabled
#
# If use bonding, shoule config the bonding port id in port_list
# and not config slave port id in port_list
# such as, port 0 and port 1 trank to a bonding port 2,
# should set `port_list=2` and config `[port2]` section

port_list=0

# Number of vdev.
nb_vdev=0

# Number of bond.
nb_bond=0

# Port config section
# Correspond to dpdk.port_list's index: port0, port1...
[port0]
addr=10.8.100.2
netmask=255.255.225.0
broadcast=10.8.100.255
gateway=10.8.100.1
lcore_list=0-3

#[port1]
#addr=192.168.2.2
#netmask=255.255.225.0
#broadcast=192.168.2.255
#gateway=192.168.2.1
#lcore_list=0-3
# lcore list used to handle this port
# the format is same as port_list
#lcore_list=0

# bonding slave port list used to handle this port
# need to config while this port is a bonding port
# the format is same as port_list
#slave_port_list=0,1

# Packet capture path, this will hurt performance
#pcap=./a.pcap

# Vdev config section
# orrespond to dpdk.nb_vdev's index: vdev0, vdev1...
#    iface : Shouldn't set always.
#    path : The vuser device path in container. Required.
#    queues : The max queues of vuser. Optional, default 1, greater or equal to the number of processes.
#    queue_size : Queue size.Optional, default 256.
#    mac : The mac address of vuser. Optional, default random, if vhost use phy NIC, it should be set to the phy NIC's mac.
#    cq : Optional, if queues = 1, default 0; if queues > 1 default 1.
#[vdev0]
##iface=/usr/local/var/run/openvswitch/vhost-user0
#path=/var/run/openvswitch/vhost-user0
#queues=1
#queue_size=256
#mac=00:00:00:00:00:01
#cq=0

# bond config section
# See http://doc.dpdk.org/guides/prog_guide/link_bonding_poll_mode_drv_lib.html
[bond0]
#mode=4
#slave=0000:0a:00.0,slave=0000:0a:00.1
#primary=0000:0a:00.0
#mac=f0:98:38:xx:xx:xx
## opt argument
#socket_id=0
#xmit_policy=l23
#lsc_poll_period_ms=100
#up_delay=10
#down_delay=50

# Kni config: if enabled and method=reject,
# all packets that do not belong to the following tcp_port and udp_port
# will transmit to kernel; if method=accept, all packets that belong to
# the following tcp_port and udp_port will transmit to kernel.
[kni]
enable=1
method=reject
# The format is same as port_list
tcp_port=80,443
#udp_port=53

# FreeBSD network performance tuning configurations.
# Most native FreeBSD configurations are supported.
[freebsd.boot]
hz=100

# Block out a range of descriptors to avoid overlap
# with the kernel's descriptor space.
# You can increase this value according to your app.
fd_reserve=1024

kern.ipc.maxsockets=262144

net.inet.tcp.syncache.hashsize=4096
net.inet.tcp.syncache.bucketlimit=100

net.inet.tcp.tcbhashsize=65536

kern.ncallout=262144

kern.features.inet6=1
net.inet6.ip6.auto_linklocal=1
net.inet6.ip6.accept_rtadv=2
net.inet6.icmp6.rediraccept=1
net.inet6.ip6.forwarding=0

[freebsd.sysctl]
kern.ipc.somaxconn=32768
kern.ipc.maxsockbuf=16777216

net.link.ether.inet.maxhold=5

net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.sendspace=16384
net.inet.tcp.recvspace=8192
#net.inet.tcp.nolocaltimewait=1
net.inet.tcp.cc.algorithm=cubic
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.sack.enable=1
net.inet.tcp.blackhole=1
net.inet.tcp.msl=2000
net.inet.tcp.delayed_ack=0

net.inet.udp.blackhole=1
net.inet.ip.redirect=0

This is my nginx.conf file:

# root account is necessary.
user  root;
# should be equal to the lcore count of `dpdk.lcore_mask` in f-stack.conf.
worker_processes  4;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;

# path of f-stack configuration file, default: $NGX_PREFIX/conf/f-stack.conf.
fstack_conf f-stack.conf;

events {
    worker_connections  102400;
    use kqueue;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        off;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    #upstream fstack_nginx_server {
    #     server 192.168.1.1:80;
    #}

    server {
        listen       80;
        server_name  localhost;

        # bulid server on kernel network stack 
        # 
        #kernel_network_stack on;

        #charset koi8-r;

        access_log /dev/null;
        #access_log  logs/host.access.log  main;

        location / {
            #proxy_pass http://fstack_nginx_server;
            root   html;
            index  index.html index.htm;
            #return 200 "<title>Welcome to F-Stack Nginx!</title>\r\n pad data:0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789";
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }

    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       5001;
    #    listen       10.8.8.8:5001;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}

    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;

    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}

}

Thank you and I appreciate any pointers that you may have.