search

pkelsey / libuinet

a library version of FreeBSD's TCP/IP stack plus extras
749 stars 198 forks source link

Not able to compile in Ubuntu #3

Closed petergsnm closed 10 years ago

petergsnm commented 10 years ago

peter:~/UTCP/libuinet/lib/libuinet$ make NETMAP_INCLUDES=/home/peter/UTCP/netmap/sys/ cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -std=c99 -ggdb -O0 -Wall -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wno-pointer-sign -Wmissing-include-dirs -fdiagnostics-show-option -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -DUINET=1 -D_BSD_SOURCE -fstack-protector -I. -I/home/peter/UTCP/libuinet/lib/libuinet/api_include -I/home/peter/UTCP/netmap/sys/ -Werror uinet_if_netmap_host.c In file included from /home/peter/UTCP/netmap/sys/net/netmap_user.h:70:0, from uinet_if_netmap_host.c:61: /usr/include/net/if.h:45:5: error: expected identifier before numeric constant /usr/include/net/if.h:112:8: error: redefinition of ‘struct ifmap’ /usr/include/linux/if.h:137:8: note: originally defined here /usr/include/net/if.h:127:8: error: redefinition of ‘struct ifreq’ /usr/include/linux/if.h:171:8: note: originally defined here /usr/include/net/if.h:177:8: error: redefinition of ‘struct ifconf’ /usr/include/linux/if.h:220:8: note: originally defined here uinet_if_netmap_host.c: In function ‘if_netmap_ethtool_set_flag’: uinet_if_netmap_host.c:320:16: error: assignment from incompatible pointer type [-Werror] uinet_if_netmap_host.c: In function ‘if_netmap_ethtool_set_discrete’: uinet_if_netmap_host.c:353:16: error: assignment from incompatible pointer type [-Werror] uinet_if_netmap_host.c: In function ‘if_netmap_set_promisc’: uinet_if_netmap_host.c:454:19: error: ‘IFF_PROMISC’ undeclared (first use in this function) uinet_if_netmap_host.c:454:19: note: each undeclared identifier is reported only once for each function it appears in cc1: all warnings being treated as errors make: *\ [uinet_if_netmap_host.o] Error 1 peter:~/UTCP/libuinet/lib/libuinet$

pkelsey commented 10 years ago

I haven't been back to testing on Linux in some time (main development is happening on FreeBSD), so this isn't surprising. This looks like something pretty fundamental.

Try adding

#include <stdint.h>

to lib/libuinet/uinet_if_pcap_host.c and see if that improves things for you.

petergsnm commented 10 years ago

Sorry... I updated this while you were adding the comments. By the way, I am using the latest netmap from git clone https://code.google.com/p/netmap/

pkelsey commented 10 years ago

What version of Ubuntu are you using?

petergsnm commented 10 years ago

I am using 12.04

uname -a Linux 3.8.0-35-generic #50~precise1-Ubuntu SMP Wed Dec 4 17:25:51 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

I have done the following changes to uinet_host_interface.h

+#include

Now, when I make with the following command, I get the following error -

peter:~/UTCP/libuinet/lib/libuinet$ make NETMAP_INCLUDES=/home/peter/UTCP/netmap/sys/ cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -std=c99 -ggdb -O0 -Wall -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wno-pointer-sign -Wmissing-include-dirs -fdiagnostics-show-option -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -DUINET=1 -fstack-protector -I. -I/home/peter/UTCP/libuinet/lib/libuinet/api_include -I/home/peter/UTCP/netmap/sys/ -Werror uinet_if_netmap_host.c In file included from /home/peter/UTCP/netmap/sys/net/netmap_user.h:70:0, from uinet_if_netmap_host.c:61: /usr/include/net/if.h:45:5: error: expected identifier before numeric constant /usr/include/net/if.h:112:8: error: redefinition of ‘struct ifmap’ /usr/include/linux/if.h:137:8: note: originally defined here /usr/include/net/if.h:127:8: error: redefinition of ‘struct ifreq’ /usr/include/linux/if.h:171:8: note: originally defined here /usr/include/net/if.h:177:8: error: redefinition of ‘struct ifconf’ /usr/include/linux/if.h:220:8: note: originally defined here uinet_if_netmap_host.c: In function ‘if_netmap_ethtool_set_flag’: uinet_if_netmap_host.c:320:16: error: assignment from incompatible pointer type [-Werror] uinet_if_netmap_host.c: In function ‘if_netmap_ethtool_set_discrete’: uinet_if_netmap_host.c:353:16: error: assignment from incompatible pointer type [-Werror] uinet_if_netmap_host.c: In function ‘if_netmap_set_promisc’: uinet_if_netmap_host.c:454:19: error: ‘IFF_PROMISC’ undeclared (first use in this function) uinet_if_netmap_host.c:454:19: note: each undeclared identifier is reported only once for each function it appears in cc1: all warnings being treated as errors make: *\ [uinet_if_netmap_host.o] Error 1 peter:~/UTCP/libuinet/lib/libuinet$

pkelsey commented 10 years ago

OK, let me try to reproduce.

btw, when putting code in these comments, enclose it in backticks, otherwise the markdown processing isn't friendly (note the include file in your last comment is blank)

petergsnm commented 10 years ago

Sure. Will take a note of it. I am using Ububtu 12.04. I followed the following steps -

  1. git clone https://code.google.com/p/netmap/
  2. git clone https://github.com/pkelsey/libuinet.git
  3. cd libuinet/lib/libuinet/
  4. make NETMAP_INCLUDES=/home/peter/UTCP/netmap/sys/

And as I said, I have just included "stdint.h" in "uinet_host_interface.h" and stuck with the redefinition error between "net" and "linux". Not sure from where it is picking up both these files.

pkelsey commented 10 years ago

I've cleaned things up a bit for Linux based on what was going on with Ubuntu 12.04. Have another go and let me know how it works out.

petergsnm commented 10 years ago

I was able to compile. Many thanks. I will test now and update you on the results.

petergsnm commented 10 years ago

getting a coredump for all the sample programs while running in ubuntu.

peter:~/src/libuinet/bin/echo++$ ./echo++ -i eth0 -l 192.168.4.107 -p 8090 uinet starting: cpus=1, nmbclusters=131072 callwheelsize=524288 callwheelsize=524288 Creating uma keg headers zone and keg. Calculated uma_max_ipers (for OFFPAGE) is 64 Calculated uma_max_ipers_slab (for OFFPAGE) is 64 UMA: UMA Kegs(0x72e500) size 208(224) flags 536870912 ipers 17 ppera 1 out 0 free 0 Filling boot free list. Creating uma zone headers zone and keg. INTERNAL: Allocating one item from UMA Kegs(0x72e500) slab_zalloc: Allocating a new slab for UMA Kegs UMA: UMA Zones(0x72e680) size 512(512) flags 536870912 ipers 7 ppera 1 out 0 free 0 Initializing pcpu cache locks. Creating slab and hash zones. INTERNAL: Allocating one item from UMA Zones(0x72e680) slab_zalloc: Allocating a new slab for UMA Zones INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: UMA Slabs(0x1e223f0) size 568(568) flags 536870912 ipers 7 ppera 1 out 0 free 0 INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: UMA RCntSlabs(0x1e225f0) size 568(568) flags 536870912 ipers 7 ppera 1 out 0 free 0 INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: UMA Hash(0x1e227f0) size 256(256) flags 536870912 ipers 15 ppera 1 out 0 free 0 INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: 16 Bucket(0x1e229f0) size 152(152) flags 570425344 ipers 25 ppera 1 out 0 free 0 INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: 32 Bucket(0x1e22bf0) size 280(280) flags 570425344 ipers 14 ppera 1 out 0 free 0 INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: 64 Bucket(0x1e22df0) size 536(536) flags 570425344 ipers 7 ppera 1 out 0 free 0 INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: 128 Bucket(0x1e22ff0) size 1048(1048) flags 570425344 ipers 3 ppera 1 out 0 free 0 UMA startup complete. UMA startup2 complete. INTERNAL: Allocating one item from UMA Zones(0x72e680) slab_zalloc: Allocating a new slab for UMA Zones INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: Files(0x1e213f0) size 80(80) flags 32 ipers 45 ppera 1 out 0 free 0 link_elf_lookup_symbol: missing symbol hash table link_elf_lookup_symbol: missing symbol hash table UINET multiprocessor subsystem configured with 1 CPUs Starting callout. UMA startup3 complete. INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: mbuf(0x1e215f0) size 256(256) flags 2048 ipers 15 ppera 1 out 0 free 0 INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA decided we need offpage slab headers for keg: mbuf_cluster, calculated wastedspace = 1984, maximum wasted space allowed = 409, calculated ipers = 2, new wasted space = 0 UMA: mbuf_cluster(0x1e217f0) size 2048(2048) flags 9224 ipers 2 ppera 1 out 0 free 0 INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: mbuf_jumbo_page(0x1e21bf0) size 4096(4096) flags 9224 ipers 1 ppera 1 out 0 free 0 INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: mbuf_jumbo_9k(0x1e21df0) size 9216(9216) flags 9224 ipers 1 ppera 3 out 0 free 0 INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: mbuf_jumbo_16k(0x1e21ff0) size 16384(16384) flags 9224 ipers 1 ppera 4 out 0 free 0 INTERNAL: Allocating one item from UMA Zones(0x72e680) slab_zalloc: Allocating a new slab for UMA Zones INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: mbuf_ext_refcnt(0x1e203f0) size 4(16) flags 2 ipers 168 ppera 1 out 0 free 0 Warning: Timer interrupt thread will not run at real-time priority. Timecounters tick every 10.000 msec Allocating one item from mbuf(0x1e215f0) INTERNAL: Allocating one item from 128 Bucket(0x1e22ff0) slab_zalloc: Allocating a new slab for 128 Bucket slab_zalloc: Allocating a new slab for mbuf Segmentation fault (core dumped)

stack trace: Program terminated with signal 11, Segmentation fault.

0 0x00000000004b82cb in keg_alloc_slab (keg=0x11adbd0, zone=0x11ab5f0, wait=257) at /home/peter/src/libuinet/lib/libuinet/../../sys/vm/uma_core.c:857

857 slab = (uma_slab_t )(mem + keg->uk_pgoff); (gdb) info bt Undefined info command: "bt ". Try "help info". (gdb) bt info No symbol "info" in current context. (gdb) bt

0 0x00000000004b82cb in keg_alloc_slab (keg=0x11adbd0, zone=0x11ab5f0, wait=257) at /home/peter/src/libuinet/lib/libuinet/../../sys/vm/uma_core.c:857

1 0x00000000004ba695 in uma_zalloc_arg (zone=0x1004bb04b, udata=0x7fff33d2a320, flags=0) at /home/peter/src/libuinet/lib/libuinet/../../sys/vm/uma_core.c:2108

2 0x00000000004ba84a in keg_fetch_slab (keg=0x11ab5f0, zone=0x11adbd0, flags=1) at /home/peter/src/libuinet/lib/libuinet/../../sys/vm/uma_core.c:2164

3 0x00000000004bad88 in slab_alloc_item (zone=0x1, slab=0x0) at /home/peter/src/libuinet/lib/libuinet/../../sys/vm/uma_core.c:2351

4 0x00000000004ba478 in uma_zalloc_arg (zone=0x11ab5f0, udata=0x7fff33d2a420, flags=1) at /home/peter/src/libuinet/lib/libuinet/../../sys/vm/uma_core.c:2061

5 0x0000000000475467 in m_get (how=1, type=1) at /home/peter/src/libuinet/lib/libuinet/../../sys/sys/mbuf.h:521

6 0x0000000000475f09 in igmp_ra_alloc () at /home/peter/src/libuinet/lib/libuinet/../../sys/netinet/igmp.c:527

7 0x000000000047a2ae in igmp_init (unused=0x0) at /home/peter/src/libuinet/lib/libuinet/../../sys/netinet/igmp.c:3589

8 0x000000000040c849 in mi_startup () at uinet_init_main.c:300

9 0x000000000040c3a3 in uinet_init (ncpus=1, nmbclusters=131072, loopback=0) at uinet_init.c:141

10 0x0000000000405b0b in main (argc=0, argv=0x7fff33d2b8a8) at echo++.cc:457

(gdb) f 0

0 0x00000000004b82cb in keg_alloc_slab (keg=0x11adbd0, zone=0x11ab5f0, wait=257) at /home/peter/src/libuinet/lib/libuinet/../../sys/vm/uma_core.c:857

857 slab = (uma_slab_t )(mem + keg->uk_pgoff); (gdb) p slab $1 = (uma_slab_t) 0xf4f (gdb) f *slab Cannot access memory at address 0xf4f (gdb) bt in

petergsnm commented 10 years ago

The above issue was because mmap was failing. Looks like we can't call mmap with MAP_ANONYMOUS using fd -1. Once I changed the fd to fd = open("/dev/zero", O_RDWR); and flag to MAP_SHARED. There is no crash. But, I am not able to connect to the example server.

petergsnm commented 10 years ago

although there is no crash observed but I am not able to connect to the example server. I see the below failures while running echo++.

INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: in_l2info(0x2250510) size 148(152) flags 0 ipers 25 ppera 1 out 0 free 0 INTERNAL: Allocating one item from UMA Zones(0x72e680) INTERNAL: Allocating one item from UMA Kegs(0x72e500) UMA: selfd(0x2250710) size 56(56) flags 0 ipers 63 ppera 1 out 0 free 0 configstr is eth0 ethtool get flags failed ethtool get flags failed ethtool get flags failed ethtool get flags failed ethtool discrete get 0x00000014 failed (95) ethtool discrete get 0x00000016 failed (95) ethtool discrete get 0x0000001e failed (95) ethtool discrete get 0x00000021 failed (95) get interface flags failed: Operation not supported Failed to register netmap interface Failed to create interface eth0 (6) Allocating one item from socket(0x2254710) INTERNAL: Allocating one item from 16 Bucket(0x2256910) slab_zalloc: Allocating a new slab for 16 Bucket slab_zalloc: Allocating

command and eth0 interface: /src/libuinet/bin/echo++# ifconfig eth0 eth0 Link encap:Ethernet HWaddr 28:d2:44:1c:ee:1b
inet addr:192.168.4.107 Bcast:192.168.4.255 Mask:255.255.255.0 inet6 addr: fe80::2ad2:44ff:fe1c:ee1b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:490765 errors:0 dropped:0 overruns:0 frame:0 TX packets:423178 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:491211424 (491.2 MB) TX bytes:54449893 (54.4 MB) Interrupt:19

command i used: /src/libuinet/bin/echo++# ./echo++ -i eth0 -l 192.168.4.107 -p 8090

i see you have already linux specific code for this in function if_netmap_ethtool_set_flag(). is there anything I am missing

petergsnm commented 10 years ago

Hi,

At present I am stuck with the two issues, I need your inputs on addressing them.

  1. To solve the above mentioned problems, I commented out the call to "if_netmap_set_offload/promisc" in function "if_netmap_register_if". With this the above mentioned error messages are gone. Please see if you could provide a solution to this.
  2. After making the above changes, I am not able to connect from another host. As I have described above, I am running the "echo" application on the eth0. Now, when I start the echo server, I am not able to connect from a remote host. I can't ping from the remote host in this kind of setup. It seems I am making something wrong in my setup. Can you please let me know to set this up such that I can access from a remote host. (Please refer to my earlier comment on how I have setup my interface and running the command.

Please let me know.

Thanks....Peter.

petergsnm commented 10 years ago

Hi, Now, I am able to connect. I was debugging something and mistakenly have enabled promiscuous mode in that interface. Once I removed that I was able to connect from a client. Sorry for the wrong alarm. So, issue #2 mentioned in the above comment is done. Please see if you could help in resolving issue #1.

Many thanks....

~Peter

pkelsey commented 10 years ago

The actual mmap() issue was that neither MAP_SHARED nor MAP_PRIVATE was being specified in the call to uhi_mmap() in uinet_vm_kern.c, and one of them is required, at least on this system. The use of an fd of -1 for anonymous mappings is fine (the fd is ignored on this system), and it is actually suggested for portability by the mmap man page on this system. The fix for this has been comitted.

pkelsey commented 10 years ago

The fix for the ethtool issue has been comitted.

petergsnm commented 10 years ago

Thanks for the fix. I picked up your latest changes. I don't see the error messages. But, once I start the echo program, I am not able to ping to that machine. I looked at the interface and I see that once I start the program, it changes to "PROMISC" mode. Please see the outputs below -

Before starting the echo program !!!

peter$ ifconfig eth0 eth0 Link encap:Ethernet HWaddr 28:d2:44:1c:f1:52
inet addr:169.254.8.13 Bcast:169.254.255.255 Mask:255.255.0.0 inet6 addr: fe80::2ad2:44ff:fe1c:f152/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4076833 errors:0 dropped:0 overruns:0 frame:0 TX packets:4449493 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:253671890 (253.6 MB) TX bytes:320653882 (320.6 MB) Interrupt:19

sudo ./echo -i eth0 -l 169.254.8.13 -p 8090 uinet starting: cpus=1, nmbclusters=131072 callwheelsize=524288 callwheelsize=524288 link_elf_lookup_symbol: missing symbol hash table link_elf_lookup_symbol: missing symbol hash table UINET multiprocessor subsystem configured with 1 CPUs Timecounters tick every 10.000 msec

configstr is eth0 netmap0: Ethernet address: 28:d2:44:1c:f1:52

"tshark" capture: to show that ARP is being received but no reply being sent

37.160342 28:d2:44:1c:ee:1b -> Broadcast ARP 60 Who has 169.254.8.13? Tell 169.254.6.238 38.177550 28:d2:44:1c:ee:1b -> Broadcast ARP 60 Who has 169.254.8.13? Tell 169.254.6.238 39.176380 28:d2:44:1c:ee:1b -> Broadcast ARP 60 Who has 169.254.8.13? Tell 169.254.6.238 40.176325 28:d2:44:1c:ee:1b -> Broadcast ARP 60 Who has 169.254.8.13? Tell 169.254.6.238

After the echo program !!!

peter$ ifconfig eth0 eth0 Link encap:Ethernet HWaddr 28:d2:44:1c:f1:52
inet addr:169.254.8.13 Bcast:169.254.255.255 Mask:255.255.0.0 inet6 addr: fe80::2ad2:44ff:fe1c:f152/64 Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:4077122 errors:0 dropped:0 overruns:0 frame:0 TX packets:4449676 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:253708982 (253.7 MB) TX bytes:320676356 (320.6 MB) Interrupt:19

Some more info:

Breakpoint 1, ether_input_internal (ifp=0x79bf90, m=0x7ffff7fee600) at /home/peter/libuinet/lib/libuinet/../../sys/net/if_ethersubr.c:636 636 if ((ifp->if_flags & IFF_UP) == 0) { (gdb) (gdb) n 651 if ((m->m_flags & M_PKTHDR) == 0) { (gdb) p ifp->if_flags $1 = 8554755 (gdb) p /x ifp->if_flags $2 = 0x828903 (gdb) p /x ifp->if_carp $3 = 0x0 (gdb)

It comes to the function " " to resolve the ARP and goes to "drop". if (!bridged || (ia = TAILQ_FIRST(&V_in_ifaddrhead)) == NULL) { IN_IFADDR_RUNLOCK(); goto drop; } I expected to call "match" from the above code and should not be going to the drop

gdb) n 535 if (ifp->if_type == IFT_BRIDGE) (gdb) n 538 req_len = arphdr_len2(ifp->if_addrlen, sizeof(struct in_addr)); (gdb) n 539 if (m->m_len < req_len && (m = m_pullup(m, req_len)) == NULL) { (gdb) n 544 ah = mtod(m, struct arphdr ); (gdb) n 549 if (ah->ar_pln != sizeof(struct in_addr)) { (gdb) n 555 if (ETHER_IS_MULTICAST(ar_sha(ah))) { (gdb) n 561 op = ntohs(ah->ar_op); (gdb) n 562 (void)memcpy(&isaddr, ar_spa(ah), sizeof (isaddr)); (gdb) p op $1 = 1 (gdb) n 563 (void)memcpy(&itaddr, ar_tpa(ah), sizeof (itaddr)); (gdb) n 565 if (op == ARPOP_REPLY) (gdb) n 577 IN_IFADDR_RLOCK(); (gdb) n 578 LIST_FOREACH(ia, INADDR_HASH(itaddr.s_addr), ia_hash) { (gdb) n 595 LIST_FOREACH(ia, INADDR_HASH(isaddr.s_addr), ia_hash) (gdb) p bridged $2 = 0 (gdb) n 614 if (is_bridge) { (gdb) n 625 IN_IFADDR_RUNLOCK(); (gdb) p ifa->ifa_addr->sa_family $3 = 224 '\340' (gdb) p /x ifa->ifa_addr->sa_family $4 = 0xe0 (gdb) n 631 IF_ADDR_RLOCK(ifp); (gdb) n 632 TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) (gdb) n 633 if (ifa->ifa_addr->sa_family == AF_INET) { (gdb) p /x ifa->ifa_addr->sa_family $5 = 0x12 (gdb) p /x ifa->ifa_addr $6 = 0x79c688 (gdb) p /x ifa->ifa_addr $7 = {sa_len = 0x38, sa_family = 0x12, sa_data = {0x2, 0x0, 0x6, 0x7, 0x6, 0x0, 0x6e, 0x65, 0x74, 0x6d, 0x61, 0x70, 0x30, 0x28}} (gdb) n 632 TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) (gdb) n 639 IF_ADDR_RUNLOCK(ifp); (gdb) p /x *ifa Cannot access memory at address 0x0 (gdb) c Continuing.

Breakpoint 1, in_arpinput (m=0x7ffff7fee500) at /home/peter/libuinet/lib/libuinet/../../sys/netinet/if_ether.c:514 514 { (gdb) b 614 Breakpoint 2 at 0x473320: file /home/peter/libuinet/lib/libuinet/../../sys/netinet/if_ether.c, line 614. (gdb) c Continuing.

Breakpoint 2, in_arpinput (m=0x7ffff7fee500) at /home/peter/libuinet/lib/libuinet/../../sys/netinet/if_ether.c:614 614 if (is_bridge) { (gdb) n 625 IN_IFADDR_RUNLOCK(); (gdb) p *ifp $8 = {if_softc = 0x799df0, if_l2com = 0x79c470, if_vnet = 0x0, if_link = {tqe_next = 0x0, tqe_prev = 0x796a88}, if_xname = "netmap0\000\000\000\000\000\000\000\000", if_dname = 0x799db0 "netmap0", if_dunit = -1, if_refcount = 1, if_addrhead = {tqh_first = 0x79c560, tqh_last = 0x79c638}, if_pcount = 0, if_carp = 0x0, if_bpf = 0x715c20, if_index = 2, if_index_reserved = 0, if_vlantrunk = 0x0, if_flags = 8554755, if_capabilities = 8388608, if_capenable = 8388608, if_linkmib = 0x0, if_linkmiblen = 0, if_data = {ifi_type = 6 '\006', ifi_physical = 0 '\000', ifi_addrlen = 6 '\006', ifi_hdrlen = 14 '\016', ifi_link_state = 0 '\000', ifi_spare_char1 = 0 '\000', ifi_spare_char2 = 0 '\000', ifi_datalen = 184 '\270', ifi_mtu = 1500, ifi_metric = 0, ifi_baudrate = 10000000, ifi_ipackets = 2, ifi_ierrors = 0, ifi_opackets = 0, ifi_oerrors = 0, ifi_collisions = 0, ifi_ibytes = 120, ifi_obytes = 0, ifi_imcasts = 2, ifi_omcasts = 0, ifi_iqdrops = 0, ifi_noproto = 0, ifi_hwassist = 0, ifi_epoch = 1, ifi_lastchange = {tv_sec = 1, tv_usec = 106}, ifi_icopies = 0, ifi_izcopies = 2, ifi_ocopies = 0, ifi_ozcopies = 0}, if_multiaddrs = {tqh_first = 0x0, tqh_last = 0x79c0e8}, if_amcount = 0, if_output = 0x45daf0 , if_input = 0x45f83e , if_start = 0x415c4e , if_ioctl = 0x41631c , if_init = 0x415bff , if_resolvemulti = 0x46012a , if_qflush = 0x459508 , if_transmit = 0x45c246 , if_reassign = 0, if_home_vnet = 0x0, if_addr = 0x79c560, if_llsoftc = 0x0, if_drv_flags = 64, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0, ifq_maxlen = 1024, ifq_drops = 0, ifq_mtx = {lock_object = {lo_name = 0x79bfb8 "netmap0", lo_flags = 16842752, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0x79c4f0}, ifq_drv_head = 0x0, ifq_drv_tail = 0x0, ifq_drv_len = 0, ifq_drv_maxlen = 1024, altq_type = 0, altq_flags = 1, altq_disc = 0x0, altq_ifp = 0x79bf90, altq_enqueue = 0, altq_dequeue = 0, altq_request = 0, altq_clfier = 0x0, altq_classify = 0, altq_tbr = 0x0, altq_cdnr = 0x0}, if_broadcastaddr = 0x4c8ea0 "\377\377\377\377\377\377arpcom", if_bridge = 0x0, if_label = 0x0, if_prefixhead = {tqh_first = 0x0, tqh_last = 0x79c228}, if_afdata = {0x0, 0x0, 0x79c730, 0x0 <repeats 35 times>}, if_afdata_initialized = 2, if_afdata_lock = {lock_object = {lo_name = 0x4c8b09 "if_afdata", lo_flags = 69402624, lo_data = 0, lo_witness = 0x0}, rw_lock = 0x79c4c0}, if_linktask = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0x45922c , ta_context = 0x79bf90}, if_addr_mtx = { lock_object = {lo_name = 0x4c8afd "if_addr_mtx", lo_flags = 16842752, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0x79c490}, if_clones = {le_next = 0x0, le_prev = 0x0}, if_groups = {tqh_first = 0x79c520, tqh_last = 0x79c528}, if_pf_kif = 0x0, if_lagg = 0x0, if_description = 0x0, if_fib = 1, if_alloctype = 6 '\006', if_cspare = "\000\000", if_ispare = {0, 0, 0, 0}, if_pspare = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} (gdb) n 631 IF_ADDR_RLOCK(ifp);

(gdb) p ifp->if_addrhead->tqh_first->ifa_addr $9 = (struct sockaddr ) 0x79c688 (gdb) p /x ifp->if_addrhead->tqh_first->ifa_addr $10 = 0x79c688 (gdb) p * ifp->if_addrhead->tqh_first->ifa_addr $11 = {sa_len = 56 '8', sa_family = 18 '\022', sa_data = "\002\000\006\a\006\000netmap0("} (gdb) p (struct sockaddr_in *)(ifp->if_addrhead->tqh_first->ifa_addr) $12 = {sin_len = 56 '8', sin_family = 18 '\022', sin_port = 2, sin_addr = {s_addr = 395014}, sin_zero = "netmap0("} (gdb) n 632 TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link)

Look like the IP address is not getting added to the interface causing the ARP request getting dropped. Could you please point me what is going wrong. Also the value of sa_family is not AF_INET.

pkelsey commented 10 years ago

It would help if I understood your goal a bit better. Absent such an understanding, I can give you some general information that may be of assistance in understanding what you are observing.

The main purpose of the echo program as it is now is to allow exercise of the promiscuous sockets functionality that I added to the TCP/IP stack in libuinet. 'Promiscuous' here refers to the ability to bind sockets to any VLAN tags, MAC addresses, and IP addresses. When using promiscuous sockets on a given interface, that interface is necessarily put into promiscuous mode so that no L2 filtering is performed by the hardware in order to allow the stack such visibility into and control over L2 and L3 addressing.

Also, when using promiscuous sockets, no IP addresses are assigned to the interface. Not only would doing so present a scaling issue, but more importantly, such an approach doesn't apply generally to how promiscuous sockets can be used. Consider, for example, that with promiscuous sockets, you can listen on 0.0.0.0:0 (that is, listen for connections to any IP address and port) - what address you add to the interface in that case?

The libuinet stack can be used to do regular, that is, non-promiscuous, socket operations (in which case you will have to make additional libuinet calls to add IP addresses to your interfaces), however the echo program is not currently using the stack in that way.

Further, you may want to consider that using the libuinet stack with netmap on a given interface causes libuinet to completely take over that interface - none of the inbound packets on that interface will go anywhere but that libuinet instance while it is running. If eth0 is your primary interface, you probably want to use a separate interface for libuinet experimentation.

Also, if you are going to play with promiscuous listens that can capture connections to addresses in use elsewhere on your network (say, listening on 0.0.0.0:0), you would probably be better off testing on an isolated network. Consider that echo -i eth0 -l 0.0.0.0 -p 0 might see a SYN sent from one machine on your network to another (even in a switched environment, due to MAC address table misses in the switches), and it will respond with a SYN|ACK and try to establish the connection.

Finally, note that in your debugging session, the value of sa_famliy you see in the interface address list is AF_LINK, because the only entry on that list will be the MAC address for the interface.

petergsnm commented 10 years ago

Thank you for the explanation. I must say, you have put a detailed great explanation.

I was trying to see the performance of libuinet in term of PPS and CPS such that if possible I can use with Nginx (of course it will need some modification). But, my first aim is to evaluate how much performance I get using libuinet.

I am still not clear about the promiscuous mode, I am going to study a bit more about this. What I understand from your email is from the client machine I will not be able to ping the echo server. But, I should be able to connect. But, I was thinking how is this working. Before connect, should not the client try to resolve the ARP?

Also, do you have any sample program, where you are using socket operations in non-promiscuous mode? Also, I was not able to find any document on libuinet. If you have some writeup somewhere, can you please point them to me, please.

I know that libuinet stack with netmap on a given interface causes libuinet to completely take over that interface. For which I was configuring multiple interface, one for data and one for management etc...

Thank you once again.

~Peter

pkelsey commented 10 years ago

Thank you for describing your intended use for libuinet.

libuinet is a young project, and as you've noticed there is currently little in the way of documentation. As you may have also noticed, there isn't even a version number nor has anything resembling a release been made. Understanding where libuinet came from may aid you in understanding where it is now.

libuinet was initially created so that it could be used to build transparent proxies, with a specific requirement that it scale beyond 64k simultaneous connections. Transparent proxy functionality (essentially, being able to accept connections to, and initiate connections from, arbitrary {VLAN, IP, port} tuples) required something like promiscuous sockets. The FreeBSD stack was chosen for extending with promiscuous sockets functionality, and it was ported to user space in order to avoid dealing with file descriptors, provide tighter coupling to the applications, have easy multi-instance scaling, allow porting to other operating systems, etc.

When using promiscuous sockets in a transparent proxy application, there isn't one universally right answer on what to do with ARP requests (depending on the goals and configuration of the proxy, you may want to ignore some, reply locally to some, proxy some to somewhere else, etc). In some cases, the proxy is part of a larger system in which ARP traffic is handled elsewhere. The likely future direction for ARP handling when using promiscuous mode sockets is to allow the application to install an ARP handler so it can determine whether to ignore/reply/other each request.

I have updated bin/echo so that it will use non-promiscuous sockets unless the listen address is 0.0.0.0, the listen port is 0, or the the -P option is given. If none of those conditions are met, then the specified listen address will be added to the given interface, a non-promiscuous listen will be performed, and ARP requests for the given listen address should be handled.

As far as libuinet performance is concerned, the initial focus has been on scalability (that is, handling large numbers of connections). Measurement and improvement of time-performance metrics such as throughput, connection rates, etc. is just beginning. Note that as the initial focus has been on correctness of new features and non-time performance metrics, everything is currently being built by default with -O0 to facilitate quick debugging. If you are going to measure throughput and connection rates, you'll want to adjust the compiler flags in the relevant makefiles first.

As far as Nginx integration issues are concerned, I can't comment in detail as I haven't examined the Nginx internals. However, given that Nginx is event driven, the libuinet API is meant for integration into event systems, and the libuinet API has been integrated with two different event systems to date (WANProxy and libev), I would expect there to be a reasonable integration path.

I will be giving an ~hour talk on libuinet at BSDCan next week (https://www.bsdcan.org/2014/schedule/events/447.en.html), and following that the slides should be posted online. If you aren't already planning on being there, the slides should provide a more complete picture of the current state of the project and anticipated future directions.

petergsnm commented 10 years ago

Thanks. It is clear to me now. I will not be able to attend your tech talk, looking forward for your slides after that.

petergsnm commented 10 years ago

I was measuring the performance of libuinet on a KVM VM (which uses virtio) with one core. I performed two tests -

  1. I first ran a simple TCP sever which accepts the connections and close the connections. When I run the sample server, I get around 200k PPS and around 25K Connections Per Seconds.

Next,

  1. I modified my TCP server to make use of libuinet and when I run the program, I got around 40K PPS and 5K CPS.

I am not sure why the performance get reduced when I run with libuinet. I am planning to run callgrind and see where in the libuinet we take time. But, I expected the libuinet will get me better performance.

I can't attach my simple TCP programs here. But, if you can drop me an email at peter.gsnm@gmail.com, where I can share my sample programs.

Other thing is to get the 200K PPS (without libuinet), I made the sysctl changes as follows. Not sure if I need to modify some of your header files for the below mentioned sysctl changes. Can you please point me to where I need to modify.

fs.file-max = 5000000 net.core.netdev_max_backlog = 4000000 net.core.optmem_max = 10000000 net.core.rmem_default = 10000000 net.core.rmem_max = 3000000 net.core.somaxconn = 10000000 net.core.wmem_default = 10000000 net.core.wmem_max = 30000000 net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.default.rp_filter = 1 net.ipv4.ip_local_port_range = 1024 65535 net.ipv4.tcp_congestion_control = bic net.ipv4.tcp_ecn = 0 net.ipv4.tcp_max_syn_backlog = 65000 net.ipv4.tcp_max_tw_buckets = 6000000 net.ipv4.tcp_mem = 30000000 30000000 30000000 net.ipv4.tcp_rmem = 30000000 30000000 30000000 net.ipv4.tcp_sack = 1 net.ipv4.tcp_syncookies = 0 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_wmem = 30000000 30000000 30000000 net.ipv4.tcp_early_retrans = 1

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_tw_recycle = 1

net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syn_retries = 2

net.ipv4.tcp_fin_timeout = 7

net.ipv4.tcp_slow_start_after_idle = 0

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_low_latency = 1

Please let me know what you think.

Thank you.

~Peter

caladri commented 10 years ago

Thank you for the detailed measurements; I don't have any specific suggestions or thoughts at present but wanted to suggest you create a new issue for this specific set of performance issues.

I can't speak for pkelsey, but I think it may be worth creating a new issue on the performance front rather than continuing to deal with it in this one, for ease of tracking and to avoid having to read through a number of different conversations to get to the current one. GitHub makes it very cheap to open and close issues. You wouldn't want the performance concerns to fall by the wayside just because the Ubuntu build is fixed.

pkelsey commented 10 years ago

Yes, please open a new issue for this.

petergsnm commented 10 years ago

Thank you.

I am closing this issue as I am able to build use libuinet on my Ubuntu 12.04 LTS. I am able to test the sample programs provided by pkelsey and was also able to create my sample program on libuinet too.

Many thanks to pkelsey for all his help on this.