search

luigirizzo / netmap

Automatically exported from code.google.com/p/netmap
BSD 2-Clause "Simplified" License
1.85k stars 534 forks source link

Error: Unable to open netmap:eth0: Invalid argument in AWS #566

Closed gsangeryee closed 5 years ago

gsangeryee commented 5 years ago

Hello, I installed netmap in AWS(Ubuntu 16.04) . When I run $ pke-gen -i eth0 , I got these error message:

065.096883 main [2855] interface is eth0
065.097475 main [2977] using default burst size: 512
065.097579 main [2985] running on 1 cpus (have 1)
065.097753 extract_ip_range [467] range is 10.0.0.1:1234 to 10.0.0.1:1234
065.097828 extract_ip_range [467] range is 10.1.0.1:1234 to 10.1.0.1:1234
065.097897 nm_open [855] overriding ARG1 0
065.097932 nm_open [859] overriding ARG2 0
065.097964 nm_open [863] overriding ARG3 0
065.097996 nm_open [867] overriding RING_CFG
065.098027 nm_open [876] overriding ifname eth0 ringid 0x0 flags 0x8001
065.128843 nm_open [947] NIOCREGIF failed: Invalid argument netmap:eth0
065.128903 main [3078] Unable to open netmap:eth0: Invalid argument
065.128937 main [3145] aborting

Is there any special settings when I use the virtual server in AWS, GCP, etc Thanks.

vmaffione commented 5 years ago

Hi, There are no special settings for AWS, GCP, etc.

Does dmesg report any netmap-related error or info? What is the output of ethtool -i eth0? Which commit id are you using (e.g. the SHA sum)?

gsangeryee commented 5 years ago

Hi, vmaffione

Thanks for your response.

dmesg report is below:

[10436109.979019] 058.140528 [1434] netmap_finalize_obj_allocator Unable to create cluster at 50758 for 'netmap_buf' allocator
[10436109.991954] error: netmap buf size (2048) < device MTU (9001)
[10436109.996039] 058.157548 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed
[10436161.811750] 109.973258 [1152] generic_netmap_attach     Emulated adapter for eth0 created (prev was           (null))
[10436161.823533] error: netmap buf size (2048) < device MTU (9001)
[10436161.829355] 109.990864 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed
[10436230.413933] 178.575440 [1152] generic_netmap_attach     Emulated adapter for eth0 created (prev was           (null))
[10436230.420516] error: netmap buf size (2048) < device MTU (9001)
[10436230.424313] 178.585821 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed
[10436262.328398] 210.489906 [1152] generic_netmap_attach     Emulated adapter for eth0 created (prev was           (null))
[10436262.336572] error: netmap buf size (2048) < device MTU (9001)
[10436262.341420] 210.502929 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed
[10436385.638393] 333.799900 [1152] generic_netmap_attach     Emulated adapter for eth0 created (prev was           (null))
[10436385.650322] error: netmap buf size (2048) < device MTU (9001)
[10436385.656712] 333.818220 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed
[10438700.941312] 649.102820 [1152] generic_netmap_attach     Emulated adapter for eth0 created (prev was           (null))
[10438700.948023] error: netmap buf size (2048) < device MTU (9001)
[10438700.951681] 649.113189 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed
[10439774.944433] 723.105941 [1152] generic_netmap_attach     Emulated adapter for eth0 created (prev was           (null))
[10439774.956091] error: netmap buf size (2048) < device MTU (9001)
[10439774.960977] 723.122486 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed
[10440247.462113] 195.623622 [1152] generic_netmap_attach     Emulated adapter for eth0 created (prev was           (null))
[10440247.469943] error: netmap buf size (2048) < device MTU (9001)
[10440247.474561] 195.636070 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed
[10454129.660118] 077.821626 [1152] generic_netmap_attach     Emulated adapter for eth0 created (prev was           (null))
[10454129.671753] error: netmap buf size (2048) < device MTU (9001)
[10454129.676412] 077.837921 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed
[10700716.711310] 664.872818 [1152] generic_netmap_attach     Emulated adapter for eth0 created (prev was           (null))
[10700716.720311] error: netmap buf size (2048) < device MTU (9001)
[10700716.725096] 664.886605 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed
[10709116.936557] 065.098065 [1152] generic_netmap_attach     Emulated adapter for eth0 created (prev was           (null))
[10709116.952371] error: netmap buf size (2048) < device MTU (9001)
[10709116.957419] 065.118928 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed
[10743838.685919] 786.847427 [1152] generic_netmap_attach     Emulated adapter for eth0 created (prev was           (null))
[10743838.698958] error: netmap buf size (2048) < device MTU (9001)
[10743838.703493] 786.865002 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed

$ ethtool -i eth0 output is below:

driver: vif
version:
firmware-version:
expansion-rom-version:
bus-info: vif-0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Which commit id are you using (e.g. the SHA sum)?

I am quite sure about this question. How can I check it?

vmaffione commented 5 years ago

Hi, The dmesg clearly shows what the problem is. You are using the emulated netmap adapter on top of a vif interface (Xen netfront paravirtualized NIC). However, the MTU of your network interface is set to 9000, which does not play well with the netmap buffer size (2048 bytes by default). If emulated adapter supported multi-fragment operation (NS_MOREFRAG), this configuration would be ok. However, since NS_MOREFRAG is not supported yet on the emulated adapter, you need the netmap buffer size to be >= of the MTU. In other words, the simplest solution is to lower your MTU:

 # ip link set dev eth0 mtu 2000

Or you could also modify the netmap buffer size to satistfy the inequality

 # echo ${NUM_BYTES} > /sys/module/netmap/parameters/buf_size

(The commit id does not matter in this case. You can get it with git log in the netmap repo, and looking at the hex number on top, e.g. commit fce88df1b49c0dfce4c6d4211f1d4ec62aa320b0.)

gsangeryee commented 5 years ago

Hi, vmaffione.

Thank you for your explanation.

I use ip link set dev etho mtu 2000 to set MTU.

But when I run $ pkt-gen -i eth0 ,the terminal frozen in these messages.

065.096883 main [2855] interface is eth0
065.097475 main [2977] using default burst size: 512
065.097579 main [2985] running on 1 cpus (have 1)
065.097753 extract_ip_range [467] range is 10.0.0.1:1234 to 10.0.0.1:1234
065.097828 extract_ip_range [467] range is 10.1.0.1:1234 to 10.1.0.1:1234

Several minutes later, I lost the connection with my AWS.

And I got this message in dmesg.

[10799081.538748] 029.700256 [1152] generic_netmap_attach     Emulated adapter for eth0 created (prev was           (null))
[10799081.549296] 029.710805 [ 330] generic_netmap_register   Emulated adapter for eth0 activated
[10799081.556208] 029.717717 [ 375] generic_netmap_register   Emulated adapter: ring 'eth0 RX0' activated
[10799081.563994] 029.725503 [ 382] generic_netmap_register   Emulated adapter: ring 'eth0 TX0' activated
[10799081.571874] 029.733382 [ 834] tc_configure              ifp eth0 qdisc netmapemu parent 4294967295 handle 0
[10799081.928128] 030.089634 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799083.284114] 031.445621 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799085.108127] 033.269633 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799087.023084] 035.184591 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799087.792109] 035.953617 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799088.684139] 036.845645 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799090.464094] 038.625601 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799094.024150] 042.185658 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799095.984132] 044.145638 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799101.152103] 049.313610 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799105.060232] 053.221738 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799108.211003] 056.372511 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799110.496116] 058.657623 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799113.156782] 061.318289 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799115.392110] 063.553618 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10799123.793574] 071.955081 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800009.252766] 957.414274 [ 834] tc_configure              ifp eth0 qdisc pfifo parent 4294967295 handle 0
[10800009.261359] 957.422867 [ 241] generic_netmap_unregister Emulated adapter: ring 'eth0 RX0' deactivated
[10800009.269316] 957.430825 [ 248] generic_netmap_unregister Emulated adapter: ring 'eth0 TX0' deactivated
[10800009.276982] 957.438490 [ 305] generic_netmap_unregister Emulated adapter for eth0 deactivated
[10800009.284341] 957.445850 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed
[10800235.894117] 184.055624 [1152] generic_netmap_attach     Emulated adapter for eth0 created (prev was           (null))
[10800235.901845] 184.063354 [ 330] generic_netmap_register   Emulated adapter for eth0 activated
[10800235.908242] 184.069751 [ 375] generic_netmap_register   Emulated adapter: ring 'eth0 RX0' activated
[10800235.914391] 184.075900 [ 382] generic_netmap_register   Emulated adapter: ring 'eth0 TX0' activated
[10800235.920758] 184.082266 [ 834] tc_configure              ifp eth0 qdisc netmapemu parent 4294967295 handle 0
[10800236.348094] 184.509600 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800236.908131] 185.069637 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800238.032115] 186.193621 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800240.123467] 188.284973 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800240.716123] 188.877629 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800241.922168] 190.083675 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800243.061926] 191.223433 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800244.769814] 192.931321 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800246.192952] 194.354460 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800249.074286] 197.235792 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800252.450088] 200.611594 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800253.761879] 201.923386 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800263.394122] 211.555629 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800264.961163] 213.122671 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800271.713314] 219.874821 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800279.973879] 228.135380 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800280.893254] 229.054762 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800281.805930] 229.967438 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800283.633663] 231.795170 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10800287.289104] 235.450611 [3815] netmap_transmit           eth0 drop mbuf that needs checksum offload
[10801170.917902] 119.079410 [ 834] tc_configure              ifp eth0 qdisc pfifo parent 4294967295 handle 0
[10801170.924192] 119.085700 [ 241] generic_netmap_unregister Emulated adapter: ring 'eth0 RX0' deactivated
[10801170.932095] 119.093604 [ 248] generic_netmap_unregister Emulated adapter: ring 'eth0 TX0' deactivated
[10801170.938515] 119.100024 [ 305] generic_netmap_unregister Emulated adapter for eth0 deactivated
[10801170.944311] 119.105820 [1057] generic_netmap_dtor       Emulated netmap adapter for eth0 destroyed
vmaffione commented 5 years ago

The drop mbuf messages are caused by the NIC offloadings not being disabled. Look at https://github.com/luigirizzo/netmap/blob/master/LINUX/README#L243-L255 to see how to disable them.

Regarding the freeze, if eth0 is the only interface of your VM, that's expected. When you open eth0 in netmap mode (e.g. by running pkt-gen or other netmap applications), eth0 gets "disconnected" from the network stack, so that it can be used by netmap. As a result, network traffic does not flow between eth0 and your SSH server, which means that your ssh session will freeze.

gsangeryee commented 5 years ago

Yes, I have only eth0 interface of my AWS VM. So, How can I check if the netmap installation is correct？ By the way, the propose of installing NETMAP is that I want to set only open port 80s and 22 of the AWS host (Ubuntu 16.04) via NETMAP- IPFW.

vmaffione commented 5 years ago

You can run the unit tests and integration tests if you wish:

$ sudo make unitest
$ sudo make intest

What do you mean by "set only open port 80 and 22"?

gsangeryee commented 5 years ago

It means that I create rules for ipfw to allow only ports 80/tcp and 22/tcp. Like this:

# The rules
$cmd 01000 allow tcp from any to any 80,22
$cmd 01500 allow tcp from any 80,22 to any

My solution is to install netmap and dummynet and implement it by setting the rules for ipfw. But I can't build in Ubuntu 16.04 with kernel 4.4.0 The error is:

loki@ip-172-31-36-20:~/dummynet$ make KERNELPATH=/usr/src/linux-headers-4.4.0-1061-aws
make[1]: Entering directory '/home/loki/dummynet/kipfw-mod'
Makefile:76: ------ arch Linux goals include_e -----------
Makefile:202: ------------- linux version 40483 (hex) ------------
Makefile:264: --- build 2.6 and newer target kipfw ----
Makefile:289:
make[1]: Leaving directory '/home/loki/dummynet/kipfw-mod'
make[1]: Entering directory '/home/loki/dummynet/kipfw-mod'
Makefile:76: ------ arch Linux goals kipfw -----------
Makefile:202: ------------- linux version 40483 (hex) ------------
Makefile:264: --- build 2.6 and newer target kipfw ----
Makefile:289:
echo "xxxxxxxxxxxxx make -C /usr/src/linux-headers-4.4.0-1061-aws V= M=`pwd` modules"
xxxxxxxxxxxxx make -C /usr/src/linux-headers-4.4.0-1061-aws V= M=/home/loki/dummynet/kipfw-mod modules
make -C /usr/src/linux-headers-4.4.0-1061-aws V= M=`pwd` modules
make[2]: Entering directory '/usr/src/linux-headers-4.4.0-1061-aws'
/home/loki/dummynet/kipfw-mod/Makefile:76: ------ arch Linux goals  -----------
/home/loki/dummynet/kipfw-mod/Makefile:202: ------------- linux version 40483 (hex) ------------
/home/loki/dummynet/kipfw-mod/Makefile:264: --- build 2.6 and newer target kipfw ----
/home/loki/dummynet/kipfw-mod/Makefile:289:
  CC [M]  /home/loki/dummynet/kipfw-mod/ip_fw2.o
  CC [M]  /home/loki/dummynet/kipfw-mod/ip_fw_pfil.o
  CC [M]  /home/loki/dummynet/kipfw-mod/ip_fw_sockopt.o
  CC [M]  /home/loki/dummynet/kipfw-mod/ip_fw_dynamic.o
  CC [M]  /home/loki/dummynet/kipfw-mod/ip_fw_table.o
  CC [M]  /home/loki/dummynet/kipfw-mod/ip_fw_log.o
  CC [M]  /home/loki/dummynet/kipfw-mod/radix.o
  CC [M]  /home/loki/dummynet/kipfw-mod/in_cksum.o
  CC [M]  /home/loki/dummynet/kipfw-mod/ip_dummynet.o
  CC [M]  /home/loki/dummynet/kipfw-mod/ip_dn_io.o
  CC [M]  /home/loki/dummynet/kipfw-mod/ip_dn_glue.o
  CC [M]  /home/loki/dummynet/kipfw-mod/dn_heap.o
  CC [M]  /home/loki/dummynet/kipfw-mod/dn_sched_fifo.o
  CC [M]  /home/loki/dummynet/kipfw-mod/dn_sched_wf2q.o
  CC [M]  /home/loki/dummynet/kipfw-mod/dn_sched_rr.o
  CC [M]  /home/loki/dummynet/kipfw-mod/dn_sched_qfq.o
  CC [M]  /home/loki/dummynet/kipfw-mod/dn_sched_prio.o
  CC [M]  /home/loki/dummynet/kipfw-mod/ipfw2_mod.o
In file included from /home/loki/dummynet/kipfw-mod/ipfw2_mod.c:100:0:
include/net/inet_hashtables.h: In function 'sk_daddr_set':
include/net/inet_hashtables.h:354:2: error: implicit declaration of function 'ipv6_addr_set_v4mapped' [-Werror=implicit-function-declaration]
  ipv6_addr_set_v4mapped(addr, &sk->sk_v6_daddr);
  ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c: In function 'ip_output':
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:362:15: error: passing argument 1 of 'dst_output' from incompatible pointer type [-Werror=incompatible-pointer-types]
    dst_output(skb);
               ^
In file included from include/net/sock.h:67:0,
                 from include/linux/tcp.h:22,
                 from include/linux/ipv6.h:72,
                 from include/net/netfilter/nf_queue.h:5,
                 from /home/loki/dummynet/kipfw-mod/ipfw2_mod.c:71:
include/net/dst.h:493:19: note: expected 'struct net *' but argument is of type 'struct sk_buff *'
 static inline int dst_output(struct net *net, struct sock *sk, struct sk_buff *skb)
                   ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:362:4: error: too few arguments to function 'dst_output'
    dst_output(skb);
    ^
In file included from include/net/sock.h:67:0,
                 from include/linux/tcp.h:22,
                 from include/linux/ipv6.h:72,
                 from include/net/netfilter/nf_queue.h:5,
                 from /home/loki/dummynet/kipfw-mod/ipfw2_mod.c:71:
include/net/dst.h:493:19: note: declared here
 static inline int dst_output(struct net *net, struct sock *sk, struct sk_buff *skb)
                   ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c: In function 'ipfw2_queue_handler':
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:559:26: error: 'struct nf_queue_entry' has no member named 'indev'
  m->m_pkthdr.rcvif = info->indev;
                          ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:568:10: error: 'struct nf_queue_entry' has no member named 'hook'
  if (info->hook == IPFW_HOOK_IN) {
          ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:569:39: error: 'struct nf_queue_entry' has no member named 'indev'
   ret = ipfw_check_hook(NULL, &m, info->indev, PFIL_IN, NULL);
                                       ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:571:39: error: 'struct nf_queue_entry' has no member named 'outdev'
   ret = ipfw_check_hook(NULL, &m, info->outdev, PFIL_OUT, NULL);
                                       ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c: At top level:
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:827:35: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
                 .hook           = call_ipfw,
                                   ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:827:35: note: (near initialization for 'ipfw_ops[0].hook')
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:831:17: error: unknown field 'owner' specified in initializer
                 SET_MOD_OWNER
                 ^
In file included from include/linux/linkage.h:6:0,
                 from include/linux/kernel.h:6,
                 from include/linux/list.h:8,
                 from include/linux/msg.h:4,
                 from /home/loki/dummynet/kipfw-mod/../glue.h:101,
                 from <command-line>:0:
include/linux/export.h:34:21: error: excess elements in struct initializer [-Werror]
 #define THIS_MODULE (&__this_module)
                     ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:821:32: note: in expansion of macro 'THIS_MODULE'
 #define SET_MOD_OWNER .owner = THIS_MODULE,
                                ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:831:17: note: in expansion of macro 'SET_MOD_OWNER'
                 SET_MOD_OWNER
                 ^
include/linux/export.h:34:21: note: (near initialization for 'ipfw_ops[0]')
 #define THIS_MODULE (&__this_module)
                     ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:821:32: note: in expansion of macro 'THIS_MODULE'
 #define SET_MOD_OWNER .owner = THIS_MODULE,
                                ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:831:17: note: in expansion of macro 'SET_MOD_OWNER'
                 SET_MOD_OWNER
                 ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:834:35: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
                 .hook           = call_ipfw,
                                   ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:834:35: note: (near initialization for 'ipfw_ops[1].hook')
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:838:3: error: unknown field 'owner' specified in initializer
   SET_MOD_OWNER
   ^
In file included from include/linux/linkage.h:6:0,
                 from include/linux/kernel.h:6,
                 from include/linux/list.h:8,
                 from include/linux/msg.h:4,
                 from /home/loki/dummynet/kipfw-mod/../glue.h:101,
                 from <command-line>:0:
include/linux/export.h:34:21: error: excess elements in struct initializer [-Werror]
 #define THIS_MODULE (&__this_module)
                     ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:821:32: note: in expansion of macro 'THIS_MODULE'
 #define SET_MOD_OWNER .owner = THIS_MODULE,
                                ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:838:3: note: in expansion of macro 'SET_MOD_OWNER'
   SET_MOD_OWNER
   ^
include/linux/export.h:34:21: note: (near initialization for 'ipfw_ops[1]')
 #define THIS_MODULE (&__this_module)
                     ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:821:32: note: in expansion of macro 'THIS_MODULE'
 #define SET_MOD_OWNER .owner = THIS_MODULE,
                                ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:838:3: note: in expansion of macro 'SET_MOD_OWNER'
   SET_MOD_OWNER
   ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c: In function 'ipfw_module_init':
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:780:28: error: passing argument 1 of 'nf_register_queue_handler' from incompatible pointer type [-Werror=incompatible-pointer-types]
 #define REG_QH_ARG(pf, fn) &(fn ## _desc)
                            ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:904:32: note: in expansion of macro 'REG_QH_ARG'
      nf_register_queue_handler(REG_QH_ARG(PF_INET, ipfw2_queue_handler) );
                                ^
In file included from /home/loki/dummynet/kipfw-mod/ipfw2_mod.c:71:0:
include/net/netfilter/nf_queue.h:31:6: note: expected 'struct net *' but argument is of type 'struct nf_queue_handler *'
 void nf_register_queue_handler(struct net *net, const struct nf_queue_handler *qh);
      ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:904:6: error: too few arguments to function 'nf_register_queue_handler'
      nf_register_queue_handler(REG_QH_ARG(PF_INET, ipfw2_queue_handler) );
      ^
In file included from /home/loki/dummynet/kipfw-mod/ipfw2_mod.c:71:0:
include/net/netfilter/nf_queue.h:31:6: note: declared here
 void nf_register_queue_handler(struct net *net, const struct nf_queue_handler *qh);
      ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:918:2: error: too few arguments to function 'nf_unregister_queue_handler'
  nf_unregister_queue_handler(UNREG_QH_ARG(PF_INET, ipfw2_queue_handler) );
  ^
In file included from /home/loki/dummynet/kipfw-mod/ipfw2_mod.c:71:0:
include/net/netfilter/nf_queue.h:32:6: note: declared here
 void nf_unregister_queue_handler(struct net *net);
      ^
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c: In function 'ipfw_module_exit':
/home/loki/dummynet/kipfw-mod/ipfw2_mod.c:942:2: error: too few arguments to function 'nf_unregister_queue_handler'
  nf_unregister_queue_handler(UNREG_QH_ARG(PF_INET, ipfw2_queue_handler) );
  ^
In file included from /home/loki/dummynet/kipfw-mod/ipfw2_mod.c:71:0:
include/net/netfilter/nf_queue.h:32:6: note: declared here
 void nf_unregister_queue_handler(struct net *net);
      ^
cc1: all warnings being treated as errors
scripts/Makefile.build:277: recipe for target '/home/loki/dummynet/kipfw-mod/ipfw2_mod.o' failed
make[3]: *** [/home/loki/dummynet/kipfw-mod/ipfw2_mod.o] Error 1
Makefile:1437: recipe for target '_module_/home/loki/dummynet/kipfw-mod' failed
make[2]: *** [_module_/home/loki/dummynet/kipfw-mod] Error 2
make[2]: Leaving directory '/usr/src/linux-headers-4.4.0-1061-aws'
Makefile:310: recipe for target 'kipfw' failed
make[1]: *** [kipfw] Error 2
make[1]: Leaving directory '/home/loki/dummynet/kipfw-mod'
Makefile:110: recipe for target 'kipfw' failed
make: *** [kipfw] Error 2
vmaffione commented 5 years ago

But netmap and dummynet are unrelated. Btw you get those errors because nobody is actively maintaining dummynet (since 2015), so it does not build on newer kernels.

The good news is that you can implement such a simple filter very efficiently with a simple netmap application, starting from the bridge example application, and adding some logic to filter out the unwanted packets by looking inside the packet headers. I would recommend looking at our tutorial to learn how to write simple netmap applications https://github.com/netmap-unipi/netmap-tutorial

gsangeryee commented 5 years ago

I found this repo https://github.com/luigirizzo/netmap-ipfw. Can this repo fulfilll my requirements?

vmaffione commented 5 years ago

I think so. You can probably use something like

./kipfw netmap:eth0 netmap:eth0^

because you want to apply the filter between the eth0 NIC and the network stack.

gsangeryee commented 5 years ago

Thank you for you advice.

  1. For this command kipfw, where can we look for more detail information about how to use it?
  2. Because I used an AWS server and used SSH (port 22 ) to log in the server, can I test the commands of netmap-ipfw on eth0 ? I run the command $ pkt-gen -i eth0 on previous test, I lost my connection with my server for a while.
  3. I run the example code below. It looks like work. But I don't know where the ipfw rule files save, is it a dynamic rule？Will it be lost, when I reboot the server.
# connect the firewall to two vale switches
    ./kipfw valeA:f valeB:f &

    # configure ipfw/dummynet
    ipfw/ipfw show  # or other

    # start the sink
    pkt-gen -i valeB:d -f rx

    # start an infinite source
    pkt-gen -i valeA:s -f tx

    # plain again with the firewall and enjoy
    ipfw/ipfw show  # or other
vmaffione commented 5 years ago

The kipfw program is experimental, so there is no documentation in addition to the README. I explained above the reason why pkt-gen -i eth0 cause the termination/freeze of your SSH connection: nothing is passing to/from the network stack. If you run ./kipfw netmap:eth0 netmap:eth0^, you are creating a firewall between the eth0 NIC rings (netmap:eth0') and the eth0 interface visible from the network stack (netmap:eth0^'). The default ruleset is "pass any to any", as you can see running ipfw/ipfw show. You can change the ruleset using the ipfw/ipfw command: you can find the documentation in the FreeBSD man page ipfw(8) (https://www.freebsd.org/cgi/man.cgi?ipfw(8)).

(Closing this as there is no issue, but more questions are welcome).

vmaffione commented 5 years ago

I don't think the ruleset is persistent; as I said, kipfw is experimental. You can always arrange a startup script that builds the rules you need.

Another option you have is to modify this program https://github.com/netmap-unipi/netmap-tutorial/blob/master/solutions/forward.c#L53-L81 (part of the netmap tutorial). If you modify the pkt_select function to match TCP (instead of UDP) and ports 20 or 80 (instead of the port contained in udp_port), you will get the functionality you are looking for (./forward -i netmap:eth0 -i netmap:eth0^).

gsangeryee commented 5 years ago

Very Helpful !

gsangeryee commented 5 years ago

I try to run ./kipfw netmap:eth0 netmap:eth0^ on my server. I can see the default ruleset is 65535 0 0 allow ip from any to any. And I also added ruleset allow tcp from any to any. But I still can't use SSH client to connect this server. The error is ssh_exchange_identification: read: Connection reset by peer

vmaffione commented 5 years ago

Maybe rules for letting ARP are missing? IOW, can you ping the VM from outside?

vmaffione commented 5 years ago

I would strongly recommend to first replace ./kipfw with bridge (from apps/bridge)

# ./bridge netmap:eth0 netmap:eth0^

and check that this works, since bridge forwards all the packets (like a null firewall). Then once this works (including SSH), switch back to ./kipfw.

gsangeryee commented 5 years ago
  1. Maybe rules for letting ARP are missing? IOW, can you ping the VM from outside? Answer: I can ping the VM from outside.

  2. bridge seems functional. The below is my testing.

    # ./bridge netmap:eth0 netmap:eth0^
bridge built Dec  1 2018 22:24:48

596.188953 nm_mmap [986] do not mmap, inherit from parent
596.189018 main [268] ------- zerocopy supported
596.189062 main [275] Wait 4 secs for link to come up...
600.189751 main [279] Ready to go, enp0s3 0x0/1 <-> enp0s3 0x1/1.
602.695614 main [330] poll timeout [0] ev 1 0 rx 0@13 tx 255, [1] ev 1 0 rx 0@4 tx 255
605.201045 main [330] poll timeout [0] ev 1 0 rx 0@15 tx 255, [1] ev 1 0 rx 0@6 tx 255
607.706543 main [330] poll timeout [0] ev 1 0 rx 0@17 tx 255, [1] ev 1 0 rx 0@8 tx 255
612.251199 main [330] poll timeout [0] ev 1 0 rx 0@25 tx 255, [1] ev 1 0 rx 0@10 tx 255
614.755088 main [330] poll timeout [0] ev 1 0 rx 0@27 tx 255, [1] ev 1 0 rx 0@12 tx 255
617.260523 main [330] poll timeout [0] ev 1 0 rx 0@29 tx 255, [1] ev 1 0 rx 0@14 tx 255  
621.991733 main [330] poll timeout [0] ev 1 0 rx 0@49 tx 255, [1] ev 1 0 rx 0@27 tx 255
... ...

    In other terminal window:

    # sudo ./kipfw netmap:eth0 netmap:eth0^`
[ 467.277327] missing.c:main       [730] initializing tick to 200
[ 467.277569] missing.c:callout_startup [365] start
init_children mod_idx value 9
+++ start module 0 ipfw ipfw at 0x631240 order 0x1
+++ start module 1 sy_ipfw SYSINIT at (nil) order 0x2
ipfw2 initialized, divert loadable, nat loadable, default to accept, logging disabled
+++ start module 2 sy_Vnet_ipfw SYSINIT at (nil) order 0x3
[ 467.280363] missing.c:callout_init [312] c 0x6321e0 mpsafe 8
[ 467.281439] missing.c:pfil_head_get [89] called
[ 467.282272] missing.c:pfil_add_hook [96] called
+++ start module 3 dummynet dummynet at 0x631510 order 0x4
DUMMYNET (nil) with IPv6 initialized (100409)
[ 467.283936] missing.c:taskqueue_create_fast [431] start dummynet fn 0x422420 ctx 0x6322c0
[ 467.284334] missing.c:taskqueue_start_threads [439] tqp 0x6322c0 count 1 (dummy)
[ 467.284833] missing.c:callout_init [312] c 0x6322e0 mpsafe 8
+++ start module 4 dn_fifo dn_fifo at 0x631ad0 order 0x5
[ 467.285386] ip_dummynet.c:load_dn_sched [2270] dn_sched FIFO loaded
+++ start module 5 dn_wf2qp dn_wf2qp at 0x631bb0 order 0x6
[ 467.285606] ip_dummynet.c:load_dn_sched [2270] dn_sched WF2Q+ loaded
+++ start module 6 dn_rr dn_rr at 0x631c90 order 0x7
[ 467.286092] ip_dummynet.c:load_dn_sched [2270] dn_sched RR loaded
+++ start module 7 dn_qfq dn_qfq at 0x631d70 order 0x8
[ 467.286404] ip_dummynet.c:load_dn_sched [2270] dn_sched QFQ loaded
+++ start module 8 dn_prio dn_prio at 0x631e50 order 0x9
[ 467.286620] ip_dummynet.c:load_dn_sched [2270] dn_sched PRIO loaded
*** Global Sysctl Table entries = 45, total size = 2364 ***
[ 467.286754] session.c:do_server  [557] +++ listening tcp 127.0.0.1:5555
[ 467.286804] netmap_io.c:netmap_add_port [328] opening netmap device netmap:enp0s3
[ 467.286908] netmap_io.c:netmap_add_port [344] --- mem_id 1
[ 467.287097] netmap_io.c:netmap_add_port [347] create sess 0x17deb60 my_netmap_port 0x17e75f0
[ 467.287147] netmap_io.c:netmap_add_port [328] opening netmap device netmap:enp0s3^
[ 467.287214] netmap_io.c:netmap_add_port [344] --- mem_id 1
[ 467.287381] netmap_io.c:netmap_add_port [347] create sess 0x17deba0 my_netmap_port 0x17e7b40
[ 467.287521] netmap_io.c:netmap_add_port [360] 0x17e7b40 enp0s3 1 <-> 0x17e75f0 enp0s3 1 SWAP
[ 467.287600] session.c:mainloop   [640] callouts 1 skipped 0
[ 468.000406] session.c:mainloop   [640] callouts 1654 skipped 1
[ 469.000273] session.c:mainloop   [640] callouts 3942 skipped 1
[ 470.000428] session.c:mainloop   [640] callouts 6311 skipped 1
[ 471.000241] session.c:mainloop   [640] callouts 8780 skipped 1
[ 472.000002] session.c:mainloop   [640] callouts 11546 skipped 1
[ 473.000024] session.c:mainloop   [640] callouts 14301 skipped 1
[ 474.000395] session.c:mainloop   [640] callouts 16807 skipped 1
[ 475.000337] session.c:mainloop   [640] callouts 19369 skipped 1
[ 476.000112] session.c:mainloop   [640] callouts 21845 skipped 1
... ...

    Open another terminal window.

    $ sudo ipfw/ipfw show
65535 40 5380 allow ip from any to any

    Add some rules, like deny port 22.

    $ sudo ipfw/ipfw add deny tcp from any to me 22 in
00100 deny tcp from any to me dst-port 22 in
$ sudo ipfw/ipfw show
00100  0    0 deny tcp from any to me dst-port 22 in
65535 49 6080 allow ip from any to any

    But the rules are invalid. I can still connect to the server via SSH.

vmaffione commented 5 years ago

Sorry, I don't understand how you did your tests. What I meant is to check if ping and or SSH (both from the outside) is working in the three following cases: A: no netmap application is running in the VM B: while ./bridge is running as above C: while ./kipfw is running as above.

gsangeryee commented 5 years ago

cases: A: no netmap application is running in the VM . Ping: OK, SSH:OK B: while ./bridge is running as above. Ping: OK, SSH:OK C: while ./kipfw is running as above. Ping: OK, SSH:OK

gsangeryee commented 5 years ago

But after case C, I add rules like this 

$ sudo ipfw/ipfw add allow icmp from me to any icmptypes 8 out
$ sudo ipfw/ipfw add allow icmp from any to me icmptypes 0 in
$ sudo ipfw/ipfw show
00100  0    0 allow icmp from me to any icmptypes 8 out
00200  0    0 allow icmp from any to me icmptypes 0 in
65535 38 3480 allow ip from any to any

The rules are invalid. I can still ping from the outside.

vmaffione commented 5 years ago

Sorry, I don't know how ipfw can be configured correctly for your use-case. Would you expect the ping to be blocked? I see "allow" everywhere...

gsangeryee commented 5 years ago

Hi, vmaffione. Yes, The aims of these two rules is that I can ping from VM to outside and I can't ping from outside to VM.

vmaffione commented 5 years ago

If I were you I would ask on the freebsd-net mailing list, to check if the rules are correct.

gsangeryee commented 5 years ago

Sure, I added these rules on FreeBSD 11.2, It's workable.

gsangeryee commented 5 years ago

As you said before, the kipfw program is experimental. So maybe it can't work in this situation.

vmaffione commented 5 years ago

Ok, also consider that the ipfw code was taken on 2012-08-03 (as you can read in the netmap-ipfw README), so that's a lot of time ago. It may be significantly different from FreeBSD 11.2.

Probably you can debug it a little bit to understand what is going on. Or make your own application based on the forward program (tutorial).