patched OVS with ports other than vale created ports

[Linerd]

My patched OVS works great with ports created with vale-ctl, but when I tried to attach other ports to the OVS bridge(the physical interfaces or the interfaces within VM), it failed. I got the "failed to add port XX, invalid argument" error.

I am trying to figure out if this is my problem or the inner limitation of the current patch. When I switch back to the original OVS I can attach whatever interface I want.

PS. just try to make sure I understand correctly, it is impossible to run OVS on two vale switches on the same host right? Coz every bridge I create will automatically be attached to vale0.

[vmaffione]

Just to understand, are you using the patch at "utils/switch-modules/ovs-datapath/0001-datapath-Add-support-for-netmap-VALE.patch" ?

[Linerd]

@vmaffione yes, actually it is slightly modified by @micchie to support OVS 2.6.0, but I assume it shouldn't be that different

[vmaffione]

Hi, Attaching a linux interface to OVS works only if the interface has native netmap support. This means that VALE ports (e.g. interfaces created with vale-ctl -n) and ixgbe,e1000,veth,etc. (with patched drivers) can be attached to an OVS bridge with "ovs-vsctl add-port". This limitation is due to a confilct: both OVS and the emulated (generic) adapter use the same functionality to intercept packets, and so they cannot work together.

The patch for OVS 2.6.1 has been published here https://github.com/luigirizzo/netmap/blob/master/utils/switch-modules/ovs-2.6.1/0001-datapath-netmap-VALE-support.patch .

For the second question: yes, it is not possible to use two VALE switches with OVS on the same host, and it is a limitation of the current patch. This is actually more a problem of OVS, since afaik in the openvswitch kernel module there is no abstraction for the "bridge" object. This means that all the OVS bridges and ports you create share the same "datapath" object in kernelspace. As a consequence, there is no an easy way to tell (in kernel space) what OVS bridge a port is part of. Actually, I was trying to circumvent this problem (see https://github.com/vmaffione/ovs/tree/netmap-2.6.1-multi), but it may be not convenient.

[Linerd]

@vmaffione when OVS is built with the new patch, the generated openvswitch.ko file in ovs/datapath/linux/ does not depend on netmap. Is it this expected? The old patch files used to build openvswitch.ko that depends on netmap

[Linerd]

@vmaffione the build process says "netmap_attach, netmap_detach, netmap_bdg_ctl undefined", thus the built ovs module does not depend on netmap. When inserted, invalid parameter error is reported.

[vmaffione]

The patch fo 2.6.1 works fine to me. Are you using 2.4.0 or 2.6.1? Of course openvswitch.ko must depend on netmap.ko

[Linerd]

I am using 2.6.1. Both OVS and netmap are the latest. What linux kernel and system are you using?

On Jan 9, 2017, 12:43 PM -0500, Vincenzo Maffione notifications@github.com, wrote:

The patch fo 2.6.1 works fine to me. Are you using 2.4.0 or 2.6.1? Of course openvswitch.ko must depend on netmap.ko — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[vmaffione]

I'm using linux 4.8.13 on archlinux. But this should not be important. Have you used the --with-netmap switch to point at the netmap sources? Are you sure you have applied the patch to the 2.6.1 tar.gz from the ovs website, or the v2.6.1 tag on the ovs github repo?

[Linerd]

I used the —with-netmap option, along with the —with-linux option. I applied it to the v2.6.1 tag on the GitHub repo.

On Jan 9, 2017, 3:30 PM -0500, Vincenzo Maffione notifications@github.com, wrote:

I'm using linux 4.8.13 on archlinux. But this should not be important. Have you used the --with-netmap switch to point at the netmap sources? Are you sure you have applied the patch to the 2.6.1 tar.gz from the ovs website, or the v2.6.1 tag on the ovs github repo? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[vmaffione]

Have you installed netmap first and then installed openvswitch modules with

´´´´

make modules_install

´´´´

Yours may be a depmod problem. You can also try to run depmod -a manually.

[Linerd]

Yes I installed netmap first. The problem occurred before installing openvswitch modules. The built openvswitch.ko modules does not depend on netmap. netmap_attach(), netmap_detach() methods are not found. Could it be that some netmap files are not installed properly on the system?

On Jan 9, 2017, 3:54 PM -0500, Vincenzo Maffione notifications@github.com, wrote:

Have you installed netmap first and then installed openvswitch modules with ´´´´ make modules_install ´´´´ Yours may be a depmod problem. You can also try to run depmod -a manually. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[vmaffione]

No. Attach and detach functions are defined in netmap_kern.h, which is not exported for now. So if you get an undefined reference error it means that either your --with-netmap path is wrong or that you are compiling netmap and ovs against two different kernel build infrastructures (you typically wantto.use se /lib/modules/´uname -r´/build for noth)

[Linerd]

@vmaffione I tried again, this time I make sure the same kernel build is used, but still can't generate the correct ovs module. The patch @micchie gave me months ago works, but his patch needs some fix for the latest netmap build. (I will email you his patch just for reference, maybe you could find his patch useful).

[vmaffione]

I think the problem is that you are running the netmap ./configure (and make) from inside the LINUX directory. Try to run them from the root of the netmap repository, I.e. a level up.

Il 10 gen 2017 5:22 AM, "Yikai" notifications@github.com ha scritto:

@vmaffione https://github.com/vmaffione I tried again, this time I make sure the same kernel build is used, but still can't generate the correct ovs module. The patch @micchie https://github.com/micchie gave me months ago works, but his patch needs some fix for the latest netmap build. (I will email you his patch just for reference, maybe you could find his patch useful).

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/luigirizzo/netmap/issues/266#issuecomment-271483002, or mute the thread https://github.com/notifications/unsubscribe-auth/AEsSwWvhQFdtVHWwEJTdiGWNMjscsKPCks5rQweWgaJpZM4LY0EY .

[Linerd]

@vmaffione now it works!

I see that this patched version of OVS still can't work with mininet(it is stuck at starting switches stage), but I believe mininet is using veth pairs to connect hosts. Is there anything I can do to make mininet work? I have already loaded the netmap version of veth module.

[vmaffione]

For sure you need to fetch the latest code to get a bug-fix for veth that just went in. The support for veth has been recently reworked (Dec 22nd 2016), so that it does not suffer anymore from previous limitations (e.g. both ends of a veth pair needed to be in netmap mode), and race conditions (crashes were possible). Now veth in native netmap mode work exactly like netmap pipes.

You can attach veth interfaces to vale switches in general, this works ok. It works ok also in conjuction with OVS patched with netmap, you can use veth interfaces similarly to how you use persistent VALE ports.

I've no idea why mininet does not work, it can simply be that the OVS patch is very experimental and incomplete. One of the problems is that broadcast does not work properly if you create more OVS bridges. A single datapath (in this case a VALE switch) is used to implement all the bridges, so the bridges are not isolated.

[Linerd]

@vmaffione

I built the latest netmap and loaded both netmap.ko and veth.ko, and I created a simple mininet topology with the command mn, and the process is stuck at creating the switches, dmesg gives me the following errors. If I don't load veth.ko, the topology can be created, but h1 cannot ping h2.

[12429.730416] (null): netmap queues/slots: TX 1/1024, RX 1/1024 [12429.730637] (null): netmap queues/slots: TX 1/1024, RX 1/1024 [12429.731912] IPv6: ADDRCONF(NETDEV_UP): h1-eth0: link is not ready [12429.734490] IPv6: ADDRCONF(NETDEV_CHANGE): h1-eth0: link becomes ready [12429.737168] (null): netmap queues/slots: TX 1/1024, RX 1/1024 [12429.737453] (null): netmap queues/slots: TX 1/1024, RX 1/1024 [12429.740032] IPv6: ADDRCONF(NETDEV_UP): h2-eth0: link is not ready [12429.743699] IPv6: ADDRCONF(NETDEV_CHANGE): h2-eth0: link becomes ready [12429.756165] device ovs-system entered promiscuous mode [12429.756900] BUG: unable to handle kernel NULL pointer dereference at 000000000000001e [12429.756914] IP: [] ovs_vport_add+0x117/0x170 [openvswitch] [12429.756927] PGD 20cefe067 PUD 20ceff067 PMD 0 [12429.756932] Oops: 0000 [#1] SMP [12429.756936] Modules linked in: veth(OE) openvswitch(OE) netmap(OE) rfcomm pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat xt_addrtype iptable_filter ip_tables xt_conntrack x_tables br_netfilter bridge stp llc aufs bnep intel_rapl snd_hda_codec_hdmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel btusb kvm irqbypass crct10dif_pclmul crc32_pclmul btrtl btbcm hp_wmi gpio_ich btintel sparse_keymap snd_hda_codec_realtek snd_hda_codec_generic bluetooth snd_hda_intel input_leds ghash_clmulni_intel aesni_intel aes_x86_64 snd_hda_codec snd_hda_core snd_hwdep snd_pcm lrw gf128mul glue_helper snd_seq_midi snd_seq_midi_event snd_rawmidi ablk_helper snd_seq cryptd snd_seq_device snd_timer snd serio_raw soundcore shpchp [12429.757024] 8250_fintek lpc_ich ie31200_edac tpm_infineon edac_core mei_me mei mac_hid nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 gre nf_conntrack_ipv6 nf_nat_ipv6 libcrc32c nf_defrag_ipv6 nf_nat nf_conntrack parport_pc ppdev lp parport autofs4 hid_generic hid_logitech_hidpp hid_logitech_dj usbhid hid uas usb_storage nouveau mxm_wmi video i2c_algo_bit ttm psmouse ahci drm_kms_helper libahci syscopyarea sysfillrect sysimgblt fb_sys_fops e1000e drm ptp pps_core pata_acpi wmi fjes [last unloaded: openvswitch] [12429.757084] CPU: 2 PID: 6770 Comm: ovs-vswitchd Tainted: G OE 4.4.0-59-generic #80-Ubuntu [12429.757089] Hardware name: Hewlett-Packard HP Z210 Workstation/1588h, BIOS J51 v01.19 08/12/2011 [12429.757093] task: ffff880210ce7000 ti: ffff880200d2c000 task.ti: ffff880200d2c000 [12429.757097] RIP: 0010:[] [] ovs_vport_add+0x117/0x170 [openvswitch] [12429.757107] RSP: 0018:ffff880200d2fa68 EFLAGS: 00010203 [12429.757110] RAX: 0000000000000016 RBX: ffffffffc08a0700 RCX: 0000000000000000 [12429.757114] RDX: 000000000002178c RSI: ffff88022dd1a0c0 RDI: ffff880225001900 [12429.757117] RBP: ffff880200d2fa78 R08: 000000000001a0c0 R09: ffffffffc0887f7e [12429.757121] R10: ffffea00088a5ec0 R11: ffff880144a0b000 R12: 0000000000000016 [12429.757125] R13: ffff880220ad5980 R14: ffff8802085cd800 R15: 0000000000000001 [12429.757129] FS: 00007fd100122980(0000) GS:ffff88022dd00000(0000) knlGS:0000000000000000 [12429.757133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12429.757137] CR2: 000000000000001e CR3: 0000000144a01000 CR4: 00000000000406e0 [12429.757140] Stack: [12429.757142] ffff880200d2faa8 ffff880221c0a300 ffff880200d2fa90 ffffffffc087b822 [12429.757149] ffff880200d2fb50 ffff880200d2fb08 ffffffffc087ccf2 ffff8800c902f814 [12429.757155] ffff8800c902f824 ffffffff00000001 0000000000000000 ffff880035357f00 [12429.757161] Call Trace: [12429.757168] [] new_vport+0x12/0x50 [openvswitch] [12429.757175] [] ovs_vport_cmd_new+0x142/0x280 [openvswitch] [12429.757181] [] genl_family_rcv_msg+0x1e4/0x3e0 [12429.757187] [] ? alloc_skb+0x5b/0x1f0 [12429.757191] [] ? genl_family_rcv_msg+0x3e0/0x3e0 [12429.757195] [] genl_rcv_msg+0x76/0xb0 [12429.757199] [] netlink_rcv_skb+0xa4/0xc0 [12429.757206] [] genl_rcv+0x28/0x40 [12429.757209] [] netlink_unicast+0x12f/0x1b0 [12429.757213] [] netlink_sendmsg+0x4a1/0x5f0 [12429.757219] [] ? aa_sock_msg_perm+0x61/0x150 [12429.757224] [] socksendmsg+0x38/0x50 [12429.757229] [] sys_sendmsg+0x281/0x290 [12429.757234] [] ? rtnl_unlock+0xe/0x10 [12429.757238] [] ? dev_ioctl+0x1ae/0x580 [12429.757243] [] ? sock_do_ioctl+0x42/0x50 [12429.757247] [] ? __fget_light+0x25/0x60 [12429.757251] [] __sys_sendmsg+0x51/0x90 [12429.757255] [] SyS_sendmsg+0x12/0x20 [12429.757260] [] entry_SYSCALL_64_fastpath+0x16/0x71 [12429.757264] Code: 83 48 8b 7b 38 e8 6a df 87 c0 84 c0 75 0c 48 c7 c0 9f ff ff ff 5b 41 5c 5d c3 4c 89 e7 ff 53 08 48 3d 00 f0 ff ff 49 89 c4 77 3c <48> 8b 40 08 49 8b 34 24 48 8b 78 60 e8 58 fb ff ff 48 8b 10 49 [12429.757320] RIP [] ovs_vport_add+0x117/0x170 [openvswitch] [12429.757327] RSP [12429.757330] CR2: 000000000000001e [12429.762939] ---[ end trace 2325cc85140cc23b ]---

[vmaffione]

Yes, the OVS patch is experimental, you need to debug it.

Some pieces of advice:

• Try to remove && vale_prefix(ovs_dp_name(parms->dp)) in datapath/vport.c::ovs_vport_add() to see if the crash goes away
  • Add print statements before and after the DEV_NETMAP block in ovs_vport_add() (e.g. D("before"); and D("after");), to check that the crash happens in that block.