search

Juniper / contrail-vrouter

Contrail Virtual Router
BSD 2-Clause "Simplified" License
218 stars 171 forks source link

vRouer w/DPDK and Mellanox NIC #94

Open nirhenn opened 7 years ago

nirhenn commented 7 years ago

Hi All,

I am trying to setup vRouter w/DPDK and ConnectX-3-Pro. Is there any documentation how to?

The problem is that I cannot bring vif to recognize the DPPK enabled interface. The current state is that install script identifies correctly and interface is binded to the DPDK.

root@Setup-H-C0:/opt/contrail/bin# ./dpdk_nic_bind.py -s

Network devices using DPDK-compatible driver

0000:05:00.0 'MT27520 Family [ConnectX-3 Pro]' drv=igb_uio unused=

Network devices using kernel driver

0000:03:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb unused=igb_uio Active 0000:03:00.3 'I350 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio

Other network devices

Since OFED + PMD drivers are taking over it is not in control of the OS any more and does not show up in ifconfig. in - /etc/contrail/contrail-vrouter-agent.conf # DPDK or legacy work mode platform=dpdk # Physical address of PCI used by dpdk physical_interface_address=0000:05:00.0 # MAC address of device used by dpdk physical_interface_mac=7c:fe:90:b1:ea:50 in - /etc/contrail/agent_param LOG=/var/log/contrail.log CONFIG=/etc/contrail/contrail-vrouter-agent.conf prog=/usr/bin/contrail-vrouter-agent kmod=vrouter pname=contrail-vrouter-agent LIBDIR=/usr/lib64 DEVICE=vhost0 dev=p785p1 vgw_subnet_ip=**VGW_SUBNET_IP** vgw_intf=**VGW_INTF_LIST** LOGFILE=--log-file=/var/log/contrail/vrouter.log
nirhenn commented 7 years ago

Adding some more finding...

Continue our earlier call, from contrail point of view it looks all pointing to the right config, pointing to the right PCI bus/slot and MAC address. When we exec: root@Setup-H-C0:~# /opt/contrail/bin/dpdk_nic_bind.py -s

Network devices using DPDK-compatible driver

0000:05:00.0 'MT27520 Family [ConnectX-3 Pro]' drv=igb_uio unused=

Network devices using kernel driver

0000:03:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb unused=igb_uio Active 0000:03:00.3 'I350 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio

Other network devices

root@Setup-H-C0:~# What is the meaning of the _Active_ ???? From the vrouter log we can see that instead of working with 05:00.0 PCI bus/slot as defined in the setup/config it takes 03:00.0 as show in the dpdk_nic_bind. root@Setup-H-C0:/var/log/contrail# less contrail-vrouter-dpdk-stdout.log EAL: PCI device 0000:03:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped Nir.
srajag commented 7 years ago

“Active” might mean that it is in use by the kernel driver (I didn’t confirm by looking at the dpdk_nic_bind script, but we see it on our setup too). Do you see an vif add messages in contrail-vrouter-dpdk-stdout.log? Does “contrail-status” report any errors? Also, please look in the agent log for errors (/var/log/contrail/contrail-vrouter-agent.log)

Raja

From: Nir Henn notifications@github.com Reply-To: Juniper/contrail-vrouter reply@reply.github.com Date: Sunday, September 18, 2016 at 7:34 AM To: Juniper/contrail-vrouter contrail-vrouter@noreply.github.com Subject: Re: [Juniper/contrail-vrouter] vRouer w/DPDK and Mellanox NIC (#94)

Adding some more finding...

Continue our earlier call, from contrail point of view it looks all pointing to the right config, pointing to the right PCI bus/slot and MAC address. When we exec: root@Setup-H-C0:~# /opt/contrail/bin/dpdk_nic_bind.py -s

Network devices using DPDK-compatible driver

0000:05:00.0 'MT27520 Family [ConnectX-3 Pro]' drv=igb_uio unused=

Network devices using kernel driver

0000:03:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb unused=igb_uio Active 0000:03:00.3 'I350 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio

Other network devices

root@Setup-H-C0:~#

What is the meaning of the Active ???? From the vrouter log we can see that instead of working with 05:00.0 PCI bus/slot as defined in the setup/config it takes 03:00.0 as show in the dpdk_nic_bind.

root@Setup-H-C0:/var/log/contrail# less contrail-vrouter-dpdk-stdout.log

EAL: PCI device 0000:03:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped

Nir.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/Juniper/contrail-vrouter/issues/94#issuecomment-247851185, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFni16RBg6CpWbCEMSerS6pK8IQLOM_Iks5qrUwQgaJpZM4J_27s.

nirhenn commented 7 years ago

The two compute nodes show: == Contrail vRouter == supervisor-vrouter: active contrail-vrouter-agent initializing (Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down) contrail-vrouter-dpdk active
contrail-vrouter-nodemgr active

root@Setup-H-C1:~# contrail-status == Contrail vRouter == supervisor-vrouter: active contrail-vrouter-agent initializing (Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down) contrail-vrouter-dpdk active
contrail-vrouter-nodemgr active

The controller shows: root@Setup-H-Master:~# contrail-status == Contrail Control == supervisor-control: active contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics == supervisor-analytics: active contrail-alarm-gen active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config == supervisor-config: active contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager active
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor active
ifmap active

== Contrail Web UI == supervisor-webui: active contrail-webui active
contrail-webui-middleware active

== Contrail Database == contrail-database: active supervisor-database: active contrail-database-nodemgr active
kafka active

== Contrail Support Services == supervisor-support-service: active rabbitmq-server active

root@Setup-H-Master:~#

on the vif it shows only one interface 0/2 but no vhost0

Potential problem from the contrail-vrouter-dpdk-stdout.log…. You can see that the DPDK boot binds to the wrong interface even that in then config it is the correct one.

Does the DPDK build is linked with Mellanox OFED drivers correctly?

srajag commented 7 years ago

We have not tested with Mellanox NICs. We use DPDK 2.1. I am not sure if it has the drivers for Mellanox NICs.

Raja

From: Nir Henn notifications@github.com Reply-To: Juniper/contrail-vrouter reply@reply.github.com Date: Monday, September 19, 2016 at 10:30 AM To: Juniper/contrail-vrouter contrail-vrouter@noreply.github.com Cc: Raja Sivaramakrishnan raja@juniper.net, Comment comment@noreply.github.com Subject: Re: [Juniper/contrail-vrouter] vRouer w/DPDK and Mellanox NIC (#94)

The two compute nodes show: == Contrail vRouter == supervisor-vrouter: active contrail-vrouter-agent initializing (Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down) contrail-vrouter-dpdk active

contrail-vrouter-nodemgr active

root@Setup-H-C1:~# contrail-status == Contrail vRouter == supervisor-vrouter: active contrail-vrouter-agent initializing (Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down) contrail-vrouter-dpdk active

contrail-vrouter-nodemgr active

The controller shows: root@Setup-H-Master:~# contrail-status == Contrail Control == supervisor-control: active contrail-control active

contrail-control-nodemgr active

contrail-dns active

contrail-named active

== Contrail Analytics == supervisor-analytics: active contrail-alarm-gen active

contrail-analytics-api active

contrail-analytics-nodemgr active

contrail-collector active

contrail-query-engine active

contrail-snmp-collector active

contrail-topology active

== Contrail Config == supervisor-config: active contrail-api:0 active

contrail-config-nodemgr active

contrail-device-manager active

contrail-discovery:0 active

contrail-schema active

contrail-svc-monitor active

ifmap active

== Contrail Web UI == supervisor-webui: active contrail-webui active

contrail-webui-middleware active

== Contrail Database == contrail-database: active supervisor-database: active contrail-database-nodemgr active

kafka active

== Contrail Support Services == supervisor-support-service: active rabbitmq-server active

root@Setup-H-Master:~#

on the vif it shows only one interface 0/2 but no vhost0

Potential problem from the contrail-vrouter-dpdk-stdout.log…. You can see that the DPDK boot binds to the wrong interface even that in then config it is the correct one.

Does the DPDK build is linked with Mellanox OFED drivers correctly?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Juniper/contrail-vrouter/issues/94#issuecomment-248061798, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFni13wYKG6f9pm_AIqr8hxcGPEx11wvks5qrsbAgaJpZM4J_27s.

nirhenn commented 7 years ago

I have also looked in a Juniper Contrail install instance we have and it looks like it uses DPDK 2.1 which do have reference to mlnx4 driver. There is a procedure from Mellanox how to compile it (DODK) with mlnx but it is not clear what will be the Contrail proceedure to update the node with the new setup. Is there dependency with the DKMS ? What is the proceedure to upadate NIC type regardless the Mellanox support? I can try to follow it and give it a try and contribute the output to the community.

srajag commented 7 years ago

You should be able to build the contrail-vrouter-dpdk binary (with any DPDK changes if required) and replace the image in /usr/bin with the new one, then run “service supervisor-vrouter restart”.

Raja

From: Nir Henn notifications@github.com Reply-To: Juniper/contrail-vrouter reply@reply.github.com Date: Monday, September 19, 2016 at 12:39 PM To: Juniper/contrail-vrouter contrail-vrouter@noreply.github.com Cc: Raja Sivaramakrishnan raja@juniper.net, Comment comment@noreply.github.com Subject: Re: [Juniper/contrail-vrouter] vRouer w/DPDK and Mellanox NIC (#94)

I have also looked in a Juniper Contrail install instance we have and it looks like it uses DPDK 2.1 which do have reference to mlnx4 driver. There is a procedure from Mellanox how to compile it (DODK) with mlnx but it is not clear what will be the Contrail proceedure to update the node with the new setup. Is there dependency with the DKMS ? What is the proceedure to upadate NIC type regardless the Mellanox support?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Juniper/contrail-vrouter/issues/94#issuecomment-248100171, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFni13dDNBwlHYHGMW_M0D-tqPWCGPwzks5qruUGgaJpZM4J_27s.

jjgrinwis commented 7 years ago

Did anyone get this to work? We're seeing the same issues and looks like the vRouter is not detecting our Mellanox cards:

EAL: PCI device 0000:02:00.0 on NUMA socket -1 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:02:00.1 on NUMA socket -1 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped ....

Looks like it’s only looking for Intel cards, as Mellanox is being ignored and not added.

root@node-110:~# lspci -s 0000:02:00.0 02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) root@node-110:~# lspci -s 0000:01:00.0 01:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

in an Intel only setup it's working fine: EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:01:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:0e:00.0 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f5540000000 EAL: PCI memory mapped at 0x7f5540080000 PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 3 PMD: eth_ixgbe_dev_init(): port 1 vendorID=0x8086 deviceID=0x10fb EAL: PCI device 0000:0e:00.1 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f5540084000 EAL: PCI memory mapped at 0x7f5540104000 PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 4 PMD: eth_ixgbe_dev_init(): port 2 vendorID=0x8086 deviceID=0x10fb

nirhenn commented 7 years ago

I also gave it a try in the past and after speaking with Mellanox I learned it is not supported. They might have it in future releases of Contrail.

Nir.

--

Nir Henn Innovation Coach, AT&T Foundry

Desk +97297765298 Mobile +972547606867 AMDOCS | EMBRACE CHALLENGE EXPERIENCE SUCCESS This email was sent using 100% recycled electrons!

STOP texting while driving. Take the pledge at itcanwait.comhttp://www.itcanwait.com/

From: jjgrinwis notifications@github.com Reply-To: Juniper/contrail-vrouter reply@reply.github.com Date: Saturday, February 18, 2017 at 11:37 AM To: Juniper/contrail-vrouter contrail-vrouter@noreply.github.com Cc: Nir Henn nir.henn@amdocs.com, State change state_change@noreply.github.com Subject: Re: [Juniper/contrail-vrouter] vRouer w/DPDK and Mellanox NIC (#94)

Did anyone get this to work? We're seeing the same issues and looks like the vRouter is not detecting our Mellanox cards:

EAL: PCI device 0000:02:00.0 on NUMA socket -1 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:02:00.1 on NUMA socket -1 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped ....

Looks like it’s only looking for Intel cards, as Mellanox is being ignored and not added.

root@node-110:# lspci -s 0000:02:00.0 02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) root@node-110:# lspci -s 0000:01:00.0 01:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

in an Intel only setup it's working fine: EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:01:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:0e:00.0 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f5540000000 EAL: PCI memory mapped at 0x7f5540080000 PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 3 PMD: eth_ixgbe_dev_init(): port 1 vendorID=0x8086 deviceID=0x10fb EAL: PCI device 0000:0e:00.1 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f5540084000 EAL: PCI memory mapped at 0x7f5540104000 PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 4 PMD: eth_ixgbe_dev_init(): port 2 vendorID=0x8086 deviceID=0x10fb

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHubhttps://github.com/Juniper/contrail-vrouter/issues/94#issuecomment-280834244, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKB1nrABRshU3o8sSrBU1sTddycrb_Kbks5rdrvJgaJpZM4J_27s. This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,

you may review at http://www.amdocs.com/email_disclaimer.asp

srajag commented 7 years ago

We have not tested this with Mellanox NICs yet.

Raja

From: jjgrinwis notifications@github.com Reply-To: Juniper/contrail-vrouter reply@reply.github.com Date: Saturday, February 18, 2017 at 1:37 AM To: Juniper/contrail-vrouter contrail-vrouter@noreply.github.com Cc: Rajagopalan Sivaramakrishnan raja@juniper.net, Comment comment@noreply.github.com Subject: Re: [Juniper/contrail-vrouter] vRouer w/DPDK and Mellanox NIC (#94)

Did anyone get this to work? We're seeing the same issues and looks like the vRouter is not detecting our Mellanox cards:

EAL: PCI device 0000:02:00.0 on NUMA socket -1 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:02:00.1 on NUMA socket -1 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped ....

Looks like it’s only looking for Intel cards, as Mellanox is being ignored and not added.

root@node-110:# lspci -s 0000:02:00.0 02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) root@node-110:# lspci -s 0000:01:00.0 01:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

in an Intel only setup it's working fine: EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:01:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:0e:00.0 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f5540000000 EAL: PCI memory mapped at 0x7f5540080000 PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 3 PMD: eth_ixgbe_dev_init(): port 1 vendorID=0x8086 deviceID=0x10fb EAL: PCI device 0000:0e:00.1 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f5540084000 EAL: PCI memory mapped at 0x7f5540104000 PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 4 PMD: eth_ixgbe_dev_init(): port 2 vendorID=0x8086 deviceID=0x10fb

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Juniper/contrail-vrouter/issues/94#issuecomment-280834244, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFni19MiSXn9aYLz59KiXEoiDCxp39fMks5rdrvFgaJpZM4J_27s.