search

FedericoRessi / devstack-ws

Workspace tools to work with OpenStack and test it using Vagrant
Apache License 2.0
0 stars 1 forks source link

ovs-dpdk doesn't start when stacking with networking-ovs-dpdk master branch. #6

Closed FedericoRessi closed 8 years ago

FedericoRessi commented 8 years ago

I am trying to stack with networking-ovs-dpdk inside a VM running Ubuntu 14.04 LTS with vivid Kernel and clouds repos.

https://github.com/FedericoRessi/devstack-ws/blob/d044f1c54be10648fd4d73c5d775d271a7d5b7b1/scripts/provision.sh#L34-L44

vagrant@compute:/opt/stack$ uname -na
Linux compute 3.19.0-42-generic #48~14.04.1-Ubuntu SMP Fri Dec 18 10:24:49 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I am using master branch of devstack and networking-ovs-dpdk. I have attached produced logs file and used local.conf.

I have a failure when launching OVS-DPDK:

Summary of stack log:

2015-12-23 08:00:31.000 | stack.sh log /opt/stack/logs/screen/xstack.sh.log.2015-12-23-080030
2015-12-23 08:00:34.352 | Installing package prerequisites
2015-12-23 08:02:05.612 | Downloading Dependencies for OVS DPDK
2015-12-23 08:03:05.337 | Installing OpenStack project source
2015-12-23 08:09:52.083 | Configuring, installing and starting OVS DPDK

Tail of full stack log.

2015-12-23 08:11:32.699 | running install_scripts
2015-12-23 08:11:32.722 | ++ popd
2015-12-23 08:11:32.723 | /opt/stack/ovs
2015-12-23 08:11:32.723 | ++ start_ovs_dpdk
2015-12-23 08:11:32.723 | ++ '[' -e /etc/init.d/ovs-dpdk ']'
2015-12-23 08:11:32.723 | ++ sudo service ovs-dpdk start
2015-12-23 08:11:32.730 | sourcing config
2015-12-23 08:11:32.735 | /opt/stack/logs/ovs-vswitchd.pid is not running
2015-12-23 08:11:32.738 | Not all processes are running restart!!!
2015-12-23 08:11:32.738 | 1
2015-12-23 08:11:32.738 | mounting hugepages
2015-12-23 08:11:32.753 | 8192
2015-12-23 08:11:32.950 | libvirt-bin stop/waiting
2015-12-23 08:11:34.973 | libvirt-bin start/running, process 22628
2015-12-23 08:11:34.975 | loading OVS_INTERFACE_DRIVER diver
2015-12-23 08:11:34.975 | Unloading any existing DPDK UIO module
2015-12-23 08:11:34.981 | Loading uio module
2015-12-23 08:11:34.989 | Loading DPDK UIO module
2015-12-23 08:11:34.999 | starting ovs db
2015-12-23 08:11:34.999 | binding nics
2015-12-23 08:11:35.001 | starting vswitchd
2015-12-23 08:11:35.029 | [ERROR] /etc/init.d/ovs-dpdk:319 No argument supplied, PCI_ARRAY[@] expected
2015-12-23 08:11:35.030 | + exit_trap
2015-12-23 08:11:35.030 | + local r=5
2015-12-23 08:11:35.030 | ++ jobs -p
2015-12-23 08:11:35.031 | + jobs=
2015-12-23 08:11:35.031 | + [[ -n '' ]]
2015-12-23 08:11:35.031 | + kill_spinner
2015-12-23 08:11:35.031 | + '[' '!' -z '' ']'
2015-12-23 08:11:35.031 | + [[ 5 -ne 0 ]]
2015-12-23 08:11:35.031 | + echo 'Error on exit'
2015-12-23 08:11:35.032 | Error on exit
2015-12-23 08:11:35.032 | + [[ -z /opt/stack/logs/screen ]]
2015-12-23 08:11:35.032 | + /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs/screen

Commits ids of repos I am using:

[devstack]
commit d21747952e32e6495b35f83d0dcea644f3a08a43
Merge: e76410c 239a978
Author: Jenkins <jenkins@review.openstack.org>
Date:   Tue Dec 22 06:04:46 2015 +0000

    Merge "Remove a stale comment about SQLALCHEMY_DATABASE_DRIVER"

[dpdk]
commit 7173acefc7cfdfbbb9b91fcba1c9a67adb4c07c9
Author: Thomas Monjalon <thomas.monjalon@6wind.com>
Date:   Wed Aug 12 00:56:05 2015 +0200

    version: 2.1.0

    Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>

[horizon]
commit c7de6d5bee940b7333b55dd6546771f2e9e72454
Author: Diana Whitten <hurgleburgler@gmail.com>
Date:   Tue Nov 17 13:59:26 2015 -0700

    Horizon Messages now inherit from a Bootstrap Theme

    Horizon messages (toasts/growls/alerts) now inherit from a theme.
    In addition, a default message type was added 'default', since it
    it possible to kick up an alert without a type specified at all.

    To test all the types, run the following command on a JavaScript
    console with Horizon loaded:

    horizon.alert('warning', 'foo');
    horizon.alert('info', 'foo');
    horizon.alert('success', 'foo');
    horizon.alert('', 'foo');
    horizon.alert('danger', 'foo');

    Partially-Implements: blueprint horizon-theme-css-reorg
    Partially-Implements: blueprint bootstrap-html-standards

    Change-Id: Ice65773fe1d2cdc0305fd01acd8bea3d3f84a9fa

[keystone]
commit 599dbea395acb7240f074c0b6d4b95b1eb4b5cfc
Merge: 07f61a0 87a9a6b
Author: Jenkins <jenkins@review.openstack.org>
Date:   Tue Dec 22 04:44:56 2015 +0000

    Merge "Update `developing.rst` to remove extensions stuff"

[networking-ovs-dpdk]
commit 8b557881ca64656dddb16171c188327f7ff47486
Merge: f4de75c c765eb7
Author: Jenkins <jenkins@review.openstack.org>
Date:   Mon Dec 14 15:38:44 2015 +0000

    Merge "OVS DPDK service shows debug logs on demand"

[neutron]
commit c4e1402e62d023a55aa4e466e36937f3b8f7ddde
Merge: 8a87821 5700afb
Author: Jenkins <jenkins@review.openstack.org>
Date:   Mon Dec 21 22:16:07 2015 +0000

    Merge "Add functional test for availability_zone support"

[nova]
commit 4af4e312df4979738bb2f443fb6502cb8582ca6d
Merge: 9eb8f1e 501a05e
Author: Jenkins <jenkins@review.openstack.org>
Date:   Tue Dec 22 06:30:00 2015 +0000

    Merge "remove unused extends_name attribute"

[ovs]
commit 88058f19ed9aadb1b22d26d93e46b3fd5eb1ad32
Author: Alex Wang <ee07b291@gmail.com>
Date:   Fri Aug 7 00:21:42 2015 -0700

    ovn-controller-vtep: Update related documentation.

    This commit conducts the following documentation changes:

    *   add a description in ovn-architecture manual for
        the life cycle about VTEP gateway.

    *   add TODOs related to ovn-controller-vtep.

    *   refine the ovn-sb, ovn-nb schema manual to require
        logical 'port' type and 'options' configuration.

    Signed-off-by: Alex Wang <ee07b291@gmail.com>
    Acked-by: Russell Bryant <rbryant@redhat.com>
    Acked-by: Justin Pettit <jpettit@nicira.com>

[python-novaclient]
commit 7c3ce18e4d65764cbeee9d0465290fc7bbd33bf5
Merge: 36e125a bbdedc6
Author: Jenkins <jenkins@review.openstack.org>
Date:   Fri Dec 18 22:44:53 2015 +0000

    Merge "Validation for arguments of list command passed by "--fields""

[requirements]
commit 017bc829755dd86a731ee4c6fde762187571ac0d
Author: Ken'ichi Ohmichi <ken-oomichi@wx.jp.nec.com>
Date:   Fri Dec 18 03:47:20 2015 +0000

    Bump minimum tempest-lib version

    Now we(Tempest team) are migrating common part of Tempest to tempest-lib
    for running Tempest-like test on each project repository.
    That work requires the latest tempest-lib. So this patch bumps minimum
    tempest-lib version.

    Change-Id: Ifa69e6c0ecc12d47bb4c53b9b436c2463f1932e2

conf-and-logs.zip

FedericoRessi commented 8 years ago

Probably my problem is using an old version of OVS and DPDK:

https://github.com/FedericoRessi/devstack-ws/blob/2aace45d180a294920319bf994e0db849444bc00/compute.local.conf#L39-L42

FedericoRessi commented 8 years ago

Even stacking with the master branch of all repos in /opt/stack I had the same result:

2015-12-23 11:42:45.717 | running install_egg_info
2015-12-23 11:42:45.718 | removing '/usr/local/lib/python2.7/dist-packages/networking_ovs_dpdk-2015.1.1.dev110-py2.7.egg-info' (and everything under it)
2015-12-23 11:42:45.719 | Copying networking_ovs_dpdk.egg-info to /usr/local/lib/python2.7/dist-packages/networking_ovs_dpdk-2015.1.1.dev110-py2.7.egg-info
2015-12-23 11:42:45.720 | running install_scripts
2015-12-23 11:42:45.743 | ++ popd
2015-12-23 11:42:45.743 | /opt/stack/ovs
2015-12-23 11:42:45.743 | ++ start_ovs_dpdk
2015-12-23 11:42:45.744 | ++ '[' -e /etc/init.d/ovs-dpdk ']'
2015-12-23 11:42:45.744 | ++ sudo service ovs-dpdk start
2015-12-23 11:42:45.750 | sourcing config
2015-12-23 11:42:45.755 | /opt/stack/logs/ovs-vswitchd.pid is not running
2015-12-23 11:42:45.758 | Not all processes are running restart!!!
2015-12-23 11:42:45.758 | 1
2015-12-23 11:42:45.759 | mounting hugepages
2015-12-23 11:42:45.771 | 8192
2015-12-23 11:42:46.002 | libvirt-bin stop/waiting
2015-12-23 11:42:48.040 | libvirt-bin start/running, process 32763
2015-12-23 11:42:48.041 | loading OVS_INTERFACE_DRIVER diver
2015-12-23 11:42:48.042 | Unloading any existing DPDK UIO module
2015-12-23 11:42:48.051 | Loading uio module
2015-12-23 11:42:48.059 | Loading DPDK UIO module
2015-12-23 11:42:48.069 | starting ovs db
2015-12-23 11:42:48.069 | binding nics
2015-12-23 11:42:48.070 | starting vswitchd
2015-12-23 11:42:48.103 | [ERROR] /etc/init.d/ovs-dpdk:319 No argument supplied, PCI_ARRAY[@] expected
2015-12-23 11:42:48.104 | + exit_trap
2015-12-23 11:42:48.104 | + local r=5
2015-12-23 11:42:48.105 | ++ jobs -p
2015-12-23 11:42:48.106 | + jobs=
2015-12-23 11:42:48.106 | + [[ -n '' ]]
2015-12-23 11:42:48.107 | + kill_spinner
2015-12-23 11:42:48.107 | + '[' '!' -z '' ']'
2015-12-23 11:42:48.108 | + [[ 5 -ne 0 ]]
2015-12-23 11:42:48.108 | + echo 'Error on exit'
2015-12-23 11:42:48.109 | Error on exit
2015-12-23 11:42:48.109 | + [[ -z /opt/stack/logs/screen ]]
2015-12-23 11:42:48.109 | + /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs/screen
2015-12-23 11:42:48.365 | + exit 5

Here the versions of repos I am using in /opt/stack:

[devstack]
commit d21747952e32e6495b35f83d0dcea644f3a08a43
Merge: e76410c 239a978
Author: Jenkins <jenkins@review.openstack.org>
Date:   Tue Dec 22 06:04:46 2015 +0000

    Merge "Remove a stale comment about SQLALCHEMY_DATABASE_DRIVER"

[dpdk]
commit 3b60ce8cbb959d7a6839f94ad995a3594c07801e
Author: John McNamara <john.mcnamara@intel.com>
Date:   Wed Dec 16 10:42:50 2015 +0000

    doc: fix missing link target in Linux guide

    Fix missing link in the Linux GSG, accidentally removed
    in previous merge:

    WARNING: undefined label: linux_gsg_compiling_dpdk

    Fixes: 29c673401c4d ("doc: improve Linux guide layout")

    Signed-off-by: John McNamara <john.mcnamara@intel.com>
    Acked-by: Bernard Iremonger <bernard.iremonger@intel.com>

[horizon]
commit 363a341e9a82d0864d369be84e1ba4470d049e32
Merge: d7d4ec0 f89247d
Author: Jenkins <jenkins@review.openstack.org>
Date:   Wed Dec 23 08:10:07 2015 +0000

    Merge "Misspelling in message"

[keystone]
commit 4ebf58566ec7c3a557987c1c9cfd1e03a91272e6
Merge: a86d979 d446e15
Author: Jenkins <jenkins@review.openstack.org>
Date:   Wed Dec 23 00:30:47 2015 +0000

    Merge "Add `keystone-manage bootstrap` command"

[networking-ovs-dpdk]
commit 8b557881ca64656dddb16171c188327f7ff47486
Merge: f4de75c c765eb7
Author: Jenkins <jenkins@review.openstack.org>
Date:   Mon Dec 14 15:38:44 2015 +0000

    Merge "OVS DPDK service shows debug logs on demand"

[neutron]
commit 0bb97acca26f0c139788bf0a33a4b27d4864bcbb
Merge: ce703f7 b0cef88
Author: Jenkins <jenkins@review.openstack.org>
Date:   Wed Dec 23 03:46:38 2015 +0000

    Merge "XenAPI: Fix netwrap to support security group"

[nova]
commit 727656d2f59e26a8c86afdad9b3709ab8efde3c1
Merge: ebe4e10 eec7a55
Author: Jenkins <jenkins@review.openstack.org>
Date:   Wed Dec 23 01:21:16 2015 +0000

    Merge "Nuke EC2 API from api-paste and remove wsgi support"

[ovs]
commit b0656bbc1e393f5012b858611d274e031760b4bf
Author: Ben Pfaff <blp@ovn.org>
Date:   Tue Dec 22 23:27:45 2015 -0800

    ovs-benchmark: Fix memory leak.

    This leak is not a real problem.

    Reported-by: xushengping
    Reported-at: https://github.com/openvswitch/ovs/pull/100
    Signed-off-by: Ben Pfaff <blp@ovn.org>

[python-novaclient]
commit 7c3ce18e4d65764cbeee9d0465290fc7bbd33bf5
Merge: 36e125a bbdedc6
Author: Jenkins <jenkins@review.openstack.org>
Date:   Fri Dec 18 22:44:53 2015 +0000

    Merge "Validation for arguments of list command passed by "--fields""

[requirements]
commit 7d6be02b67e39562f07ebe94000ccfa8ff665857
Merge: e347058 b5e167d
Author: Jenkins <jenkins@review.openstack.org>
Date:   Wed Dec 23 00:45:46 2015 +0000

    Merge "Add openstack/murano-agent to projects.txt"
FedericoRessi commented 8 years ago

Even disabling apparmor

https://github.com/FedericoRessi/devstack-ws/commit/ec5285477e0e53ec8682d52078204775a12eefe3

as sudgested here:

https://github.com/openstack/networking-ovs-dpdk/blob/ee78497a0ae93f55316a29461b9cfa19e6a5f2fd/doc/source/getstarted/ubuntu.rst

I have the same failure here:

https://github.com/openstack/networking-ovs-dpdk/blob/c765eb74734cb34e508b5f30e40563456a357bcf/devstack/ovs-dpdk/ovs-dpdk-init#L319

I have to investigate better inside networking-ovs-dpdk

FedericoRessi commented 8 years ago

Probably after solving this bug I will see this other problem:

https://bugs.launchpad.net/ubuntu/+source/pciutils/+bug/1516095

Proposed workaround for it is running following just before stacking:

sudo /opt/stack/DPDK-*/tools/dpdk_nic_bind.py –b ixgbe <pci_address>

Where would be one of following:

vagrant@control:~$ lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:02.0 VGA compatible controller: InnoTek Systemberatung GmbH VirtualBox Graphics Adapter
00:03.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
00:04.0 System peripheral: InnoTek Systemberatung GmbH VirtualBox Guest Service
00:07.0 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
00:08.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
00:09.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
00:0a.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
00:0d.0 SATA controller: Intel Corporation 82801HM/HEM (ICH8M/ICH8M-E) SATA Controller [AHCI mode] (rev 02)
FedericoRessi commented 8 years ago

After this change: 9b3035bfa631a1969913ecb0331b3491f3535343

Something has changed and now it fails here:

https://github.com/openstack/networking-ovs-dpdk/blob/c765eb74734cb34e508b5f30e40563456a357bcf/devstack/ovs-dpdk/ovs-dpdk-init#L551

2015-12-23 15:07:19.593 | running install_scripts
2015-12-23 15:07:19.617 | ++ popd
2015-12-23 15:07:19.617 | /opt/stack/ovs
2015-12-23 15:07:19.618 | ++ start_ovs_dpdk
2015-12-23 15:07:19.618 | ++ '[' -e /etc/init.d/ovs-dpdk ']'
2015-12-23 15:07:19.618 | ++ sudo service ovs-dpdk start
2015-12-23 15:07:19.625 | sourcing config
2015-12-23 15:07:19.629 | /opt/stack/logs/ovs-vswitchd.pid is not running
2015-12-23 15:07:19.633 | Not all processes are running restart!!!
2015-12-23 15:07:19.633 | 1
2015-12-23 15:07:19.633 | mounting hugepages
2015-12-23 15:07:19.650 | 8192
2015-12-23 15:07:19.973 | libvirt-bin stop/waiting
2015-12-23 15:07:22.004 | libvirt-bin start/running, process 22615
2015-12-23 15:07:22.006 | loading OVS_INTERFACE_DRIVER diver
2015-12-23 15:07:22.006 | Unloading any existing DPDK UIO module
2015-12-23 15:07:22.012 | Loading uio module
2015-12-23 15:07:22.019 | Loading DPDK UIO module
2015-12-23 15:07:22.029 | starting ovs db
2015-12-23 15:07:22.029 | binding nics
2015-12-23 15:07:22.029 | starting vswitchd
2015-12-23 15:07:22.171 | sudo RTE_SDK=/opt/stack/dpdk RTE_TARGET=build /opt/stack/dpdk/tools/dpdk_nic_bind.py -b igb_uio 0000:00:09.0
2015-12-23 15:07:22.345 | sudo RTE_SDK=/opt/stack/dpdk RTE_TARGET=build /opt/stack/dpdk/tools/dpdk_nic_bind.py -b igb_uio 0000:00:0a.0
2015-12-23 15:07:22.582 | sudo ovs-vsctl --no-wait --may-exist add-port br-eth2 dpdk0 -- set Interface dpdk0 type=dpdk
2015-12-23 15:07:22.611 | sudo ovs-vsctl --no-wait --may-exist add-port br-eth3 dpdk1 -- set Interface dpdk1 type=dpdk
2015-12-23 15:07:23.785 | Waiting for ovs-vswitchd to start...
2015-12-23 15:07:24.854 | 0
2015-12-23 15:07:24.918 | libvirt-bin stop/waiting
2015-12-23 15:07:26.941 | libvirt-bin start/running, process 23089
2015-12-23 15:07:26.944 | [ERROR] /etc/init.d/ovs-dpdk:551 ovs-vswitchd application failed to start
2015-12-23 15:07:26.947 | + exit_trap
2015-12-23 15:07:26.947 | + local r=8
2015-12-23 15:07:26.947 | ++ jobs -p
2015-12-23 15:07:26.948 | + jobs=
2015-12-23 15:07:26.949 | + [[ -n '' ]]
2015-12-23 15:07:26.949 | + kill_spinner
2015-12-23 15:07:26.950 | + '[' '!' -z '' ']'
2015-12-23 15:07:26.950 | + [[ 8 -ne 0 ]]
2015-12-23 15:07:26.951 | + echo 'Error on exit'
2015-12-23 15:07:26.954 | Error on exit
2015-12-23 15:07:26.955 | + [[ -z /opt/stack/logs/screen ]]
2015-12-23 15:07:26.955 | + /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs/screen
FedericoRessi commented 8 years ago

It looks like I have jumped to the next problem: I need more RAM for my compute node.

From ovs-vswitchd.log:

EAL: Not enough memory available on socket 0! Requested: 2048MB, available: 1972MB
PANIC in rte_eal_init():
Cannot init memory
7: [/usr/sbin/ovs-vswitchd() [0x411623]]
6: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f2730b88ec5]]
5: [/usr/sbin/ovs-vswitchd() [0x40fe32]]
4: [/usr/sbin/ovs-vswitchd() [0x68c842]]
3: [/usr/sbin/ovs-vswitchd() [0x448db7]]
2: [/usr/sbin/ovs-vswitchd() [0x407c30]]
1: [/usr/sbin/ovs-vswitchd() [0x44ec98]]
FedericoRessi commented 8 years ago

It looks like commit 2fdf40e78679f413e1ebb5a218c558edeb43db1e fixed this problem.