openvswitch / ovs-issues

Issue tracker repo for Open vSwitch
10 stars 3 forks source link

Correct procedure to install OVS+dpdk for RHEL 8 #263

Open davehouser1 opened 2 years ago

davehouser1 commented 2 years ago

Note: already posted about some of this here. Already tried following the instructions here. DPDK installs fine. But Openvswitch will not install. There are not clear instructions that I could find on how to install Openvswitch on RHEL8. This page states this:

RHEL distributes openvswitch rpm package that supports kernel datapath. DPDK accelerated Open vSwitch can be installed using openvswitch-dpdk package.

However I could not find anyway to install via yum or dnf, none of the current repos include openvswitch or openvswitch-dpdk I shifted to installing using an RPM. You can read about how I was able to do this with a regular ovs deployment here. The process works for configuring ovs without --with-dpdk=<value> parameter. However when I include that parameter in the .spec file configure section, then create an rpm, the rpm will not install. I get these errors:

# rpm -Uvh --force ./openvswitch-2.17.2-1.el8.x86_64.rpm
error: Failed dependencies:
        librte_acl.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_bbdev.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_bitratestats.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_bpf.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_cfgfile.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_cmdline.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_compressdev.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_cryptodev.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_distributor.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_dmadev.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_eal.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_efd.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_ethdev.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_eventdev.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_fib.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_flow_classify.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_gpudev.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_graph.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_gro.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_gso.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_hash.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_ip_frag.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_ipsec.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_jobstats.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_kni.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_kvargs.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_latencystats.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_lpm.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_mbuf.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_member.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_mempool.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_meter.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_metrics.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_net.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_node.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_pcapng.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_pci.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_pdump.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_pipeline.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_port.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_power.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_rawdev.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_rcu.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_regexdev.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_reorder.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_rib.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_ring.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_sched.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_security.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_stack.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_table.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_telemetry.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_timer.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_vhost.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        python3 < 3.6.6-13 conflicts with (installed) platform-python-3.6.8-45.el8.x86_64
        python3 < 3.6.6-9 conflicts with (installed) platform-python-devel-3.6.8-45.el8.x86_64

DPDK is installed and I can find with pkg-config

# pkg-config --modversion libdpdk
21.11.1

I tried using --with-dpdk=static --with-dpdk=shared and --with-dpdk=yes (Which I believe is the same as static) none worked. I was able to compile an install on Ubuntu 18.04 with out these issues.

I am not clear on why the rpm install needs these shared libraries but the non dpdk installs dont. It looks like many of these dependencies are shared objects. Were they not installed? is dpdk flavor installs looking for these? How do I correct this problem so I can install ovs with dpdk?

Also if anyone knows how to fix the Python3 conflict errors that would be great. I just ignore them with --nodeps on install of the rpm.

UPDATE: well it seems I could just yum install dpdk, and that took care of most of the dependencies. Whats left is the following:

# rpm -Uvh --force ./openvswitch-2.17.2-1.el8.x86_64.rpm
error: Failed dependencies:
        librte_gpudev.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_jobstats.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_kni.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        librte_power.so.22()(64bit) is needed by openvswitch-2.17.2-1.el8.x86_64
        python3 < 3.6.6-13 conflicts with (installed) platform-python-3.6.8-45.el8.x86_64
        python3 < 3.6.6-9 conflicts with (installed) platform-python-devel-3.6.8-45.el8.x86_64

All of these librte_* are related to dpdk in some way I think. Not sure why these are not being found.

igsilya commented 2 years ago

RPM is looking for symbols and libraries available in other RPM packages, not actual files on the filesystem. So, you need a DPDK RPM package installed.

You also need the RPM that is the same with which you built OVS with. So, you need to either build a DPDK package, install it and build OVS with it, or, since RHEL8 happened to have a compatible version of DPDK, install dpdk-devel and dpdk packages from RHEL repositories and build with them. These packages do not provide static libraries, IIRC, so you have to use --with-dpdk=shared.

On a clean system, you should be able to install dpdk-devel and dpdk packages from RHEL8 repo and build an OVS 2.17+ with it with the following spec change:

diff --git a/rhel/openvswitch.spec.in b/rhel/openvswitch.spec.in
index 9903dd10a..b8d2740c3 100644
--- a/rhel/openvswitch.spec.in
+++ b/rhel/openvswitch.spec.in
@@ -32,14 +32,13 @@ License: ASL 2.0
 Release: %{release_number}%{?dist}
 Source: openvswitch-%{version}.tar.gz
 Buildroot: /tmp/openvswitch-rpm
-Requires: logrotate, hostname, python >= 2.7, python-six
-BuildRequires: python-six
+Requires: logrotate, hostname, python3 >= 3.4, dpdk == 21.11
 BuildRequires: openssl-devel
 BuildRequires: checkpolicy, selinux-policy-devel
 BuildRequires: autoconf, automake, libtool
 BuildRequires: python3-sphinx
 BuildRequires: unbound-devel
-BuildRequires: libunwind-devel
+BuildRequires: dpdk-devel == 21.11

 %bcond_without check
 %bcond_with check_datapath_kernel
@@ -70,7 +69,7 @@ Tailored Open vSwitch SELinux policy

 %build
 ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=%{_localstatedir} \
-    --libdir=%{_libdir} --enable-ssl --enable-shared
+    --libdir=%{_libdir} --enable-ssl --enable-shared --with-dpdk=shared
 make %{_smp_mflags}
 make selinux-policy

@@ -102,7 +101,6 @@ rm \
     $RPM_BUILD_ROOT/usr/share/man/man8/ovs-test.8 \
     $RPM_BUILD_ROOT/usr/share/man/man8/ovs-l3ping.8
 (cd "$RPM_BUILD_ROOT" && rm -rf usr/%{_lib}/*.la)
-(cd "$RPM_BUILD_ROOT" && rm -rf usr/include)

 install -d -m 0755 $RPM_BUILD_ROOT%{_rundir}/openvswitch
 install -d -m 0755 $RPM_BUILD_ROOT%{_localstatedir}/log/openvswitch
@@ -255,6 +253,7 @@ exit 0
 %{_libdir}/lib*.a
 %{_libdir}/pkgconfig
 %{_includedir}/openvswitch/*
+%{_includedir}/openflow/*

 %files selinux-policy
 %defattr(-,root,root)
davehouser1 commented 2 years ago

I tried doing as you suggested. It took care of a lot of those lib rte dependencies. But there are a few left that still show up as not being found when installing the rpm, I detail this at the end of my post as an update. Any idea why those are not covered?

On Thu, Sep 1, 2022, 8:45 AM Ilya Maximets @.***> wrote:

RPM is looking for symbols and libraries available in other RPM packages, not actual files on the filesystem. So, you need a DPDK RPM package installed.

You also need the RPM that is the same with which you built OVS with. So, you need to either build a DPDK package, install it and build OVS with it, or, since RHEL8 happened to have a compatible version of DPDK, install dpdk-devel and dpdk packages from RHEL repositories and build with them. These packages do not provide static libraries, IIRC, so you have to use --with-dpdk=shared.

On a clean system, you should be able to install dpdk-devel and dpdk packages from RHEL8 repo and build an OVS 2.17+ with it with the following spec change:

diff --git a/rhel/openvswitch.spec.in b/rhel/openvswitch.spec.in index 9903dd10a..b8d2740c3 100644 --- a/rhel/openvswitch.spec.in +++ b/rhel/openvswitch.spec.in @@ -32,14 +32,13 @@ License: ASL 2.0 Release: %{release_number}%{?dist} Source: openvswitch-%{version}.tar.gz Buildroot: /tmp/openvswitch-rpm -Requires: logrotate, hostname, python >= 2.7, python-six -BuildRequires: python-six +Requires: logrotate, hostname, python3 >= 3.4 BuildRequires: openssl-devel BuildRequires: checkpolicy, selinux-policy-devel BuildRequires: autoconf, automake, libtool BuildRequires: python3-sphinx BuildRequires: unbound-devel -BuildRequires: libunwind-devel +BuildRequires: dpdk-devel == 21.11

%bcond_without check %bcond_with check_datapath_kernel @@ -70,7 +69,7 @@ Tailored Open vSwitch SELinux policy

%build ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=%{_localstatedir} \

  • --libdir=%{_libdir} --enable-ssl --enable-shared
  • --libdir=%{_libdir} --enable-ssl --enable-shared --with-dpdk=shared make %{_smp_mflags} make selinux-policy

@@ -102,7 +101,6 @@ rm \ $RPM_BUILD_ROOT/usr/share/man/man8/ovs-test.8 \ $RPM_BUILD_ROOT/usr/share/man/man8/ovs-l3ping.8 (cd "$RPM_BUILD_ROOT" && rm -rf usr/%{_lib}/*.la) -(cd "$RPM_BUILD_ROOT" && rm -rf usr/include)

install -d -m 0755 $RPM_BUILD_ROOT%{_rundir}/openvswitch install -d -m 0755 $RPM_BUILD_ROOT%{_localstatedir}/log/openvswitch @@ -255,6 +253,7 @@ exit 0 %{_libdir}/lib.a %{_libdir}/pkgconfig %{_includedir}/openvswitch/ +%{_includedir}/openflow/*

%files selinux-policy %defattr(-,root,root)

— Reply to this email directly, view it on GitHub https://github.com/openvswitch/ovs-issues/issues/263#issuecomment-1234230217, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOL4D3LN2Z6J5LB465EHCWLV4CQPXANCNFSM6AAAAAAQBVZRBI . You are receiving this because you authored the thread.Message ID: @.***>

igsilya commented 2 years ago

I tried doing as you suggested. It took care of a lot of those lib rte dependencies. But there are a few left that still show up as not being found when installing the rpm, I detail this at the end of my post as an update. Any idea why those are not covered?

If you have unresolved DPDK dependencies, t meant that you built OVS with a version of DPDK that is not installed from the RPM. You need to clean up your system. Delete the dpdk and dpdk-devel packages, then find and delete all the dpdk libraries remaining as they were not part of the package. Then re-build OVS from scratch. Also, make sure that pkg-config points to the right set of libraries.

davehouser1 commented 2 years ago

Ok so this all seems like an issue with how I am building RPMs, and the fact I am trying to use RPMs to install as the specs are so old (the install actual creates a generated init.d file instead of creating a systemd unit that controls ExecStart of a binary).

I did find that fastpath does have the latest 2.17.3 version of openvswitch. I did the following

# sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
# subscription-manager repos --enable=fast-datapath-for-rhel-8-x86_64-rpms
# sudo sudo yum install https://rdoproject.org/repos/rdo-release.rpm
# sudo yum install dpdk
# sudo yum install dpdk-devel
# pkg-config --modversion libdpdk
21.11.0
# yum install openvswitch2.17.x86_64

After installing ovs seems to meet all my needs. However I am not 100% sure that dpdk is enabled on this version from fastpath. I was able to create a netdev enabled bridge, enable dpdk interfaces, and add dpdk PF's and VF's, with no errors. So I can assume that its working?

My only consern is that instructions show I should be installing ovs-dpdk (which is not available in fastpath), also running ovs-vsctl --version shows

ovs-vsctl (Open vSwitch) 2.17.3
DB Schema 8.3.0

On Ubuntu, it would say "dpdk 21.11.0" at the bottom of the version. Anyway to know for user if DPDK working with ovs?

igsilya commented 2 years ago

If fast-datapath-for-rhel-8-x86_64-rpms is available to you, you should not need EPEL or RDO repositories. The main FDP repository should be enough. openvswitch2.17 package there is statically linked with DPDK, so you don't need to install neither dpdk or dpdk-devel packages.

If you were able to add DPDK ports without errors, OVS has support for them. IIRC, Ubuntu has a custom patch to add the DPDK version to the ovs-vsctl --version output. The better way to check is to run ovs-vswitchd --version instead.

igsilya commented 2 years ago

Another way to build OVS from sources is to use the fedora spec file instead. It is way more up to date. You'll need codeready-builder-for-rhel-8-x86_64-rpms repo enabled for the groff package on rhel8, or you may just remove the groff from the build requirements in the spec. It is optional.