manylinux1_x86_64 not correctly linking GOMP_4.0

[rlhelinski]

I just added the following OpenMP pragmas in various places in a C extension:

#pragma omp parallel for collapse(2)
...
#pragma omp parallel for
...
#pragma omp parallel for schedule(static, 1)

and accordingly added the compiler flag -fopenmp. Everything builds and works when I build my extension with GCC 7.5.0 and MSVC 2017 (14.13). However, I get the following error when I {build wheel file, install wheel file, run tests} in the quay.io/pypa/manylinux1_x86_64 docker image (ID 2faa435c5118):

======================================================================
ERROR: Failure: ImportError (/opt/python/cp27-cp27m/lib/python2.7/site-packages/myext/../myext.libs/libgomp-3300acd3.so.1.0.0: version `GOMP_4.0' not found (required by /opt/python/cp27-cp27m/lib/python2.7/site-packages/myext/cmyext.so))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/python/cp27-cp27m/lib/python2.7/site-packages/nose/loader.py", line 407, in loadTestsFromName
    module = resolve_name(addr.module)
  File "/opt/python/cp27-cp27m/lib/python2.7/site-packages/nose/util.py", line 312, in resolve_name
    module = __import__('.'.join(parts_copy))
  File "/opt/python/cp27-cp27m/lib/python2.7/site-packages/myext/__init__.py", line 1, in <module>
    from .cmyext import *
ImportError: /opt/python/cp27-cp27m/lib/python2.7/site-packages/myext/../myext.libs/libgomp-3300acd3.so.1.0.0: version `GOMP_4.0' not found (required by /opt/python/cp27-cp27m/lib/python2.7/site-packages/myext/cmyext.so)

----------------------------------------------------------------------

This error appears only for Python versions cp27-27m, cp27-cp27mu and cp36-cp36m. The other builds are working OK. I am afraid I do not know enough about libgomp to understand how this is supposed to work. I do understand that MSVC 2017 only implements OpenMP 2.0. I have not specified anywhere which version of OpenMP to use.

I can provide more details if desired.

[rlhelinski]

I forgot to mention, of course, that I also added the -fopenmp flag for the linking step (i.e., extension.extra_link_args) in setup.py.

[snnn]

Very likely you put a wrong libgomp.so in your package. You need to get the one from devtoolset, not the system's default. I can confirm it works well in manylinux2014. And before upgrading to 2014, I used manylinux1 before, I didn't find any problem.

[rlhelinski]

How would I do that, exactly? It sounds like an extra option in the call to setup() in setup.py? Thanks

[snnn]

Probably your rpath setting is wrong. Before running the auditwheel tool, you may use the ldd tool to check which gomp your extension points to. It should be hardcoded to a path under devtoolset's installation dir.

[rlhelinski]

Is there some documentation that you could point me to on using rpath and ldd?

[snnn]

Dynamic Section is an optional section in ELF files for holding relevant dynamic linking information.

Sometimes, some build scripts will hardcode specific library paths when linking binaries (using the -rpath or -R flag of gcc). This is commonly referred to as an rpath. Normally, the dynamic linker and loader (ld.so) resolve the executable's dependencies on shared libraries and load what is required. However, when -rpath or -R is used, the location information is then hardcoded into the binary and is examined by ld.so in the beginning of the execution.

You can use the "readelf" tool to view such information.

$ readelf -d something.so 

Dynamic section at offset 0x26bd00 contains 33 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libonnxruntime.so.1.5.2]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN:/home/snnn/src/gcc]
 0x000000000000000c (INIT)               0x408000
 0x000000000000000d (FINI)               0x5a15b4
 0x0000000000000019 (INIT_ARRAY)         0x665590
 0x000000000000001b (INIT_ARRAYSZ)       296 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x6656b8
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x400340
 0x0000000000000005 (STRTAB)             0x401ea0
 0x0000000000000006 (SYMTAB)             0x4003a0
 0x000000000000000a (STRSZ)              10429 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x66d000
 0x0000000000000002 (PLTRELSZ)           6144 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x4066d0
 0x0000000000000007 (RELA)               0x404ba0
 0x0000000000000008 (RELASZ)             6960 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x4049a0
 0x000000006fffffff (VERNEEDNUM)         7
 0x000000006ffffff0 (VERSYM)             0x40475e
 0x0000000000000000 (NULL)               0x0

RPATH is an array of hardcoded paths in dynamic section to specify directories for searching for the NEEDED dependencies. In this example, the array has two entries:"$ORIGIN" and "/home/snnn/src/gcc". But you don't want to see the second one in any of our published prebuilt binaries, because it is an hardcoded absolute path that should only exist on my dev box, it doesn't exist on yours. Also, indeed the first entry is not needed too.

RPATH is old and should be deprecated, but it has the highest precedence compared to the other similar mechanisms(like LD_LIBRARY_PATH). When used properly, it is the most reliable and actually very good.

So my suggestion is: if your project is organized by cmake, you can just use the default settings.

1. At build time cmake will link the executables and shared libraries with full RPATH to all used libraries in the build tree. So all the binaries work perfectly on the build machine, they would never pick up a wrong dependency.

2. When you create the wheel, auditwheel will clear the RPATH of these targets and take over all the remaining tasks from your hands.

All you need to do is to make sure you give auditwheel enough information so that the handover is successful. Here I mean if all the executables have RPATH set correctly, then auditwheel should be able to find all the dependencies correctly.

If you don't use cmake, you can still follow the same guideline.

[mayeut]

Thanks @snnn for the detailed answers. Since there's not been feedback from @rlhelinski for quite some time after your last comment, I consider I can close this issue as it's probably now fixed on his end.

[rlhelinski]

Yes, thanks @snnn for the information. My project does not use cmake. I did not know that it is useful for building Python extensions. I am just using setuptools. Let my take a look at what my NEEDED paths look like.

I have been extremely busy and have just not had a chance to rebuild the wheel files that I am working on.