search

RRZE-HPC / pylikwid

Python interface for the LIKWID C API (https://github.com/RRZE-HPC/likwid)
GNU General Public License v2.0
43 stars 13 forks source link

Installation fails (pip and manual) #8

Closed d4k0 closed 6 years ago

d4k0 commented 6 years ago

Hi,

I wanted to use pylikwid for my thesis, but unfortunately the installation fails with an error message. This is what I get when using pip to install pylikwid:

$ pip install --user pylikwid
Collecting pylikwid
  Cache entry deserialization failed, entry ignored
  Using cached https://files.pythonhosted.org/packages/bd/73/a8baea30ebd31e4ba6ae5110a83dfdacdc4c5c527ff5e1abf8038499a753/pylikwid-0.2.4.tar.gz
Building wheels for collected packages: pylikwid
  Running setup.py bdist_wheel for pylikwid ... error
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-0jLJ6K/pylikwid/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpTM5RDHpip-wheel- --python-tag cp27:
  running bdist_wheel
  running build
  running build_ext
  building 'pylikwid' extension
  creating build
  creating build/temp.linux-x86_64-2.7
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/lib/include -I/usr/include/python2.7 -c pylikwid.c -o build/temp.linux-x86_64-2.7/pylikwid.o
  creating build/lib.linux-x86_64-2.7
  x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/pylikwid.o -L/usr/lib/lib -l: -o build/lib.linux-x86_64-2.7/pylikwid.so
  /usr/bin/x86_64-linux-gnu-ld: -l: kann nicht gefunden werden
  collect2: error: ld returned 1 exit status
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for pylikwid
  Running setup.py clean for pylikwid
Failed to build pylikwid
Installing collected packages: pylikwid
  Running setup.py install for pylikwid ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-0jLJ6K/pylikwid/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-ZyeHaw-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    running install
    running build
    running build_ext
    building 'pylikwid' extension
    creating build
    creating build/temp.linux-x86_64-2.7
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/lib/include -I/usr/include/python2.7 -c pylikwid.c -o build/temp.linux-x86_64-2.7/pylikwid.o
    creating build/lib.linux-x86_64-2.7
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/pylikwid.o -L/usr/lib/lib -l: -o build/lib.linux-x86_64-2.7/pylikwid.so
    /usr/bin/x86_64-linux-gnu-ld: -l: kann nicht gefunden werden
    collect2: error: ld returned 1 exit status
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-0jLJ6K/pylikwid/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-ZyeHaw-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-0jLJ6K/pylikwid/

I then tried to install it manually, but I get the same error:

$ python setup.py build_ext -I /usr/local/include/ -L /usr/local/lib/ -R /usr/local/lib/
Searching for LIKWID installation
Using LIKWID installation at /usr/lib
/usr/lib/lib
running build_ext
building 'pylikwid' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/lib/include -I/usr/local/include/ -I/usr/include/python2.7 -c pylikwid.c -o build/temp.linux-x86_64-2.7/pylikwid.o
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/pylikwid.o -L/usr/lib/lib -L/usr/local/lib/ -Wl,-R/usr/local/lib/ -l: -o build/lib.linux-x86_64-2.7/pylikwid.so
/usr/bin/x86_64-linux-gnu-ld: -l: kann nicht gefunden werden
collect2: error: ld returned 1 exit status
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

At first, the error message looked strange, but then I saw in the logs of your build server that liblikwid.so.4.3 is missing after -l: at the end of the command. However, this file is found on the system when I search for it (I installed LIKWID through apt and it works). I also tried older versions from PyPI, but I get the same error.

Is there maybe something wrong in the script or needs to be adjusted? I also noticed that your build server fails as well (but at another position).

Here's some more information about my system:

OS: Ubuntu 18.04 (freshly installed)
gcc: 7.3.0
LIKWID: 4.3.1 (installed using apt)
Python: 2.7.15
TomTheBear commented 6 years ago

Hi, it seems that the location detection of the LIKWID library fails for you. It detects the liblikwid.so in /usr/lib but uses /usr/lib/lib later during build. Maybe it uses also /usr/lib/lib when checking the version number of the LIKWID library, which does not exist and consequently returns the empty string. It seems I should do more checking in setup.py.

The build on TravisCI does not fail. It also runs the tests successfully but isn't able to deploy the new package to PyPI because I havn't increased the version number with my last commit. This causes the error there.

TomTheBear commented 6 years ago

I did some fixes to setup.py, can you please retry.

No need to try it, I installed Ubuntu 18.04 in a virtual machine. The problem is, that the tools are installed at /usr/bin, so the installation routine checks in /usr/lib but actually the library is installed in /usr/lib/x86_64-linux-gnu/.

d4k0 commented 6 years ago

Thanks for your fast reply. I tried the latest version 0.2.6 from PyPI and from the GitHub repository, but I get the same error.

Installation using pip:

$ pip install --user pylikwid
Collecting pylikwid
  Downloading https://files.pythonhosted.org/packages/97/ae/0e6a77c122fe82dac4f987995924831ecdac322a7bd7e48dd93d00dee4d4/pylikwid-0.2.6.tar.gz
Building wheels for collected packages: pylikwid
  Running setup.py bdist_wheel for pylikwid ... error
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-9q0CPy/pylikwid/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpnTJTF2pip-wheel- --python-tag cp27:
  Searching for LIKWID installation
  Using LIKWID libray at /usr/lib
  running bdist_wheel
  running build
  running build_ext
  building 'pylikwid' extension
  creating build
  creating build/temp.linux-x86_64-2.7
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/lib/include -I/usr/include/python2.7 -c pylikwid.c -o build/temp.linux-x86_64-2.7/pylikwid.o
  creating build/lib.linux-x86_64-2.7
  x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/pylikwid.o -L/usr/lib/lib -l: -o build/lib.linux-x86_64-2.7/pylikwid.so
  /usr/bin/x86_64-linux-gnu-ld: -l: kann nicht gefunden werden
  collect2: error: ld returned 1 exit status
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for pylikwid
  Running setup.py clean for pylikwid
Failed to build pylikwid
Installing collected packages: pylikwid
  Running setup.py install for pylikwid ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-9q0CPy/pylikwid/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-8dbyWm-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    Searching for LIKWID installation
    Using LIKWID libray at /usr/lib
    running install
    running build
    running build_ext
    building 'pylikwid' extension
    creating build
    creating build/temp.linux-x86_64-2.7
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/lib/include -I/usr/include/python2.7 -c pylikwid.c -o build/temp.linux-x86_64-2.7/pylikwid.o
    creating build/lib.linux-x86_64-2.7
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/pylikwid.o -L/usr/lib/lib -l: -o build/lib.linux-x86_64-2.7/pylikwid.so
    /usr/bin/x86_64-linux-gnu-ld: -l: kann nicht gefunden werden
    collect2: error: ld returned 1 exit status
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-9q0CPy/pylikwid/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-8dbyWm-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-9q0CPy/pylikwid/

Manual installation:

$ python setup.py build_ext -I /usr/local/include/ -L /usr/local/lib/ -R /usr/local/lib/
Searching for LIKWID installation
Using LIKWID libray at /usr/lib
running build_ext
building 'pylikwid' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/lib/include -I/usr/local/include/ -I/usr/include/python2.7 -c pylikwid.c -o build/temp.linux-x86_64-2.7/pylikwid.o
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/pylikwid.o -L/usr/lib/lib -L/usr/local/lib/ -Wl,-R/usr/local/lib/ -l: -o build/lib.linux-x86_64-2.7/pylikwid.so
/usr/bin/x86_64-linux-gnu-ld: -l: kann nicht gefunden werden
collect2: error: ld returned 1 exit status
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

There are two differences:

  1. The pip version now also displays 

    Searching for LIKWID installation
Using LIKWID libray at /usr/lib

  2. During the manual installation /usr/lib/lib after Using LIKWID libray at /usr/lib is now missing

liblikwid.so.4.3 is still missing in the command. I searched for the liblikwid files again and I also noticed that they are located at /usr/lib/x86_64-linux-gnu/. Could this be the culprit?

TomTheBear commented 6 years ago

Yes, the changed library path is the problem. You can wait until I fixed the install script or you do it manually: Adjust the paths in setup.py in Extension(): include_dirs = ["/usr/include"] # Check that one with find /usr -file likwid.h libraries = ["likwid"] library_dirs = ["/usr/lib/x86_64-linux-gnu/"]

The whole script fun is required in most cases only if multiple versions of LIKWID are installed to address exactly which version to link with.

d4k0 commented 6 years ago

Sorry for the late reply, I had technical problems with my laptop.

I just wanted to confirm that the installation now works with the changes to setup.py. likwid.h is in /usr/include, so nothing had to be changed.

However, I had to call python setup.py install with sudo. Otherwise I would get a permission error:

copying build/lib.linux-x86_64-2.7/pylikwid.so -> /usr/local/lib/python2.7/dist-packages
error: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pylikwid.so'

If I install the next version with the fixed install script using pip, will the manually installed version be overwritten? Or do I have to remove it somehow before?

TomTheBear commented 6 years ago

I'm happy to read that the manual installation worked for you.

I fixed the setup.py (https://github.com/RRZE-HPC/pylikwid/commit/9d7ea9d7a2172b173bde21b54514ff0eac2cf67f). It works on Ubuntu 18.04 with Python 2.7 and Python 3.x. sudo is required because you install pylikwid at system-level. If you install it in your home python setup.py install --home, you don't need sudo.

I'm not sure whether pip overwrites the manual installation but, as far as I know, pip is just calling python setup.py build && python setup.py install. So I assume it overwrites it but to be sure it would be better to remove the files manually. There should be the pylikwid.so and an .egg file.

d4k0 commented 6 years ago

I just wanted to report that the installation now works. Thanks for your quick help!