Libraries installed in /usr/local not detected?

[cbhushan]

I have few custom version of libraries installed in /usr/local. These are openssl, sqlite, libffi. I was able even able to build python 3.8 with these libraries.

portable-python was installed using pipx (in the same python 3.8 I built above). Now when I try to use portable-python to build portable 3.7, I get following error.

# portable-python --version
portable-python, version 1.7.0

# portable-python build 3.7.17
INFO Created folder /data_no_backup2/tmp/portpy/Python-3.8.17/build/logs
INFO portable-python v1.7.0, current folder: /data_no_backup2/tmp/portpy/Python-3.8.17
INFO selected: None (0 modules)
INFO no config
INFO Platform: linux-x86_64
INFO Build report:
 libffi       3.4.4    failed  broken, can't compile without libffi-dev
 zlib         1.2.13   shared  has /usr/include/zlib.h
 xz           5.4.3    shared  has /usr/include/lzma.h
 bzip2        1.0.8    shared  has /usr/include/bzlib.h
 readline     8.2      shared  has /usr/include/readline/readline.h
  +- ncurses  6.4              sub-module of readline:8.2
 openssl      1.1.1u   absent  no {include}/openssl/ssl.h
 sqlite       3.42.0   absent  no ['{include}/sqlite3.h']
 bdb          6.2.32   absent  no {include}/dbm.h
 gdbm         1.23     shared  has /usr/include/gdbm.h
 uuid         1.0.3    shared  has /usr/include/uuid/uuid.h
 tkinter      8.6.10   shared  has /usr/include/tk.h
  +- tcl      8.6.10           sub-module of tkinter:8.6.10
  +- tk       8.6.10           sub-module of tkinter:8.6.10
  +- tix      8.4.3.6          sub-module of tkinter:8.6.10
ERROR Problematic modules: libffi:3.4.4
INFO Overall compilation took 2 ms 768 \u03bcs

libffi.s0 files are in /usr/local as seen below, and so are other include files:

[root@c49d16031019 Python-3.8.17]# find /usr -iname '*libffi*'
/usr/lib64/libffi.so.5
/usr/lib64/libffi.so.5.0.6
/usr/local/lib64/libffi.so.8
/usr/local/lib64/libffi.so
/usr/local/lib64/libffi.la
/usr/local/lib64/libffi.a
/usr/local/lib64/libffi.so.8.1.2
/usr/local/share/info/libffi.info
/usr/local/lib/pkgconfig/libffi.pc

[root@c49d16031019 Python-3.8.17]# ls -1 /usr/local/include/
ffi.h
ffitarget.h
openssl
sqlite3ext.h
sqlite3.h

Is there a way to tell portable-python to look for libraries in /usr/local? Or any other suggestions to avoid above error?

[zsimic]

portable-python tries hard to avoid things like /usr/local, because that would mean the compiled binary would imply users to have the same (or compatible) /usr/local as well.

For libffi, it will use the system one if you have libffi-dev (as the upstream setup does). portable-python can however also compile statically libffi, so that the produced binary works wether or not target user has /usr/local/lib*

Why don't you simply compile libffi statically? You can do so via config or by adding -m libffi on the command line

[zsimic]

OOh, "broken, can't compile without libffi-dev" -> that reminds me, I indeed observed that even if you do want libffi compiled statically, for whatever reason the upstream setup will fail compilation, libffi-dev is unfortunately needed.

I don't know why, probably some bug in upstream ./configure

[cbhushan]

Thank you for the response. I am actually trying to get things working in older CentOS 6.x environment, and upstream libffi-devel is too old for python 3.8. So, I had to build my own version in /usr/local. And the same case for openssl, sqlite.

OOh, "broken, can't compile without libffi-dev" -> that reminds me, I indeed observed that even if you do want libffi compiled statically, for whatever reason the upstream setup will fail compilation, libffi-dev is unfortunately needed. I don't know why, probably some bug in upstream ./configure

Based on your above response, it seems there is no way to get portable-python work in my scenario with older CentOS 6.x environment. If so, feel free to close the issue or mark as unfixable etc.

PS: BTW, I think that it would help users to understand difference between portable-python and python-build-standalone (Github link). My understanding, which may be wrong, is that portable-python targets to create binaries that will be used on same/very similar environment. Correct?

[zsimic]

Good point, that's a good doc to have indeed. I looked initially at python-build-standalone, but I was having trouble to make it successfully compile, and it didn't create portable binaries, I submitted a PR at the time here for one issue preventing portability: https://github.com/indygreg/python-build-standalone/pull/61

It looks like it was eventually fixed indeed now.

I'd need to re-examine python-build-standalone to list exact differences, originally it was:

• portable-python is "simpler", doesn't try to support cross-compiling
• it can produce portable binaries (that can be unpacked in any folder and work from there)
• it can optionally statically compile openssl, libffi, etc

[zsimic]

Try installing libffi-devel, and then instruct portable-python to statically compile libffi (with for example -m libffi) -> it should work then. Ie: you will get a statically compiled libffi (the libffi-devel will just be there to satisfy the upstream cpython ./configure at compile time - I'm sorry I couldn't find what exactly was making it consistently fail without it otherwise)