Closed eacf80b2-9081-4882-a7f5-27e9644c3853 closed 4 years ago
This is something that's becoming more and more of a problem as more people get systems that run both 32-bit and 64-bit code. There are patches (926209 858809) in the tracker to resolve this issue, I don't know exactly what state they are in. They seem fairly old, one is closed and one is still open.
The Fedora RPMs include the following patches for lib64 (attached).
Any thoughts on what we need to do to allow building lib64 as a part of the standard release? Or do we just want to wait for these terrible transition days to end and rely on 32+64 packagers to deal with it?
Here's a short run-down of the situation: Some systems can run both 32 and 64 bit software. On these systems you *CAN* install a native 64-bit tool-chain, but some people set up both 32 and 64 bit software on these systems for compatibility. You can do 32+64 in a couple of ways, one is to set up a "chroot" environment for the secondary (typically 32-bit) set of libraries and programs. The other is to run the system with 64-bit libraries in /usr/lib64 and 32-bit in /usr/lib (because most software that needs compatibility would be running against /usr/lib, not /usr/lib32).
It's a kludge to be sure, but if you need any legacy software that you do not have source for, this is a way of getting 64-bits while still retaining the ability to run 32-bit This is not a problem for systems which only run 64-bit code, nor is it a problem for these 32+64 systems which are running only 64-bit distributions on them.
Thoughts? Sean
The problem with 64 bit builds should be discussed before we release 2.6. Another bug day topic.
Can someone update the priority so that this is looked at before the 2.6 release?
Placing the entire library tree in /usr/lib64 is wasteful on dual 32/64bit installation, but placing just the C modules there is contrary to python import logic and may cause problems to relative imports.
I have suggested what I believed was a workable solution: have 64-bit python search lib64-dynload subdirectories instead of lib-dynload.
See http://mail.python.org/pipermail/python-dev/2007-April/072653.html
Currently $(prefix)/pythonX.Y/lib-dynload is inserted in the sys.path,
but I think it would be better to handle this inside the importer in a
way similar to how the importer looks for both foo.so and foomodule.so
when importing foo. This would allow submodules and user modules treated
the same way.
Similar problem report: http://bugs.python.org/issue1019715
3rd party C modules are put in site-packages, so just having importer of 64-bit python look at lib64-dynload is not enough for solving this.
To work around this problem, I did some hacks on my local Python to look at lib and lib64. It worked, but just as belopolsky said, this is wasteful and ugly.
for completenes, here's a patch that's in use in SUSE. it's advantage over Fedora's is that it works on both 32bit and 64bit installs
both patches assume that everybody uses lib64 for 64bit libs, which is not true for Debian/Ubuntu. Even the FHS doesn't mandate the use of lib64.
well in our patch, at least, the directory is governed by sys.lib which is defined through configure. i don't understand the configure language well enough, but i'd assume that making it parametrized isn't too hard?
I think this is duplicate of bpo-858809.
If I understand correctly, using lib32 or lib64 is a kludge. Debian and Ubuntu want to come up with a better way to do this: http://wiki.debian.org/ReleaseGoals/MultiArch https://wiki.ubuntu.com/MultiarchSpec
Kind regards.
thanks for the info; however, it still means that python needs to learn to live in places other than "/usr/lib"
jan matejek wrote:
jan matejek \jmatejek@suse.cz\ added the comment:
thanks for the info; however, it still means that python needs to learn to live in places other than "/usr/lib"
The main problem is that Python's configuration system is not geared up to having the lib directories for platform dependent and platform independent parts use different names.
It currently only supports using different path *prefixes* for such setups (--prefix and --exec-prefix), e.g. /usr and /usr64 would work just fine. It doesn't follow --libdir.
Note that Tarek is currently working on a cleanup of the installation schemes (see distutils/commands/install.py, distutils/sysconfig.py and site.py) which will then also be used by site.py to setup sys.path.
A new modules will unite all these settings, so this should be the target of any patches regarding installation and path lookup schemes.
BTW: The "sys.lib" setting mentioned on the tracker is not standard. The naming also doesn't look right, since the name of the "lib" directory path component is an OS feature, not a system one that you configure. os.lib_dir would be more appropriate.
Adding Tarek to the ticket.
Please note another aspect of this problem will bite all Python developers on Ubuntu 11.04. With the introduction of multiarch, not all stdlib Python extension modules can be built out of the box, as seen here:
https://bugs.launchpad.net/ubuntu/+source/db4.8/+bug/738213/comments/13
Ubuntu's source package was hacked to make things work, by calling out to dpkg-architecture and adding the resulting directories to library_dirs and include_dirs search paths. That patch would obviously have to be modified at the very least to be robust on non-Debian/Ubuntu platforms.
I'm not sure what the right solution is for upstream.
Here's a fix that works for me on Ubuntu 11.04.
I should note that I'd love to backport this to Python 3.2, 3.1, 2.7 and 2.6 since none of them can build entirely now on multiarch systems. Since it only affects search order in the build process, one could argue that it's not a new feature :).
Barry: does it allow to install Python into /usr/lib/whateverarch, or is it just a partial fix for something slightly unrelated to this issue?
A proper fix is to introduce sys.libdir, which would be controllable by --libdir=${value} option of configure
. If --libdir=${value} is not passed, then sys.libdir would default to sys.prefix + "/lib".
sysconfig, distutils etc. would have to use sys.libdir.
On Mar 29, 2011, at 07:28 PM, Antoine Pitrou wrote:
Antoine Pitrou \pitrou@free.fr\ added the comment:
Barry: does it allow to install Python into /usr/lib/whateverarch, or is it just a partial fix for something slightly unrelated to this issue?
Antoine, you're right that the problem and fix I'm talking about is probably different than the original bug report. I really should create a new issue, since mine isn't about building multiarch for Python, but instead just being able to build Python on a multiarch system. My problem is thankfully much simpler.
I retract my patch for this bug because the issue described here is actually different than the one I want to fix. See bpo-11715 for the problem of building Python on multiarch Debian and Ubuntu (e.g. Ubuntu 11.04).
On Mar 29, 2011, at 07:38 PM, Arfrever Frehtes Taifersar Arahesis wrote:
A proper fix is to introduce sys.libdir, which would be controllable by --libdir=${value} option of
configure
. If --libdir=${value} is not passed, then sys.libdir would default to sys.prefix + "/lib". sysconfig, distutils etc. would have to use sys.libdir.
Please note that I'm not interested (right now \<wink>) in building Python multiarch, but building Python *on* multiarch. So I think simply adding the right search paths to setup.py is the right way to go, and I've opened a separate issue for it.
heh, that's easy, just add the multiarch id to the extension name ;-)
On 29.03.2011 21:28, Antoine Pitrou wrote:
> Barry: does it allow to install Python into /usr/lib/whateverarch, no, it looks for headers and libraries in more directories. But really, this whole testing for paths is wrong. Just use the compiler to search for headers and libraries, no need to check these on your own.
> or is it just a partial fix for something slightly unrelated to this issue?
IMO, unrelated to the original report.
On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:
no, it looks for headers and libraries in more directories. But really, this whole testing for paths is wrong. Just use the compiler to search for headers and libraries, no need to check these on your own.
You're probably right about that, but reimplementing Python's build system is out of scope for right now. ;) For one thing, doing so wouldn't allow me to backport to older Pythons, which I really want to do.
On Mar 29, 2011, at 10:11 PM, Matthias Klose wrote:
heh, that's easy, just add the multiarch id to the extension name ;-)
Clever! :)
I currently think that sys.libdir should be only basename of libdir (e.g. "lib" or "lib64") to allow to easily use it with something else than sys.prefix.
On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:
no, it looks for headers and libraries in more directories. But really, this whole testing for paths is wrong. Just use the compiler to search for headers and libraries, no need to check these on your own.
Do all compilers provide this info, including Windows ones? If so, that would be a nice feature for distutils2.
Arfrever Frehtes Taifersar Arahesis (Arfrever) on 2012-05-12 19:00
I currently think that sys.libdir should be only basename of libdir (e.g. "lib" or "lib64") to allow to easily use it with something else than sys.prefix.
With the new sysconfig module I don’t think we need to clutter sys with another attribute.
Éric Araujo wrote:
Éric Araujo \merwok@netwok.org\ added the comment:
On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote: > no, it looks for headers and libraries in more directories. But really, this > whole testing for paths is wrong. Just use the compiler to search for headers > and libraries, no need to check these on your own.
Do all compilers provide this info, including Windows ones? If so, that would be a nice feature for distutils2.
This only works for a handful of system library paths, not the extra ones that you may need to search for local installations of libraries and which you have to inform the compiler about :-)
Many gcc installations, for example, don't include the /usr/local or /opt/local dir trees in the search. On Windows, you have to run the correct vc*.bat files to have the paths setup and optional software rarely adds the correct paths to LIB and INCLUDE.
The compiler also won't help with the problem Sean originally pointed to: building software on systems that can run both 32-bit and 64-bit and finding the right set of libs to link at.
Another problem is finding the paths to the right version of a library (both include files and corresponding libraries).
While it would be great to have a system tool take care of setting things up correctly, I don't know of any such tool, so searching paths and inspecting files using REs appears to be the only way to build a general purpose detection scheme.
mxSetup.py (included in egenix-mx-base) uses such a scheme, distutils has one too.
Is this issue still relevant?
Yes. We still have a patch for two things:
"$prefix/lib" is hardcoded in many places. Lib64 is probably not going away anytime soon, so it would be nice if this was solved once and for all ;) The good thing is that with sysconfig, we don't have to do much beyond teaching sysconfig about the right values.
To reiterate, our current solution is to introduce "sys.lib" (and "sys.arch", but that is never used anymore) that is either "lib" or "lib64", and use this in place of the string "lib" wherever appropriate. We find the value for sys.lib through configure magic.
the patch in msg202343 is wrong, hardcoding lib64 on Debian/Ubuntu. At least the configure check should check for lib64 as a directory and not a symlink, and only then default to lib64.
two other issues with the patch:
I would like to see any new OS-dependent locations in the sysconfig module, not the sys module.
Please don't depend on uname for the autoconf check, but on the gnu host triplet.
Please don't add another `arch' attribute to sys. We already have enough of these in sysconfig.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Dne 7.11.2013 16:56, Matthias Klose napsal(a):
Matthias Klose added the comment:
the patch in msg202343 is wrong, hardcoding lib64 on Debian/Ubuntu.
This patch is provided for reference only - it works for us at SUSE. I'll be happy to spend some time improving it for general usage, if this has any chance of being commited.
At least the configure check should check for lib64 as a directory and not a symlink, and only then default to lib64.
Maybe this should be detected differently altogether. Perhaps by working with LIBDIR, which is known to configure?
two other issues with the patch:
- I would like to see any new OS-dependent locations in the sysconfig module, not the sys module.
how would you propose to put the value into sysconfig in the first place? It seems to rely heavily on existing attributes from sys.
Please don't depend on uname for the autoconf check, but on the gnu host triplet.
Please don't add another `arch' attribute to sys. We already have enough of these in sysconfig.
We don't use this one anymore, and i'm not entirely sure that we ever did. I am happy to drop it.
----------
___ Python tracker \report@bugs.python.org\ \http://bugs.python.org/issue1294959\
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBAgAGBQJSe7pdAAoJEIskb84DCy7LvwcP/2n74K2XDsRu7K6OV9S4SzDa v7vpDVhAgTBQlHglY+wavUQU2WLBlGyVEk2xHDV8WdI4zU7rAbn7XAW5URxznctq t/Ptvt0IsDAqONrF8ezg8/eTUkcP3nV2Hk90RNe0gliDH6uc0wekKUZzVaTObO1L 3vM8XfEtTQstmK1VxQVpYolUPZm8n7Fe8NEPA6A8bu8CU736cg+wWdbDrr6Mjowo OuO4b56J1P3BIQkBcOLe3mH20Bv8O03P9iNADwYHUOayvgthFWCmoDzh0Y1dQa9/ ynT+G9BuYyXOli6Yr15W0L8OFU+nwxByK81lEClz6UonCvoStnEWnXIN3JYW15Yb rNwb5HKNmKB16yx/RuV3WCvlKbg6ziMlfWGW6qTA1g0P0ivU+sRVQXv5gI8NHcQ9 /4jmaUh7Dr1T4KHujI57Z99kLQHvSlHEM3v4aT96IZNaPghkA+e7TjhMdHmvYtQz YoY75FIy0xVStdXzw8zbM2LVlKp8vxncrjYbuzJYgG1jiYwmF6gDoztXRVP/zwei PypIiui4QaQc32V5dCwpQYpAvpgCVHm6sGSO0HbLWSUK71M8f1YU2BPwvglHb6jh N9tmYfmahvQSyIFOZdb4C6HLlzEezENdYYIf7oMW07z8SOOLU+8eKP13dp6NWINP HCSU34LLYTfwnQ+27aMk =raBO -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On Nov 07, 2013, at 04:05 PM, jan matejek wrote:
> - I would like to see any new OS-dependent locations in the sysconfig > module, not the sys module.
how would you propose to put the value into sysconfig in the first place? It seems to rely heavily on existing attributes from sys.
Actually, I think it mostly parses these values from the installed Makefile. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux)
iQIcBAEBCAAGBQJSe8T6AAoJEBJutWOnSwa/6oAQAJM3IksGydXk9CEp7rbSrHea DvYUccmyon1xrmu5RjWnz6ZbJdWlFwx8ouFpbzmZvfAF8E2m0HliNzW+/w28sik5 F37p5/7hScQ2x/AcmdnVrzDotMkvcvMILsCDIhSy/nPIWkI4hAuRGLdPJmgiE3HE b3hjQDCgsVDrp8arJioepx6xWSMpE1dQ/KsL6DDuNl3rUNVR1E1jTYEqY12+fOjc Mcgz6lVndyZreY958iQE/UR7MKoW5lcLnin44PtMYrizwWv82kgwhkqU5tubnTBD m9tD16uCz7xvATM5XI8nmXeLcLMSMfUdaG+4ny//cIjDKYIC6XXoZvCgE7iSEws1 pEIQCmrs6mpk6d83Yz/XDXc4OqjqC+tUPY1TGNqAI/nm416uoKUuO/f1eU397EV+ 2RqJqev8Ho8Sgk7skFJGwcCfTO4yDR40+0wm3u2BiM9bTcnGiJaC7z2TAp9eb4Qs jo/cmYi3BbqPu9Xx3P4oX11NHmjTPBUcZjqsJa8w8q3lf9r5haE5EqlLaNgDnGtL efu7OMom2yQHXdwIJ2efmefjoby812uNFSbTiMvDxZTVCCUyBczBT/Q7gu/4S9Ks Mv3oY1bkq6qAXKKOwzKoblzHJ6VW+A3Rn15Lh6Tb2kj1pTbdS9fFJASWr6CprjWi XbCdez4dMhd+PGwgxHs6 =/r/D -----END PGP SIGNATURE-----
On Nov 07, 2013, at 03:40 PM, jan matejek wrote:
To reiterate, our current solution is to introduce "sys.lib" (and "sys.arch", but that is never used anymore) that is either "lib" or "lib64", and use this in place of the string "lib" wherever appropriate. We find the value for sys.lib through configure magic.
PEP-421 added sys.implementation, which contains provisions for implementation-specific additions. So a better place to put these non-standard values is sys.implementation._lib and sys.implementation._arch, either instead of or in addition to sysconfig variables.
I disagree about sys.implementation. It's useless and wrong for cross builds. Please use sysconfig instead. What sysconfig is maybe missing is a set of variables which you can rely on.
configure.ac should accept an option, which allows to set any custom libdir.
Examples (architecture: libdir) in Gentoo: x32: libx32 mips o32: lib mips n32: lib32 mips n64: lib64
On Nov 07, 2013, at 04:56 PM, Matthias Klose wrote:
I disagree about sys.implementation. It's useless and wrong for cross builds. Please use sysconfig instead. What sysconfig is maybe missing is a set of variables which you can rely on.
Agreed that sysconfig is a better place for more general values. My point was that if OpenSUSE wants to carry deltas that are specific only to its platform, then sys.implementation._\<whatever> is the standard place to put it.
Is there agreement on what needs doing here? I'd like to see this into 3.5 before it reaches its 10th birthday :)
Also see bpo-15631
at this again, when porting SUSE patches to 3.6.0 :) ( :( )
Last time there was a discussion, Barry suggested using sysconfig variables to find the proper libdir. Trouble is, to fill out the variables, sysconfig itself uses two sources: a) compiled-in information from the binary, and b) Makefile items Makefile seems an obvious location for storing the libdir info, except, well, to FIND the makefile in the first place, we are using sysconfig variables.... you see where this is heading.
So, given that sysconfig is the correct Primary Source for libdir info, we have three options to get it there:
And options (2) and (3) seem wrong, so i'm going with (1) for now
The downstream patch we currently use in Fedora [0].
[0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/plain/00102-lib64.patch
Attached is a patch that I'd like to propose for inclusion.
It introduces a new configure option "--with-custom-platlibdir=basename $libdir
. This is converted to makefile variable "platlibdir", which is used in getpath.c to generate value of the lib_python variable.
sysconfig and distutils.sysconfig retrieve the variable from makefile (i learned about existence of _sysconfigdata so that is where the variable now lives) to correctly substitute in posix_prefix (and unix_prefix for distutils install) scheme.
Sysconfig is then used in pydoc and trace module, instead of locally calculating paths. Perhaps distutils.sysconfig should also use more of sysconfig instead of duplicating the functionality?
As it stands, the python stdlib is installed into /usr/$platlibdir, because I changed SCRIPTDIR to $libdir instead of $prefix/lib. Maybe this should be also separately configurable? In any case, third-party modules get installed either into /usr/lib or /usr/$platlibdir depending on whether they are arch-independent or not. Hence the final modification in site.py that adds /usr/lib to search path on systems where $platlibdir != "lib".
This caused test_sysconfig to fail because it checks that posix_user and posix_prefix schemes generate "similar" paths. I have left the posix_user scheme alone, so all its paths are based on "lib", where the posix_prefix scheme uses $platlibdir. Maybe posix_user scheme should be modified as well? Anyway, for now, I have accounted for the changes in the test.
What now? (please don't make me write a PEP please don't make me write a PEP please don't make......)
I don't think we need a PEP for this, however I would like to name the new macro libsubdir, or sublibdir, to make it clear it's just the libdir component. Or make the value absolute, like it already is for libdir ...
On Jan 18, 2017, at 03:57 PM, Matthias Klose wrote:
I don't think we need a PEP for this
Correct. Generally we don't need PEPs for build system changes.
Pinging here. Is there some way to push the issue forward?
the patch looks ok. was it tested on a Debian or Ubuntu system to produce the same layout with this patch and without the new configure option?
I closed bpo-858809 as a duplicate of this issue.
I marked bpo-15631 as a duplicate of this issue.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields: ```python assignee = None closed_at =
created_at =
labels = ['type-bug', '3.9', 'build']
title = 'Add sys.platlibdir and configure --with-platlibdir to use /usr/lib64 on Fedora and SuSE'
updated_at =
user = 'https://bugs.python.org/jafo'
```
bugs.python.org fields:
```python
activity =
actor = 'eric.araujo'
assignee = 'none'
closed = True
closed_date =
closer = 'vstinner'
components = ['Build']
creation =
creator = 'jafo'
dependencies = []
files = ['1777', '1778', '1779', '14726', '32529', '46301', '48892', '48962', '48966']
hgrepos = ['12']
issue_num = 1294959
keywords = ['patch']
message_count = 96.0
messages = ['26314', '59310', '64592', '64593', '81099', '82929', '91559', '91769', '91781', '92541', '94941', '94982', '94986', '94989', '132424', '132503', '132505', '132506', '132511', '132535', '132537', '132538', '132540', '132542', '132544', '132546', '160482', '160671', '160691', '202341', '202343', '202354', '202359', '202365', '202366', '202367', '202368', '202378', '222643', '242608', '285126', '285222', '285556', '285737', '285738', '307231', '308555', '361844', '361847', '361849', '361850', '361851', '361853', '361854', '361882', '361888', '361890', '361893', '361894', '361897', '361899', '361900', '361901', '361902', '361903', '361907', '361909', '362324', '363288', '363441', '363728', '363811', '363816', '363894', '363896', '363900', '364024', '364104', '364105', '369323', '369326', '369327', '369352', '369354', '369360', '369365', '369368', '369467', '369744', '370238', '370242', '370248', '370704', '371008', '382964', '399339']
nosy_count = 26.0
nosy_names = ['lemburg', 'barry', 'doko', 'jafo', 'jcea', 'belopolsky', 'vstinner', 'matejcik', 'tarek', 'ivazquez', 'mcepl', 'eric.araujo', 'Arfrever', 'akitada', 'andybuckley', 'petr.viktorin', 'catalin.iacob', 'mgorny', 'python-dev', 'piotr.dobrogost', 'eitan.adler', 'cstratak', 'hroncok', 'miss-islington', 'pillarsdotnet', 'carlos.velasco']
pr_nums = ['3698', '11755', '18381', '18476', '18482', '18483', '18917', '18960', '20332', '20497', '20725']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1294959'
versions = ['Python 3.9']
```