search

python / cpython

The Python programming language
https://www.python.org
Other
63.46k stars 30.39k forks source link

libpython3.12.a does not add libraries from the module, as in 3.11 #115489

Closed armanincredible closed 2 months ago

armanincredible commented 9 months ago

Bug report

Bug description:

I'm building python statically.

In python3.11.8 I have this output:

ar rcs libpython3.11.a Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/dynload_shlib.o     Modules/config.o Modules/main.o Modules/gcmodule.o Modules/atexitmodule.o  Modules/faulthandler.o  Modules/posixmodule.o  Modules/signalmodule.o  Modules/_tracemalloc.o  Modules/_codecsmodule.o  Modules/_collectionsmodule.o  Modules/errnomodule.o  Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o  Modules/itertoolsmodule.o  Modules/_sre/sre.o  Modules/_threadmodule.o  Modules/timemodule.o  Modules/_weakref.o  Modules/_abc.o  Modules/_functoolsmodule.o  Modules/_localemodule.o  Modules/_operator.o  Modules/_stat.o  Modules/symtablemodule.o  Modules/pwdmodule.o  Modules/_asynciomodule.o  Modules/_bisectmodule.o  Modules/_contextvarsmodule.o  Modules/_csv.o  Modules/_datetimemodule.o  Modules/_heapqmodule.o  Modules/_json.o  Modules/_lsprof.o Modules/rotatingtree.o  Modules/_multiprocessing/multiprocessing.o Modules/_multiprocessing/semaphore.o  Modules/_opcode.o  Modules/_pickle.o  Modules/_queuemodule.o  Modules/_randommodule.o  Modules/socketmodule.o  Modules/_statisticsmodule.o  Modules/_struct.o  Modules/_typingmodule.o  Modules/_zoneinfo.o  Modules/arraymodule.o  Modules/audioop.o  Modules/binascii.o  Modules/cmathmodule.o  Modules/mathmodule.o  Modules/mmapmodule.o  Modules/selectmodule.o  Modules/_elementtree.o  Modules/pyexpat.o  Modules/_blake2/blake2module.o Modules/_blake2/blake2b_impl.o Modules/_blake2/blake2s_impl.o  Modules/md5module.o  Modules/sha1module.o  Modules/sha256module.o  Modules/sha512module.o  Modules/_sha3/sha3module.o  Modules/cjkcodecs/_codecs_cn.o  Modules/cjkcodecs/_codecs_hk.o  Modules/cjkcodecs/_codecs_iso2022.o  Modules/cjkcodecs/_codecs_jp.o  Modules/cjkcodecs/_codecs_kr.o  Modules/cjkcodecs/_codecs_tw.o  Modules/cjkcodecs/multibytecodec.o  Modules/unicodedata.o  Modules/_posixsubprocess.o  Modules/_multiprocessing/posixshmem.o  Modules/fcntlmodule.o  Modules/grpmodule.o  Modules/ossaudiodev.o  Modules/resource.o  Modules/spwdmodule.o  Modules/syslogmodule.o  Modules/termios.o  Modules/_cryptmodule.o  Modules/_ctypes/_ctypes.o Modules/_ctypes/callbacks.o Modules/_ctypes/callproc.o Modules/_ctypes/stgdict.o Modules/_ctypes/cfield.o  Modules/_gdbmmodule.o  Modules/_lzmamodule.o  Modules/zlibmodule.o  Modules/xxsubtype.o  Modules/_ctypes/_ctypes_test.o Python/deepfreeze/deepfreeze.o Modules/getpath.o Python/frozen.o

But in 3.12 I have this:

ar rcs libpython3.12.a Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/typevarobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o  Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/assemble.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/flowgraph.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/ceval_gil.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/instrumentation.o Python/intrinsics.o Python/legacy_tracing.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/tracemalloc.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/perf_trampoline.o Python/dynload_shlib.o     Modules/config.o Modules/main.o Modules/gcmodule.o Modules/atexitmodule.o  Modules/faulthandler.o  Modules/posixmodule.o  Modules/signalmodule.o  Modules/_tracemalloc.o  Modules/_codecsmodule.o  Modules/_collectionsmodule.o  Modules/errnomodule.o  Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o  Modules/itertoolsmodule.o  Modules/_sre/sre.o  Modules/_threadmodule.o  Modules/timemodule.o  Modules/_typingmodule.o  Modules/_weakref.o  Modules/_abc.o  Modules/_functoolsmodule.o  Modules/_localemodule.o  Modules/_operator.o  Modules/_stat.o  Modules/symtablemodule.o  Modules/pwdmodule.o Python/deepfreeze/deepfreeze.o Modules/getpath.o Python/frozen.o

You can see that some files are missing, such as Modules/_posixsubprocess.o

Then these missed files builded into .so libraries. What doesn't happen during building python3.11, from logs:

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_asyncio              _bisect               _blake2            
_codecs_cn            _codecs_hk            _codecs_iso2022    
_codecs_jp            _codecs_kr            _codecs_tw         
_contextvars          _crypt                _csv               
_ctypes               _ctypes_test          _datetime          
_elementtree          _gdbm                 _heapq             
_json                 _lsprof               _lzma              
_md5                  _multibytecodec       _multiprocessing   
_opcode               _pickle               _posixshmem        
_posixsubprocess      _queue                _random            
_sha1                 _sha256               _sha3              
_sha512               _socket               _statistics        
_struct               _typing               _zoneinfo          
array                 audioop               binascii           
cmath                 fcntl                 grp                
math                  mmap                  ossaudiodev        
pyexpat               resource              select             
spwd                  syslog                termios            
unicodedata           zlib

As far as I can understand, because of this I get the following result on the builded python3.12:

Python 3.12.0 (main, Feb 14 2024, 22:34:23) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import math
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: Dynamic loading not supported
>>> import _posixsubprocess
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: Dynamic loading not supported
>>> import _operator
>>> import _stat
>>>

Also see the difference between logs :

In python 3.11 I got:

*** WARNING: renaming "_xxsubinterpreters" since importing it failed: Dynamic loading not supported
*** WARNING: renaming "_testcapi" since importing it failed: Dynamic loading not supported
*** WARNING: renaming "_testclinic" since importing it failed: Dynamic loading not supported
*** WARNING: renaming "_testinternalcapi" since importing it failed: Dynamic loading not supported
*** WARNING: renaming "_testbuffer" since importing it failed: Dynamic loading not supported
*** WARNING: renaming "_testimportmultiple" since importing it failed: Dynamic loading not supported
*** WARNING: renaming "_testmultiphase" since importing it failed: Dynamic loading not supported
*** WARNING: renaming "_xxtestfuzz" since importing it failed: Dynamic loading not supported
*** WARNING: renaming "readline" since importing it failed: Dynamic loading not supported
*** WARNING: renaming "_dbm" since importing it failed: Dynamic loading not supported
*** WARNING: renaming "nis" since importing it failed: Dynamic loading not supported
*** WARNING: renaming "_decimal" since importing it failed: Dynamic loading not supported
*** WARNING: renaming "_uuid" since importing it failed: Dynamic loading not supported

In 3.12:

maksetup: '_sha2' was handled by previous rule.
maksetup: '_sha3' was handled by previous rule.
maksetup: '_codecs_cn' was handled by previous rule.
maksetup: '_codecs_hk' was handled by previous rule.
maksetup: '_codecs_iso2022' was handled by previous rule.
maksetup: '_codecs_jp' was handled by previous rule.
maksetup: '_codecs_kr' was handled by previous rule.
maksetup: '_codecs_tw' was handled by previous rule.
maksetup: '_multibytecodec' was handled by previous rule.
maksetup: 'unicodedata' was handled by previous rule.
maksetup: '_posixsubprocess' was handled by previous rule.
maksetup: '_posixshmem' was handled by previous rule.
maksetup: 'fcntl' was handled by previous rule.
maksetup: 'grp' was handled by previous rule.
maksetup: 'ossaudiodev' was handled by previous rule.
maksetup: 'resource' was handled by previous rule.
maksetup: 'spwd' was handled by previous rule.
maksetup: 'syslog' was handled by previous rule.
maksetup: 'termios' was handled by previous rule.
maksetup: '_crypt' was handled by previous rule.
maksetup: '_ctypes' was handled by previous rule.
maksetup: '_gdbm' was handled by previous rule.
maksetup: '_lzma' was handled by previous rule.
maksetup: 'zlib' was handled by previous rule.
maksetup: '_ctypes_test' was handled by previous rule.
...

also

make: [Makefile:2382: libinstall] Error 1 (ignored)
Traceback (most recent call last):
  File "/home/arman/IspCode/jetos.python/ispras-python/lib/python3.12/compileall.py", line 17, in <module>
    import struct
  File "/home/arman/IspCode/jetos.python/ispras-python/lib/python3.12/struct.py", line 13, in <module>
    from _struct import *
ImportError: Dynamic loading not supported
make: [Makefile:2383: libinstall] Error 1 (ignored)
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/arman/IspCode/jetos.python/src/Python/Python-3.12.0/Lib/lib2to3/pgen2/driver.py", line 26, in <module>
    from . import grammar, parse, token, tokenize, pgen
  File "/home/arman/IspCode/jetos.python/src/Python/Python-3.12.0/Lib/lib2to3/pgen2/grammar.py", line 16, in <module>
    import pickle
  File "/home/arman/IspCode/jetos.python/src/Python/Python-3.12.0/Lib/pickle.py", line 33, in <module>
    from struct import pack, unpack
  File "/home/arman/IspCode/jetos.python/src/Python/Python-3.12.0/Lib/struct.py", line 13, in <module>
    from _struct import *
ImportError: Dynamic loading not supported
make: [Makefile:2384: libinstall] Error 1 (ignored)

It looks like Modules/Setup doesn't work in 3.12 like in 3.11... I attached Setup files into issue. Setup_3.11.txt Setup_3.12.txt

Building:

if [ $PYTHON_BASE_VERSION -ge 11 ]; then
    CONFIGURE_FLAGS="--without-ensurepip --with-trace-refs"
fi

if [ $PYTHON_BASE_VERSION -ge 12 ]; then
    CONFIGURE_FLAGS="${CONFIGURE_FLAGS} --with-system-libmpdec"
fi

CFLAGS="-fPIC -static -I${SRC_DIR}/dependencies/include" \
LDFLAGS="-L${SRC_DIR}/dependencies/lib -static" LIBS="-lz -lexpat" \
../"Python-${PYTHON_VERSION}/configure" ${CONFIGURE_FLAGS} --prefix="${SRC_DIR}/${PYTHON_DIR}" \
  --disable-shared --without-doc-strings --with-system-expat || xtfatal "Unable to configure Python"

if [ "$(uname)" = "Linux" ]; then
  make "$PARALLEL_NUM" LDFLAGS="-static" LINKFORSHARED=" " || xtfatal "Unable to make Python"
else
  make "$PARALLEL_NUM" || xtfatal "Unable to make Python"
fi
make install || xtfatal "Unable to install Python"

Can you please tell me how to add necessary objects to the static library 3.12?

CPython versions tested on:

3.11, 3.12

Operating systems tested on:

Linux

armanincredible commented 2 months ago

I found solution of this problem in configuration. Now I pass MODULE_BUILDTYPE=static and modules are building statically.