Closed mstorsjo closed 1 year ago
Just wondering: Is there something we could test in CI that would have failed without this? (so I can create an issue for it)
Just wondering: Is there something we could test in CI that would have failed without this? (so I can create an issue for it)
Hmm, not sure.
In the case of regular msys2 setups, you have libffi installed in the toolchain default include directory, so everything is found pretty much automatically. When cross compiling python for llvm-mingw, we build libffi into a separate prefix directory and configure cpython-mingw with CFLAGS="-I$PREFIX/include"
- https://github.com/mstorsjo/llvm-mingw/blob/master/build-python.sh#L148.
So to test this case, you'd need to make sure that libffi isn't installed systemwide, but then install it in a custom directory and point to it this way.
In this case, libffi wasn't picked up from there. Additionally, the failure mode was quite strange; the build did complete, but the log printed messages that the _ctypes
module wasn't built. At runtime I then got errors when using any python function that internally uses _ctypes
.
In the case of regular msys2 setups, you have libffi installed in the toolchain default include directory, so everything is found pretty much automatically. When cross compiling python for llvm-mingw, we build libffi into a separate prefix directory and configure cpython-mingw with
CFLAGS="-I$PREFIX/include"
- https://github.com/mstorsjo/llvm-mingw/blob/master/build-python.sh#L148.
It seemed like python was moving towards using pkg-config for most of this, I think this is why the tcl/tk patch was no longer necessary. I guess it's not finding libffi this way though?
Additionally, the failure mode was quite strange; the build did complete, but the log printed messages that the
_ctypes
module wasn't built. At runtime I then got errors when using any python function that internally uses_ctypes
.
This is "by design" of the python build process. It is pretty annoying when it "successfully" skips building standard modules that you thought it was going to include. I added a hack to MSYS2-packages python recipe to try to detect this when it bit me there: https://github.com/msys2/MSYS2-packages/blob/ddf03909a6a5bbf3f0332953cbc3939b542502b4/python/PKGBUILD#L124-L125
In the case of regular msys2 setups, you have libffi installed in the toolchain default include directory, so everything is found pretty much automatically. When cross compiling python for llvm-mingw, we build libffi into a separate prefix directory and configure cpython-mingw with
CFLAGS="-I$PREFIX/include"
- https://github.com/mstorsjo/llvm-mingw/blob/master/build-python.sh#L148.It seemed like python was moving towards using pkg-config for most of this, I think this is why the tcl/tk patch was no longer necessary. I guess it's not finding libffi this way though?
Hmm, maybe that would work? The python cross compilation that a user contributed to llvm-mingw didn't use that for setting it up so far.
In https://github.com/msys2-contrib/cpython-mingw/blob/mingw-v3.11.4/setup.py#L1476-L1496 it first looks for LIBFFI_INCLUDEDIR
(dunno if this gets set if it is found with pkg-config?), and if not set, it looks for ffi.h
manually among the known include directories.
Additionally, the failure mode was quite strange; the build did complete, but the log printed messages that the
_ctypes
module wasn't built. At runtime I then got errors when using any python function that internally uses_ctypes
.This is "by design" of the python build process. It is pretty annoying when it "successfully" skips building standard modules that you thought it was going to include. I added a hack to MSYS2-packages python recipe to try to detect this when it bit me there: https://github.com/msys2/MSYS2-packages/blob/ddf03909a6a5bbf3f0332953cbc3939b542502b4/python/PKGBUILD#L124-L125
Ah, "nice"!
Another bit which seems weird but apparently "by design", is some missing dependency for parallel builds. In one (highly parallel) build I saw that it started building those modules before it had completed linking libpython3.11.dll
, so all the native modules it tried to build failed with a -lpython3.11 not found
or similar. But then later on in the build process, it retried building them again, and then they succeeded.
In https://github.com/msys2-contrib/cpython-mingw/blob/mingw-v3.11.4/setup.py#L1476-L1496 it first looks for
LIBFFI_INCLUDEDIR
(dunno if this gets set if it is found with pkg-config?), and if not set, it looks forffi.h
manually among the known include directories.
Looks like LIBFFI_INCLUDEDIR does indeed come from pkg-config in configure https://github.com/msys2-contrib/cpython-mingw/blob/2b2d1a5985b12c4c053e99d65809e1ff7dbd07ec/configure.ac#L3869-L3874
But I don't know if pkg-config would be usable in your (bootstrappping-type) scenario anyway. When I was testing cross-compiling this I had to add to PKG_CONFIG_PATH so it could find ncurses. You may not care about having ncurses support in the python you are building, as much as you care about _ctypes.
When cross compiling python for llvm-mingw, we build libffi into a separate prefix directory and configure cpython-mingw with
CFLAGS="-I$PREFIX/include"
- https://github.com/mstorsjo/llvm-mingw/blob/master/build-python.sh#L148.
thanks, I've created https://github.com/msys2-contrib/cpython-mingw/issues/146
In https://github.com/msys2-contrib/cpython-mingw/blob/mingw-v3.11.4/setup.py#L1476-L1496 it first looks for
LIBFFI_INCLUDEDIR
(dunno if this gets set if it is found with pkg-config?), and if not set, it looks forffi.h
manually among the known include directories.Looks like LIBFFI_INCLUDEDIR does indeed come from pkg-config in configure
But I don't know if pkg-config would be usable in your (bootstrappping-type) scenario anyway. When I was testing cross-compiling this I had to add to PKG_CONFIG_PATH so it could find ncurses. You may not care about having ncurses support in the python you are building, as much as you care about _ctypes.
Thanks! I tested setting PKG_CONFIG_LIBDIR
now, and that did indeed seem to fix the issue, so I can do without this patch for now. That's probably the best way forward, instead of relying on parsing $CFLAGS
.
(For cross compilation, pkg-config works fine, but one has to remember to use PKG_CONFIG_LIBDIR
, which overrides all default local builtin directories, while setting things with PKG_CONFIG_PATH
lets pkg-config still find things in the default system directories like /usr/lib/pkgconfig
.)
When cross compiling python for llvm-mingw, we build libffi into a separate prefix directory and configure cpython-mingw with
CFLAGS="-I$PREFIX/include"
- https://github.com/mstorsjo/llvm-mingw/blob/master/build-python.sh#L148.thanks, I've created #146
Thanks, I appreciate the effort you put into maintaining a working cross compilation setup of python for mingw! That allows me to generally not need to spend time/focus/effort on this :-)
(For cross compilation, pkg-config works fine, but one has to remember to use
PKG_CONFIG_LIBDIR
, which overrides all default local builtin directories, while setting things withPKG_CONFIG_PATH
lets pkg-config still find things in the default system directories like/usr/lib/pkgconfig
.)
Thanks, I didn't know about that variable and its difference from PKG_CONFIG_PATH
- I just used the one I heard of, and it worked so I didn't look any further :grin:
(For cross compilation, pkg-config works fine, but one has to remember to use
PKG_CONFIG_LIBDIR
, which overrides all default local builtin directories, while setting things withPKG_CONFIG_PATH
lets pkg-config still find things in the default system directories like/usr/lib/pkgconfig
.)Thanks, I didn't know about that variable and its difference from
PKG_CONFIG_PATH
- I just used the one I heard of, and it worked so I didn't look any further 😁
Yep, it generally works fine though as long as you provide everything you need. But if there's a library installed in the host form, but not wanted for the cross compilation target, and the build script looks for it, you need to set the LIBDIR
version.
In cpython-mingw-3.10, this was part of a patch named "allow static tcltk".
This is relevant if cross compiling cpython-mingw.