msys2-contrib / cpython-mingw

A friendly fork of CPython which adds support for Mingw-w64 + clang/gcc. See https://github.com/msys2-contrib/cpython-mingw/wiki for details
https://github.com/msys2-contrib/cpython-mingw/wiki
Other
38 stars 12 forks source link

Allow picking up include/lib dirs from CFLAGS/LDFLAGS for Windows targets #145

Closed mstorsjo closed 1 year ago

mstorsjo commented 1 year ago

In cpython-mingw-3.10, this was part of a patch named "allow static tcltk".

This is relevant if cross compiling cpython-mingw.

lazka commented 1 year ago

Just wondering: Is there something we could test in CI that would have failed without this? (so I can create an issue for it)

mstorsjo commented 1 year ago

Just wondering: Is there something we could test in CI that would have failed without this? (so I can create an issue for it)

Hmm, not sure.

In the case of regular msys2 setups, you have libffi installed in the toolchain default include directory, so everything is found pretty much automatically. When cross compiling python for llvm-mingw, we build libffi into a separate prefix directory and configure cpython-mingw with CFLAGS="-I$PREFIX/include" - https://github.com/mstorsjo/llvm-mingw/blob/master/build-python.sh#L148.

So to test this case, you'd need to make sure that libffi isn't installed systemwide, but then install it in a custom directory and point to it this way.

In this case, libffi wasn't picked up from there. Additionally, the failure mode was quite strange; the build did complete, but the log printed messages that the _ctypes module wasn't built. At runtime I then got errors when using any python function that internally uses _ctypes.

jeremyd2019 commented 1 year ago

In the case of regular msys2 setups, you have libffi installed in the toolchain default include directory, so everything is found pretty much automatically. When cross compiling python for llvm-mingw, we build libffi into a separate prefix directory and configure cpython-mingw with CFLAGS="-I$PREFIX/include" - https://github.com/mstorsjo/llvm-mingw/blob/master/build-python.sh#L148.

It seemed like python was moving towards using pkg-config for most of this, I think this is why the tcl/tk patch was no longer necessary. I guess it's not finding libffi this way though?

Additionally, the failure mode was quite strange; the build did complete, but the log printed messages that the _ctypes module wasn't built. At runtime I then got errors when using any python function that internally uses _ctypes.

This is "by design" of the python build process. It is pretty annoying when it "successfully" skips building standard modules that you thought it was going to include. I added a hack to MSYS2-packages python recipe to try to detect this when it bit me there: https://github.com/msys2/MSYS2-packages/blob/ddf03909a6a5bbf3f0332953cbc3939b542502b4/python/PKGBUILD#L124-L125

mstorsjo commented 1 year ago

In the case of regular msys2 setups, you have libffi installed in the toolchain default include directory, so everything is found pretty much automatically. When cross compiling python for llvm-mingw, we build libffi into a separate prefix directory and configure cpython-mingw with CFLAGS="-I$PREFIX/include" - https://github.com/mstorsjo/llvm-mingw/blob/master/build-python.sh#L148.

It seemed like python was moving towards using pkg-config for most of this, I think this is why the tcl/tk patch was no longer necessary. I guess it's not finding libffi this way though?

Hmm, maybe that would work? The python cross compilation that a user contributed to llvm-mingw didn't use that for setting it up so far.

In https://github.com/msys2-contrib/cpython-mingw/blob/mingw-v3.11.4/setup.py#L1476-L1496 it first looks for LIBFFI_INCLUDEDIR (dunno if this gets set if it is found with pkg-config?), and if not set, it looks for ffi.h manually among the known include directories.

Additionally, the failure mode was quite strange; the build did complete, but the log printed messages that the _ctypes module wasn't built. At runtime I then got errors when using any python function that internally uses _ctypes.

This is "by design" of the python build process. It is pretty annoying when it "successfully" skips building standard modules that you thought it was going to include. I added a hack to MSYS2-packages python recipe to try to detect this when it bit me there: https://github.com/msys2/MSYS2-packages/blob/ddf03909a6a5bbf3f0332953cbc3939b542502b4/python/PKGBUILD#L124-L125

Ah, "nice"!

Another bit which seems weird but apparently "by design", is some missing dependency for parallel builds. In one (highly parallel) build I saw that it started building those modules before it had completed linking libpython3.11.dll, so all the native modules it tried to build failed with a -lpython3.11 not found or similar. But then later on in the build process, it retried building them again, and then they succeeded.

jeremyd2019 commented 1 year ago

In https://github.com/msys2-contrib/cpython-mingw/blob/mingw-v3.11.4/setup.py#L1476-L1496 it first looks for LIBFFI_INCLUDEDIR (dunno if this gets set if it is found with pkg-config?), and if not set, it looks for ffi.h manually among the known include directories.

Looks like LIBFFI_INCLUDEDIR does indeed come from pkg-config in configure https://github.com/msys2-contrib/cpython-mingw/blob/2b2d1a5985b12c4c053e99d65809e1ff7dbd07ec/configure.ac#L3869-L3874

But I don't know if pkg-config would be usable in your (bootstrappping-type) scenario anyway. When I was testing cross-compiling this I had to add to PKG_CONFIG_PATH so it could find ncurses. You may not care about having ncurses support in the python you are building, as much as you care about _ctypes.

lazka commented 1 year ago

When cross compiling python for llvm-mingw, we build libffi into a separate prefix directory and configure cpython-mingw with CFLAGS="-I$PREFIX/include" - https://github.com/mstorsjo/llvm-mingw/blob/master/build-python.sh#L148.

thanks, I've created https://github.com/msys2-contrib/cpython-mingw/issues/146

mstorsjo commented 1 year ago

In https://github.com/msys2-contrib/cpython-mingw/blob/mingw-v3.11.4/setup.py#L1476-L1496 it first looks for LIBFFI_INCLUDEDIR (dunno if this gets set if it is found with pkg-config?), and if not set, it looks for ffi.h manually among the known include directories.

Looks like LIBFFI_INCLUDEDIR does indeed come from pkg-config in configure

https://github.com/msys2-contrib/cpython-mingw/blob/2b2d1a5985b12c4c053e99d65809e1ff7dbd07ec/configure.ac#L3869-L3874

But I don't know if pkg-config would be usable in your (bootstrappping-type) scenario anyway. When I was testing cross-compiling this I had to add to PKG_CONFIG_PATH so it could find ncurses. You may not care about having ncurses support in the python you are building, as much as you care about _ctypes.

Thanks! I tested setting PKG_CONFIG_LIBDIR now, and that did indeed seem to fix the issue, so I can do without this patch for now. That's probably the best way forward, instead of relying on parsing $CFLAGS.

(For cross compilation, pkg-config works fine, but one has to remember to use PKG_CONFIG_LIBDIR, which overrides all default local builtin directories, while setting things with PKG_CONFIG_PATH lets pkg-config still find things in the default system directories like /usr/lib/pkgconfig.)

mstorsjo commented 1 year ago

When cross compiling python for llvm-mingw, we build libffi into a separate prefix directory and configure cpython-mingw with CFLAGS="-I$PREFIX/include" - https://github.com/mstorsjo/llvm-mingw/blob/master/build-python.sh#L148.

thanks, I've created #146

Thanks, I appreciate the effort you put into maintaining a working cross compilation setup of python for mingw! That allows me to generally not need to spend time/focus/effort on this :-)

jeremyd2019 commented 1 year ago

(For cross compilation, pkg-config works fine, but one has to remember to use PKG_CONFIG_LIBDIR, which overrides all default local builtin directories, while setting things with PKG_CONFIG_PATH lets pkg-config still find things in the default system directories like /usr/lib/pkgconfig.)

Thanks, I didn't know about that variable and its difference from PKG_CONFIG_PATH - I just used the one I heard of, and it worked so I didn't look any further :grin:

mstorsjo commented 1 year ago

(For cross compilation, pkg-config works fine, but one has to remember to use PKG_CONFIG_LIBDIR, which overrides all default local builtin directories, while setting things with PKG_CONFIG_PATH lets pkg-config still find things in the default system directories like /usr/lib/pkgconfig.)

Thanks, I didn't know about that variable and its difference from PKG_CONFIG_PATH - I just used the one I heard of, and it worked so I didn't look any further 😁

Yep, it generally works fine though as long as you provide everything you need. But if there's a library installed in the host form, but not wanted for the cross compilation target, and the build script looks for it, you need to set the LIBDIR version.