cross and native build of python for mingw* hosts

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 15 years ago

BPO	3871
Nosy	@birkenfeld, @doko42, @amauryfa, @giampaolo, @tarekziade, @merwok, @mingwandroid
Dependencies	bpo-3754: cross-compilation support for python build
Files	py3k_mingwmsys_2_autogen.diff: Building Python with MinGW/MSys v2 (correct) python-py3k-20110520-MINGW.patch python-py3k-20110825-MINGW.patch python-py3k-20120318-MINGW.patch python-py3k-20120318-MINGW-330a2.patch: Patch updated to build against 3.3.0a2 python-py3k-20120318-MINGW-FIXES-USE-POSIX-GETPATH-330a2.patch: Fixes for MinGW cross and beginnings of Darwin cross. python-py3k-20120607-MINGW.patch: version for 3.3.0a4 python-py3k-20120729-MINGW.patch py3k-20121004-MINGW.patch: extract from repository dated 2012-10-04 py3k-20121227-MINGW.patch py3k-20130110-MINGW.patch 0003-MINGW-implement-exec-prefix-i.e.-use-content-of-pybu.patch: MINGW-implement exec prefix, i.e. use content of pybuilddir.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/tarekziade' closed_at = created_at = labels = ['type-feature', 'build'] title = 'cross and native build of python for mingw* hosts' updated_at = user = 'https://bugs.python.org/rpetrov' ``` bugs.python.org fields: ```python activity = actor = 'rpetrov' assignee = 'tarek' closed = True closed_date = closer = 'doko' components = ['Cross-Build'] creation = creator = 'rpetrov' dependencies = ['3754'] files = ['18437', '22037', '23035', '24924', '25761', '25765', '25851', '26572', '27474', '28471', '28701', '28857'] hgrepos = [] issue_num = 3871 keywords = ['patch', 'needs review'] message_count = 108.0 messages = ['73239', '73241', '73699', '74090', '76699', '77243', '77246', '77259', '79959', '80019', '80028', '80037', '80347', '80349', '80970', '80997', '81170', '82114', '90819', '98224', '100073', '101008', '101011', '101059', '105860', '107899', '108229', '108745', '110686', '112608', '113214', '113218', '113236', '113263', '113265', '113274', '114777', '114780', '115856', '115863', '121078', '121086', '121087', '121091', '121094', '121095', '121097', '121099', '121133', '121134', '121144', '121242', '121244', '121245', '121246', '121247', '121251', '127144', '128271', '133253', '136339', '136382', '142931', '142973', '142989', '143150', '161973', '161976', '161977', '162422', '162423', '162424', '162465', '164399', '164413', '164415', '164419', '164421', '164448', '164449', '164451', '164453', '164473', '164477', '166741', '172301', '177467', '177533', '178433', '178471', '178631', '179732', '179734', '180595', '180662', '180694', '180695', '180703', '180704', '180705', '180707', '180708', '180742', '181182', '181185', '181518', '181519', '199695'] nosy_count = 20.0 nosy_names = ['georg.brandl', 'doko', 'lkcl', 'amaury.forgeotdarc', 'davidfraser', 'giampaolo.rodola', 'schmir', 'scott.tsai', 'tarek', 'eric.araujo', 'rpetrov', 'rschoon.old', 'WhiteTiger', 'BreamoreBoy', 'LRN', 'alesko', 'tshepang', 'kalev', 'Ray.Donnelly', 'jhuntley'] pr_nums = [] priority = 'normal' resolution = 'rejected' stage = 'resolved' status = 'closed' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue3871' versions = ['Python 3.4'] ```

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 15 years ago

This is a completely new patch against trunk that try to resolve mingw32 build. The first version to show problems with current python builds system, to propose solutions/work-arounds and to be stating point for discussion. Also I prefer some issues to be resolved later when we agree on the basic issues.

The patch is tested in cross-compilation environment and native build has to be tested too. Also the patch show how to use cross-compilation environment and require patch posted in bpo-3754 ("minimal cross-compilation support for configure") to be applied first.

The new patch share many commons with bpo-841454 "Cross building python for mingw32" posted by Andreas Ames, on 2003-11-13), but build process use distutils instead scons. Also some mingw related changes from bpo-1597850 ("Cross compiling patches for MINGW" - Han-Wen Nienhuys, from 2006-11-16) are shared too, but we differ in concept how to use cross-compilation environment. The correct way is to use $host variable (see bpo-3754).

Also my patch use python installed on build system along with some hacks to overcome distutils limitations.

The main issue "how to select correct compiler" in cross-compilation environment isn't resolved at all. The patch is tested with installed python version 2.4. Usually the cross-compiler is installed in same directories as native compiler but tools are prefixed by \<CPU>-\<VENDOR>- (see meanign of "host triplet"). In this environment we can't cross-compile even if we select mingw as compiler in arguments for "setup.py": The method get_versions() from cygwinccompiler.py call find_executable for to find gcc, ld, dllwrap but on build system they point to native compiler and that aren't prefixed. Also the expression "result = re.search('(\d+\.\d+(\.\d+))',out_string)" return as example '2.17.50.0.6' and later StrictVersion fail since value is not in format N.N{.N{A{N}}}. If we replace '\' with '?" in regular expression we will get correct result, but this isn't right solution. So the question is 'how to pass compiler to distutils' (as example for python 2.4) remain open.

Work-around: install mingw cross-suite in own directory (MINGW_ROOT) and prepend PATH environment variable with both paths:

$MINGW_ROOT/bin
$MINGW_ROOT/\<MINGW_HOST>/bin where \<MINGW_HOST> is something like i586-pc-mingw32 (depend from you host selected when you build suite). In this case find_executable will find mingw ld and since -v option return something like "GNU ld version 2.17.50 20060824", regular expresion value that make StrictVersion happy.

Note that in both environment configure script find compiler tools prefixed as example by "i586-pc-mingw32-". My mingw is build for host "i386-mingw32msvc"(this isn't correct, but is acceptable for host. For next version I plan to use for build the correct one, i.e i386-pc-mingw32). So in the final Makefile I see: CC= i386-mingw32msvc-gcc AR= i386-mingw32msvc-ar RANLIB= i386-mingw32msvc-ranlib

Also setup.py set compiler attribute linker_so - for details see comments in the patch.

Another point is to link installed python in top-build directory so that on posix system distutils read variables from our makefile instead from installed in build system. In this case python_build (an internal to distutils flag) is set.

A, but not common, build issue is that some python 2.4 versions will search for pyconfig-32.h instead pyconfig.h. Symptoms: \======================================= .... running build running build_ext error: ./pyconfig-32.h: No such file or directory .... \======================================= I see this on cent-os (5.0?), but I can't see on slackware 11.0. This patch don't address this. Just go in top-build directory and create a link, i.e. "ln -s ./pyconfig.h pyconfig-32.h".

If you system support emulation after build you may run python. The build python search for modules. After patch modules are with suffix same as for native build - .pyd. You may thin them to the top-build directory or to follow binary distribution - create directory DLLs, enter into it and link pyd-files: $ for f in ../build/lib.*/*.pyd; do ln -sf $f; done

Differences between mingw build python and binary distribution(native build): Mingw build follow posix rules and will create more modules than native build. In the native build they are part from main executable(buildin). This is the list: array, audioop, binascii, _bisect, _bytesio, cmath, _codecs_cn, _codecs_hk, _codecs_iso2022, _codecs_jp, _codecs_kr, _codecs_tw, _collections, cPickle, cStringIO, _csv, datetime, _fileio, _functools, future_builtins, _heapq, _hotshot, imageop, itertools, _json, _locale, _lsprof, math, mmap, msvcrt, _multibytecodec, operator, parser, _random, strop, _struct, _subprocess, time, _weakref, _winreg, zlib. I think that this inconsistency is problem of native build and for now I don't address it.

Because in my environment I still don't have installed mingw port for some externals, build of following modules isn't tested: _bsddb.pyd _msi.pyd _sqlite3.pyd _tkinter.pyd

Known issues: The float, math and other related tests fail(under emulation and on nt5.1). The reason is that C runtime function strtod() can't parse as example numbers with exponent lower than -308, can't parse inf, nan, etc.. The python source contain a file Python/strtod.c and my attempt to modify and use it didn't succeed. Also the configure lack test for "working" strtod and I guess that tests fail on posix systems without C99 support. The library mingwex from mingw runtime version 3.15 has working replacement for stdtod function. For now my patch will not address old strtod function.

In mingw definition of some functions(as example getaddrinfo and getnameinfo) depend of value of WINVER. This issue is commented in the patch. If user define WINVER geater the 0x0500 to CPPFLAGS at configure time the build will use C-runtime function otherwise fake-function from getaddrinfo.c and getnameinfo.c. As is commented in the patch if program is linked with C-runtime functions I expect to fail to run on w2k. I prefer to left selection of C-runtime for future mingw patches.

The other issue is that I see failure for some tests that use sockets. This problem isn't investigated yet.

We may group some changes in configure but this require to reorder some commands as example '--with-pydebug'. Since this isn't mingw issue I prefer don't change current order. Another example is LIBM - it is set by configure but setup.py use own rule.

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 15 years ago

P.S.: this patch cover changes in the python C-code proposed in issue 1412448 as include only necessary modifications.

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 15 years ago

For the protocol: bpo-2445 impact proposed patch. Also I finish the tests and I will upload soon new patch - I the current patch ( rpetrov, 2008-09-15 02:08) "Modules/selectmodule.c" isn't ported and this prevent socked and related modules to work. Also I understand the one of the reasons python 2.6x to be linked with msvcr90 - many of bugs related to math functions are fixed in this version of runtime.

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 15 years ago

note: updated patch contain unsynchronized with trunk code in ./Objects/fileobject.c (after /* EINVAL is returned when an invalid filename or ... )

182980c4-e4f4-4e83-8e3e-662d412bbf71 commented 15 years ago

I'm interested in this patch so I'm adding myself to the Nosy list.

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 15 years ago

The new patch is attached. Now the patch allow native build for mingw* host OS. The build id tested in MSYS environment. The issue title is updated to reflect this. The main changes:

native build require four more build-in modules (_functools, operator ,_locale, _winreg) otherwise setup.py fail to load;
changes in distutils for python 2.6+. The previous patch work on python 2.4 run in cross environment;
don't build with libintl;
use long double functions ldexp() atan2() from mingwex library as replacement for buggy msvcrt implementation.

The new patch don't include changes related to bpo-4288 (see also bpo-4279).

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 15 years ago

About the regression test results test. The result from 2008-11-29 is: 306 tests OK. 1 test failed: test_sundry 54 tests skipped: .... Those skips are all expected on win32.

To get those results you mingwex library has to contain fixes for two issues(see mingw32 bug tracker).

2136252 (fixed in trunk): c99, printf and float zero precision;
2117590 (pending): correction of function asinh - inverse hyperbolic sine. For the python regression tests copysign is enough.

About "test_sundry" depend on _msi module. I don't plan to work on mingw build for _msi module. Note that mingw build is with OS system default msvcrt so there is no "Assembly Hell".

The trunk from 2008-12-07 fail in addition on test_anydbm, test_bsddb and test_whichdb. No idea what is wrong. It seems to that break is related to updates. As example dbm module still fail to build on my linux.

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 15 years ago

The changes related to bpo-4483 break build on dbm based on "Berkeley DB" so I update mingw patch to restore bulld to previous state.

Now I expect test_anydbm, test_bsddb and test_whichdb to succeed as before one week.

f3d5ffa9-22a0-49e2-b73f-9da73147fe13 commented 15 years ago

roumen, hi, i'm interested in collaborating with you to get python compiled under wine (running msys+mingw32 under wine, on linux). bpo-4954 incorporates much of your work, and takes a slightly different direction for the configure setup - the time taken to complete configure under wine is intolerable (3 hours). also i have found that by setting PATH=%PATH%;c:/mingw/bin in the environment vars, the python.exe that's produced can successfully be used to run setup.py build.

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 15 years ago

Hi Luke, Initially I start build in cross environment. Later I setup a posix environment, based on msys and in the last patch is commented which module has to be build-in for native build.

I'm not surprised that you succeed to build in emulated environment and I know that born shell work slow in this environment.

f3d5ffa9-22a0-49e2-b73f-9da73147fe13 commented 15 years ago

roumen, hi, can you add:

   BASECFLAGS="-mthreads $BASECFLAGS"
   LDFLAGS="-mthreads $LDFLAGS"

when compiling with threads, and... a second request: _block_ people from compiling without mthreads, because there's a bug in wine's msvcrt _filbuf routine where it doesn't perform CRLF (at all) and it should.

http://bugs.winehq.org/show_bug.cgi?id=16970

f3d5ffa9-22a0-49e2-b73f-9da73147fe13 commented 15 years ago

roumen, hi, can you add:

   BASECFLAGS="-mthreads $BASECFLAGS"
   LDFLAGS="-mthreads $LDFLAGS"

when compiling with threads, and... a second request: _block_ people from compiling without mthreads, because there's a bug in wine's msvcrt _filbuf routine where it doesn't perform CRLF (at all) and it should.

http://bugs.winehq.org/show_bug.cgi?id=16970

terryjreedy commented 15 years ago

Does this supercede bpo-3754? If so, please add this as superseder and close it.

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 15 years ago

Does this supercede bpo-3754? - no. The issue bpo-3754 introduce macro AC_CANONICAL_HOST(add standardize "host triplets") + fixes for cross-compilation. Changes to configure script in this patch show how to use "host triplet" for mingw builds.

508f7627-3797-4332-8a63-a38898672f33 commented 15 years ago

maybe you can group all your patch into one single issue since they are related, and remove old patches,

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 15 years ago

The proposed patch for this issue include parts of other pending issues

so its all is single file. If python team don't like idea for "canonical host names" (part of bpo-3754) this patch can be modified do not use host-triplet. Also some discussions in py-dev list show that small patches are preferred. I'm ready to split into small patches to be reviewed.

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 15 years ago

attached patch for trunk-20090204:

removed patch from bpo-4587 (not relevant for mingw) ;
improved static build now succeed (use --disable-shared). this is follow-up of bpo-4494 + ... (don't expect loadable modules to work on windows) ;
restore win32.S removed in r69260. thanks thomas.

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 15 years ago

updated patch do not include Lib/distutils/... :

.../command/build_scripts.py, fixed in another non-related bpo-4524 ( distutils build_script command failed with --with-suffix=3)
.../tests/test_build_ext.py, fixed by issues related to builddir \<> srcdir. Now distutils support srcdir on all platforms and this include win32.

and to apply cleanly to trunk 2009-02-14 (after minimal cross patch - see above).

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 14 years ago

The last patch is updated to support build with GNU C Compiler v 4.4.0 for windows.

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 14 years ago

system python has to be at least revision 77704 (trunk)

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 14 years ago

updated patch to trunk:

apply cleanly
support builddir \<> sourcedir
regression tests pass if run from any drive (windows hosts)

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 14 years ago

distutils from trunk is restored to state before 9 months ago

508f7627-3797-4332-8a63-a38898672f33 commented 14 years ago

Roumen, I am moving all the new work in distutils into distutils2, and distutils is going to be frozen.

Do you want to work on this feature in distutils2 ? (which is distutils code base before I started to revert things)

Let me know so I mark this issue under "Distutils2"

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 14 years ago

Tarek Ziadé wrote:

Roumen, I am moving all the new work in distutils into distutils2, and distutils is going to be frozen.

Last patch is uploaded as the previous patches didn't apply cleanly (as compiler_obj is restored to compiler and etc.)

Do you want to work on this feature in distutils2 ? (which is distutils code base before I started to revert things)

Yes. Also I don't know where is distutils2 repository.

Let me know so I mark this issue under "Distutils2"

No idea, because I expect Distutils2 to work with python 3+ and this patch is not tested(ported) to 3+.

Roumen

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 14 years ago

Common to all platforms part of patches lets call it "build modules with system python" is moved to bpo-3754.

0953ef5c-9468-4d47-b308-207d95cfcb6c commented 13 years ago

Roumen thanks for this work, but will you upload a new version? Just because i can't get it to work, i can't even patch it^^

Maybe it's the SVN Client i use (Tortoise SVN) or that i don't know how to apply this patch :D It would be nice when you help me

Kind regards

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 13 years ago

Patches to this issue require first patch from bpo-3754. The last patch here now is from 2010-05-16. This patch cannot be applied to trunk as it will fail after update for readline in setup.py. You could use trunk version before this readline update.

Now I'm working on readline and ncurses support for mingw . All patches before skip this as MSVC build don't support them. As I could not found reason to limit mingw users I will post soon version without limitation.

P.S. Many thanks to евроком (www.ekk.com now blizzo) - a company that left me for a week without internet connection.

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 13 years ago

require patch "...20100626-CROSS..." from bpo-3871;
setup.py don't ignore readline and ncurses for host platform : python tests with readline pass, ncurses (current development) is not usable yet;
follow restored build with python specific windows version of libffi.

83d2e70e-e599-4a04-b820-3814bbdb9bef commented 13 years ago

Could someone with knowledge of the Python build systems please comment on this, thanks.

9255d2b8-ed7c-4cd1-bf9c-ba91bbc36b00 commented 13 years ago

Here's an alternative patch that enables Python building with MinGW/MSys I've skimmed rpetrov's patch, and it seems that my patch is more about building Python than about distutils (i've patched distutils only as far as the compilation of Python itself (with the extensions already included in the trunk) requires.

Brief patch description - what i did, and why: Python/import.c:

briefly fixed an issue with mkdir() already pointed out in code comments

Python/dynload_win.c:

renamed strcasecmp to pystrcasecmp to avoid conflicts with the same function defined in MinGW

Python/thread.c: Python/ceval_gil.h:

Made sure that Python won't try to include pthreads header when NT threads are available

Include/osdefs.h:

Made sure that the PYCC_GCC won't lead Python into defining *nix path separation scheme
Fixed redefinition of MAXPATHLEN (one of the few redefinitions i did fix, there are just too many of them)

configure.in:

Added corret MACHDEP="win32" for MinGW
Added lots of new AC_SUBST'ed variables, they are used later to alter Makefile and Setup functionality MACHDEP_PATH_SEPARATOR - to put ';' instead of ':' as path separator in machdep NTOBJS - extra object files to insert into makefile FROZEN_MAIN - controls the inclusion of Python/frozenmain.o into libpython. Why is it even there? Prevents the library from linking. NTLDFLAGS - extra LDFLAGS to add on Windows SHAREDINSTALL - controls which sharedinstall target is built (works around MSys weirdness, see below) USE_MODULE - controls the inclusion of modules in Setup.config.in ms_windows - indicates that we're building on Windows through the rest of configure.in Special MINGW32_NT/1.* case entry sets all of the above as well as some some #defines that traditionally are just hard-coded in PC/pyconfig.h (not all the defines from PC/pyconfig.h are here, there might be a need to add some more)
Added MINGW32 to trigger CYGWIN cases or added separate MINGW32* cases in most uname tests.
Added MS_COREDLL definition (why is it separate from other Windows-specific AC_DEFINEs? I don't remember...)
Added a separate check for CreateThread, a special variable (ac_cv_nt_threads) to indicate its success, and some code to switch on NT threads and switch off any other threads (including pthreads which ARE available on Windows)
Added checks for some headers (pwd.h, sys/ioctl.h etc) and functions (spawn, spawnv, system etc) that are (are they?) traditionally available on *nix and are not checked for.
Added correct DYNLOADFILE definition for MinGW
Worked around AC_STRUCT_ST_BLOCKS weirdness by removing fileblocks.o from the list of object files (this one might not be portable, should check that it works on *nix)
Fixed a strange bug in AC_COMPILE_IFELSE around "have_prototypes=yes" - when left with an empty argument [] it probuces broken (?) code in configure script (or maybe it's bash that is broken, i don't know)
Fixed /dev/ptmx detection - MSys will tell you it's available, but it is only an MSys thing, Python shouldn't think it is there.
Fixed PY_FORMAT_SIZE_T definition - AFAIK MinGW does not allow %z format in printf (in my test cases i've found that some definitions can prevent GCC from throwing an error about this format, but i couldn't reproduce this GCC behaviour in Python build system).
Added ws2tcpip.h into socket test
Added PC as a subdirectory

setup.py:

Added module's location into sys.path. This is required to correctly import multiprocessing (it imports itself recursively, and since its .pyd file is in unusual location, it couldn't find itself)
Added some config_h_vars.get() checks for some extensions (which is why some extra function/header checks were necessary in configure.in) - no setup.py won't try to build extensions that can't be built
Added "'msys' in sys.builtin_module_names" kind of checks in some places. When successful, this check indicates that Python is built in MSys, and alters otherwise win32-oriented logic in setup.py and some other modules to adapt to MSys environment
Changed the way MODULE_NAME is passed to sqlite3 extension (see below)
Added extra library dependencies for multiprocessing extension
Prevented setup.py from adding libX11 dependency to tkinter on Windows
Added some msysize() calls (see below) to fix the paths passed to os.system() calls
Added extra library dependencies for ctypes extension

PC/errmap.h:

Re-generated it with new generrmap.c (see below)

PC/generrmap.c:

Added an implementation of _dosmaperr() function, which is available in MSVC MSVCRT static library, but not in its MinGW counterpart (code is from PostgreSQL, should be compatible with Python license)

Lib/distutils/spawn.py:

Fixed spawn_nt to use os.system() in conjunction with 'env.exe' program to call external tools - works like a charm in MSys ('env.exe' is an MSys tool, lives in /bin, and it hooks up MSys shell just as well as a call to sh.exe would). I don't know why i'm not calling sh.exe directly here. Just saw os.system()/env pair somewhere else, and it stuck.

Lib/distutils/util.py:

Added a primitive msysize() function - an MSys path converter. Keeps MSys shell happy by avoiding paths with '\' path separators and '[A-Za-z]:' drive letters

Lib/distutils/ccompiler.py:

Rigged ccompiler to use unixcompiler subclass for MSys. I didn't know at the time that there is the cygwincompiler, which should have been closer to MSys than unixcompiler. However from the looks of things it took only few lines to fix unixcompiler to work with MSys, so i kept it this way.

Lib/distutils/command/install.py:

Added special installation schemes for MSys. Not sure if they are used during the installation of Python itself, but they look cool. It always bothered me that Python's layout on Windows is significantly different from the one on *nix, that breaks most autotools magic aimed at detecting/using Python. Now i have a chance to set things right.

Lib/distutils/command/build_ext.py:

OK, this one is a real hack. There is virtually no way to find include and library paths for GCC: they USUALLY point to /mingw/include/* and /mingw/lib , but even that is configurable in MSys, and Python doesn't have direct access to that data - asking GCC should always work, so we call GCC via os.system() and ask it to provide include and lib directories (GCC output is fed into a file, since os.system() doesn't have any piping)
Added libpython naming scheme for Msys (similar to the one used for 'unix')

Lib/distutils/sysconfig.py:

Added _init_msys() initialization function which combines _init_nt(), _init_posix() and some custom fixes (which come from PC/pyconfig.h for MSVC)

Lib/sysconfig.py:

Added some more "'msys' in sys.builtin_module_names" checks to make sure that some code branches get executed or not.

Makefile.pre.in:

Added all the variables from configure.in to insert correct object modules and compiler/linker flags
Added explicit --export-all-symbols for building libpython
Added some extravagant rules to build PC/generrmap.h and build exceptions.o which depends on it
Added special msys sharedinstall variant with fixed paths (otherwise, with normal sharedinstall, some stuff ends up in $(DESTDIR)/MSys/1.0.11/usr/local or something like that due to MSys substituting /usr/local or any other prefix for its Win32 counterpart, which in this particular case is completely wrong, since prefix is always a relative path which shouldn't be fixed)
Added some Windows-specific clean target commands

Modules/makesetup:

Added some MinGW specific definitions (cloned from Cygwin's ones, didn't have to alter them after that)

Modules/_ctypes/callproc.c:

Replaced some MS_WIN32 with _MSC_VER, since the code they guard works only with MSVC

Modules/_ctypes/_ctypes_test.c:

Prevented preprocessor from inserting dllexport attributes (their presence disables automatic symbol exporting for module initialization function)

Modules/getpath.c:

Fixed reduce() to work with paths with mixed separators and with paths with Windows-style drive letters

Modules/socketmodule.h:

Avoided some clever MSSDK detection trap

Modules/_multiprocessing/semaphore.c:

Fixed some #ifdef guards (the test is ambiguous: it can refer to broken sem_getvalue() - on *nix, or to the lack of one - on Windows)

Modules/Setup.dist:

Used MACHDEP_PATH_SEPARATOR instead of hard-coding ':' separator into MACHDEPPATH
Removed posixmodule and pwdmodule from the list of unconditionals modules, now they are conditional (see below)

Modules/_sqlite/connection.c: Modules/_sqlite/cache.c: Modules/_sqlite/statement.c: Modules/_sqlite/row.c: Modules/_sqlite/sqlitecompat.h: Modules/_sqlite/cursor.c: Modules/_sqlite/prepare_protocol.c: Modules/_sqlite/module.c:

I had some annoying problems passing MODULE_NAME=\"sqlite3\" - with escaping backslashes. It doesn't work well with my msysize(). Eventually i gave up and removed the need for doublequotes in the definitions by using stringifying feature of C preprocessor. Looks clumsy, but i do not insist on naming the macro Py_SQLITE_TO_STRING(), it can be something shorter...

Modules/python.c:

Fixed main function prototype - MinGW doesn't work with wmain()

Modules/Setup.config.in:

Made some modules configurable - some of them are not available on Windows, while others are only required to be built early on Windows. Adds MACHDEP_PATH_SEPARATOR definition as well.

Modules/posixmodule.c:

Widened some include guards to include MinGW, added some includes for MinGW, disabled some structure re-definitions (it should be noted that i've used a patched version of MinGW's w32api package that already includes symlink functionality - the patch should eventually make it into official w32api release, so this particular fix in posixmodule might not be necessary for a while)

Modules/main.c:

Fixed PATH_MAX redefinition

9255d2b8-ed7c-4cd1-bf9c-ba91bbc36b00 commented 13 years ago

New version of my patch. Addresses some issues i've found since the last one. Diff against a recent trunk is attached.

Changes (relative to previous patch):

Python/pythonrun.c:

Added a call to _set_abort_behavior(). That should fix http://bugs.python.org/issue9116

Python/dynamic_annotations.c:

Changed _MSC_VER to MS_WINDOWS, since the code does not seem to be MSVC-specific

configure.in:

Added AC_DEFINE for Py_SOCKET_FD_CAN_BE_GE_FD_SETSIZE (for MSVC-based builds it is hardcoded in PC/pyconfig.h)
Added -no-undefined LDFLAG for Windows builds - helps find missing library dependencies at buildtime
Added -fno-common and -mms-bitfields CFLAGS for Windows builds. -fno-common could fix some hidden problem with aligned commons, and -mms-bitfields should improve compatibility with MSVC-built binaries
Added -D_GNU_SOURCE CFLAG. Defining this enables MinGW non-ANSI extensions, including the custom printf() implementation that supports %z format
Added AC_DEFINE for HAVE_GETPEERNAME, because testing lws2_32 for it is tricky
Removed PY_FORMAT_SIZE_T fiddling, since with GNU extensions %z is supported

setup.py:

_subprocess module is now built on win32 to support subprocess extension
msvcrt, winreg and winsound modules are built as well

PC/winreg.c:

Added REG_LEGAL_CHANGE_FILTER definition (MinGW does not define it)

PC/dl_nt.c:

Fixed PFN_ADDREFACTCTX and PFN_RELEASEACTCTX - according to MSDN, they return nothing
Fixed the way AddRefActCtx is called (GetLastError() is being checked for errors instead of return value)

PC/generrmap.c:

Added return 0 to prevent generrmap.exe from "failing" (returning something other than 0)

PC/msvcrtmodule.c:

Added #ifdef guards around crtdbg.h, because only MSVC provides it

Lib/site.py:

Extended os.sep == '/' (why isn't os.name tested?) to msys
Extended addbuilddir calling condition to msys

Lib/distutils/sysconfig.py:

Extended some posix conditions to msys, restricted some nt conditions to non-msys

Lib/distutils/tests/test_bdist_msi.py:

Disabled this test for MSys (MinGW can't build msi module)

Lib/distutils/ccompiler.py:

An extra hack to add $(srcdir) to library_dirs at build time

Lib/distutils/sysconfig.py:

Some MSys-specific configuration magic

Lib/ntpath.py:

Added a reminder that realpath() can now be implemented

Lib/sysconfig.py:

Added msys-only variation of directory layout (copied from posix)

Lib/test/test_site.py:

Widened os.sep == '/' condition to MSys

Makefile.pre.in:

Fixed CFLAGS order for Objects/exceptions.o target (won't include pyconfig.h from PC anymore)

Modules/getpath.c:

Massive fixes. Most thigs are copied from PC/getpathp.c (who had the bright idea of having two separate, but almost identical headers?)
Ripped a readlink() implementation from posixmodule.c to use here. By the way, it was buggy, so i had to fix it as well.

Modules/socketmodule.h:

Added some definitions for MinGW - these aren't in current MinGW w32api package, and will (probably) never get there.

Modules/_multiprocessing/win32_functions.c: Modules/_multiprocessing/multiprocessing.h:

Added more gurads on crtdbg stuff, because it is MSVC-only

Modules/selectmodule.c:

Changed some guards from _MSC_VER to MS_WINDOWS, because the code is not MSVC-specific

Modules/posixmodule.c:

Changed some guards to work for MinGW as well
Discovered why i previously had to fix mkdir() call. Now that's not necessary anymore (but i left previous fix as it is, can't hurt to be careful)

I ran some tests and tried to make my Python build pass them, which partially fueled my bugfixing efforts.

_ctypes test (namely - test_callconv_1) crashes the interpreter (SIGSEGV in libffi), so i had to comment it out.

test_capi fails (but that should be now fixed in MSVC builds, can't test it in a MinGW build) test_cmath fails (i have no idea why) test_cmd_line fails (i think it is not adapted to MSys; and i'm not sure it needs to be - after all, this Python build is NOT intended to be ran in MSys most of the time) test_distutils fails (i've fixed that only partially) test_email fails (wrong crlfs) test_format fails (?) test_heapq fails (but when i re-run it, it passes; because of pre-compilation?) test_platform fails (doesn't work well with symlinked python, because realpath isn't implemented on NT yet) test_smtplib fails (no idea - why) test_sundry fails (?) test_sysconfig fails (didn't try to fix) test_tarfile fails (didn't try to fix) test_tcl fails (didn't try to fix) test_trace (didn't try to fix) test_urllib2 (didn't try to fix)

test_multiprocessing is skipped because there's no test for multiprocessing in Lib/test/

282 tests OK.

merwok commented 13 years ago

FYI, distutils is frozen because even minor bug fixes have broken third-party tools in the past, that’s why new features and bug fixes land in distutils2. Only some bug fixes that are sure not to break things make their way into distutils. distutils2 will be reintegrated into the stdlib in 3.2 or 3.3, but standalone releases will be made for Python 2.4 up to 2.7 and 3.x.

9255d2b8-ed7c-4cd1-bf9c-ba91bbc36b00 commented 13 years ago

Where's the distutils2 code?

merwok commented 13 years ago

See http://bitbucket.org/tarek/distutils2

0953ef5c-9468-4d47-b308-207d95cfcb6c commented 13 years ago

@LRN what program did u used to create the patch? And what to apply it?

Just because i can't get Tortoise to apply the patch (An unknown line type was found in line 678 !) and MSYS's patch also does not work.

9255d2b8-ed7c-4cd1-bf9c-ba91bbc36b00 commented 13 years ago

Uh, sorry about that. I've replaced the second diff with the correct one.

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 13 years ago

update patch still for 2.7 with

restored build of ctypes (broken after recent updates)
python build with ncurses library, test pass
python build with readline library, test pass

f3d5ffa9-22a0-49e2-b73f-9da73147fe13 commented 13 years ago

On Sun, Aug 8, 2010 at 12:27 AM, Éric Araujo \report@bugs.python.org\ wrote:

Éric Araujo \merwok@netwok.org\ added the comment:

FYI, distutils is frozen because even minor bug fixes have broken third-party tools in the past, that’s why new features and bug fixes land in distutils2. Only some bug fixes that are sure not to break things make their way into distutils. distutils2 will be reintegrated into the stdlib in 3.2 or 3.3, but standalone releases will be made for Python 2.4 up to 2.7 and 3.x.

eric,

it's been a year since i did the mingw32 port so i'll try to recall what it was that i did, but i have recollections of also having to add to distutils, specifically adding an extra compiler class and an extra linker class, derived from cygwin i believe (and then adding, obviously, platform detection logic and options to make use of them).

using logic analysis on the conditions, it can be shown that there is nothing that can satisfy the conditions (one of which is "distutils is frozen", another is "distutils will not go into python2.N or 3.x" etc). thus, something has to give.

detailed analysis and recommendations follow:

if distutils2 is to be "standalone" it literally makes it impossible to build python 2.4 up to 2.7 and 3.x using mingw32, because the build infrastructure will be entirely missing. how can you build python 2.7 using mingw32 if one of the critical dependencies - actually saying "use mingw32" - is missing?!!

the decision to keep distutils frozen therefore forces the mingw32 developers to create a "special" version of distutils, one which has to be added as a patch to python - let's call it distutils_mingw32 - which has the exact same functionality as distutils except that it is extended to have a mingw32 compiler and linker class. adding such a patch (which adds a copy of distutils called distutils_mingw32) would be extremely disruptive, as it would defeat the entire purpose of distutils, and i would be extremely surprised if it was accepted.

so, unpalatable-as-defined-by-previous-negative-experience as it may be, the only sensible option is to do an extremely careful and thorough review of LRN's distutils compiler and linker class additions, and ensure that they are _purely_ additional classes, even at the expense of duplicated code.

please remember that mingw32 is an _additional platform, not a change to any _existing platforms.

l.

p.s. the only other option i can think of would be to add in build-time options where you can specify the fully-qualified path to where distutils2 can be found. this would be somewhat... contrived, but would "work".

f3d5ffa9-22a0-49e2-b73f-9da73147fe13 commented 13 years ago

sorry to have to ask, but could we get some feedback please so that this issue may move forward? currently there is a conflict between what is required and what is stated as being "absolute law".

let's imagine that it is reasonable to expect distutils2 to be used. python proceeds with the mingw32 patches using distutils2, python 2.N is compiled and released. an average user installs python 2.N mingw32 and they run "python setup.py install" on a package - what happens?

they get a compile error, don't they?

why do they get a compile error?

because the setup.py tries to build some c code, and at the top of *their* setup.py is "import distutils".

so you now force thousands of developers to patch allll their setup.py scripts - scripts which have worked for years - _just_ so that those packages can cope with the use of distutils2 for the mingw32 platform?

so we have another "reducto ad absurdum" which demonstrates that it is impractical to expect distutils to be frozen.

thus, we are forced to consider alternative options, such as "monkey-patching" of distutils, to satisfy the requirements.

would the following be acceptable, to be inserted somewhere in the path so that it is guaranteed to work at all times?

import mingw32_distutils_compiler
import sys
if "distutils.compiler" in sys.modules:
    sys.modules['distutils.compiler'] = mingw32_distutils_compiler

where mingw32_distutils_compiler performs an import of distutils.compiler and performs a monkey-patch mish-mash to combine and replace various parts of the compiler module at run-time, to get round the "freeze" objections.

would that be acceptable? please say no, because the long-term viability and maintainability of such practices is virtually nil.

basically i'm pointing out to you, eric, that the freeze on distutils is unworkable and impractical and unnecessary.

this isn't "bug-fixing" of distutils, it's absolutely necessary and critically required because this is an entirely new platform.

it's not cygwin: cygwin uses standard gcc but with weird outputs and quirks.
it's not standard unix gcc: you need to output ".dll" not ".so"
it's not msvc: mingw32-gcc doesn't accept "/this /that" for options.

therefore i'm very sorry to have to spell it out but a new compiler and linker type - a NEW compiler and linker type - an ADDITIONAL compiler and linker type - IS required in order to cater for this platform, and there's no getting away from that fact.

please could we have some acknowledgement of this fact, and acceptance of this fact, and a plan for moving forward with careful review of this patch, so that LRN and others do not have their time spent pursuing a direction involving distutils2 which will be completely fruitless?

many many thanks,

l.

9255d2b8-ed7c-4cd1-bf9c-ba91bbc36b00 commented 13 years ago

A small correction to previous post: two new compiler-and-linker types, not one.

First - MinGW toolset without MSys, working with normal Windows command interpreter, which can be invoked simply by calling "cmd" (or whatever we get from %ComSpec%). Obviously, it doesn't work. Well, it works, but only \~5% of all software packages will have a makefile that doesn't have to be generated by configure script (which we can't run, since this variant of MinGW doesn't have bash) and that doesn't call any shell builtins or *nix tools. On the other hand, SCons, CMake and other build systems not based on autotools should work fine without bash and/or should be able to generate makefiles that don't rely on bash or *nix tools.

Second - MinGW toolset with MSys, working with bash, which can't be invoked by calling "sh.exe" (even if you have full absolute path to sh.exe), because you must cast a few black magic spells by calling sh.exe with "--login" to hook up /etc/profile, which sets up PATH and other environment variables; without it you won't be able to run anything that isn't in c:/msys/1.0/bin - including mingw32-gcc.exe, which lives in c:/mingw/bin, unless you'd want to duplicate /etc/profile in Python (which is possible, remind me to try that), and you have to be careful to not give it any \-separated paths (even in scaped form) at command line, because bash won't understand them, and path mangling doesn't kick in until you're *within* bash, and that includes the name of the script you want to run - it must be /-separated and not absolute (because bash will pass it as $0 without mangling).

d8d5aad8-e55b-4500-a3a0-9ea982d771ff commented 13 years ago

And the patch for py3k (future 3.2) . Note require python-py3k-20101112-CROSS.patch from bpo-3754 to be applied first.

amauryfa commented 13 years ago

I'm trying to read the patch. It contains many interesting things (and others I have no opinon about), but it is very large, and makes it difficult to comment or find why some change were made etc.

I do believe that better supporting the mingw32 platform is a worthy goal, but a single large patch, even after review, is something difficult to trust.

I suggest to present this work in another form. Maybe something like a Mercurial Patch Queue? This would give us a collection of smaller patches, each with its own interest. Each one can be discussed, modified, reviewed separately and by different people. Large easy tasks (like a massive rename) are distinct from small-but-hard changes. And when something is updated, the reviewers don't have to parse the whole file again; they just look at the recent patches.

As an example: among the many changes, you had to handle the Modules/getpath.c vs. PC/getpathp.c inconsistency. You chose to have both filenames in the Makefile, but in my opinion it would be better to try to merge the files.

merwok commented 13 years ago

Let’s try again. The current situation is that building with mingw32 is not supported. Adding support for mingw32 is by definition a new feature. You seem to agree on that: “a NEW compiler and linker type - an ADDITIONAL compiler and linker type”.

Second, distutils is frozen. That’s the policy decided by Tarek, who’s spent literal months working on PEPs to move the situation forward. It’s just not possible to fix things in distutils, so it’s frozen and new work happens in distutils2, at the suggestion of Python creator Guido van Rossum. Please accept the feature freeze.

In conclusion, such pervasive changes will not happen in distutils.

f3d5ffa9-22a0-49e2-b73f-9da73147fe13 commented 13 years ago

erik, i'm really sorry, but the freeze on distutils simply cannot be accepted: there really is no other way, as you can see from the previous walkthrough analysis, and is reinforced by the further analysis below.

simply put: if the freeze on distutils is not lifted, then this entire set of work, which has been going on for years and _precedes_ the distutils freeze by at least 18 months, is completely, utterly and totally wasted.

let's walk through the situation where distutils2 is forced to be used.

what you're asking for is, basically, that every single third party package, of which there must be tens of thousands, must be patched for compilation on mingw32... _just_ so that it says "if sys.platform == 'mingw32': from distutils2.core import setup else: from distutils.core import setup", is that correct?

does that strike you as being completely and utterly unreasonable an expectation, to ask third parties to modify setup.py scripts which have worked perfectly well for years, many of which are likely to no longer even have a maintainer?

that leaves patching - which should be nothing more than _adding_ to - not "changing existing compilers" - ADDING an extra compiler - distutils as the only option.

now, that can be done monkey-patch style (i.e. at runtime, by adding in code very early on in python startup, or perhaps by patching the import system to replace the word "distutils" with "distutilsmingw32") or it can be done... by lifting the distutils freeze.

perhaps i should ask: what _exactly is the problem, and why do several teams complete and utter failure to do the correct thing by making changes to _existing compile platforms have anything to do with _this_ team's patches which add a new totally separate platform that has absolutely nothing to do with any of the other platforms?

i could say more, but i believe the point is clear: none of the people involved in seeing mingw32 added as a platform had _anything_ to do with the past failures, so why "punish" and mis-trust the python-mingw32 for other peoples' failures?

508f7627-3797-4332-8a63-a38898672f33 commented 13 years ago

We don't punish anyone here. We try to avoid breaking distutils because it's really easy to get broken and get people upset. (you ask me)

This bug has been opened for years and no one commited it because it's very very hard to review. it involves many parts, many people.

I am not sure how we should do this, but here's my proposal for distutils2 at least:

make this new feature a standalone package that patches distutils
release it for 2.x
let's add your work in distutils2 as well, so it's back in the stdlib in 3.x

f3d5ffa9-22a0-49e2-b73f-9da73147fe13 commented 13 years ago

I'm trying to read the patch. It contains many interesting things (and others I have no opinon about), but it is very large, and makes it difficult to comment or find why some change were made etc.

amaury: unfortunately, the development on adding mingw32 as a platform has been ongoing for well over three years, with absolute silence and rejection of its existence. it was only when a fourth person showed interest in it (LRN) that it became "accepted" - but that was... only two months ago!

basically, this situation should never have been allowed to get this far: the very first patch that was created, three nearly four years ago, should have been accepted, and then an incremental process could have been taken, _and_ the silly situation in which distutils gets frozen during the time when people have been completely ignoring this ongoing work would never have occurred.

but, that's the situation: the bed has been made, and now developers have to lie in it. sorry to be the one that's pointing this out, but...

anyway.

your idea to split this into a series has merit: personally i much prefer git, because of git-format-patch, but i have to say i've never done "patch regeneration" based on a "review / change-patch-in-middle-of-series / regenerate git-format-patch" cycle. should be fun! :)

but the very very first thing that has to happen - before any of this work is begun - is for the distutils freeze to be lifted, or for someone to come up with a _sensible_ alternative solution.

if that cannot be done, then roumen and LRN won't _stop_ working on python-mingw32: the end result will be that they just... continue to create a single patch file that will just get larger and larger.

508f7627-3797-4332-8a63-a38898672f33 commented 13 years ago

I understand your frustration, but I can't lift the freeze because any change to distutils potentially break third party projects out there. We suffered from that in the past and we decided to proceed on a new version and freeze this one.

Having this compiler released on its own project is absolutely possible and acceptable and can be done in a way that works with distutils1 and distutils2, and I'd happily integrate it to distutils2 today once we manage to review it.

amauryfa commented 13 years ago

but the very very first thing that has to happen - before any of this work is begun - is for the distutils freeze to be lifted, or for someone to come up with a _sensible_ alternative solution.

Tarek's approach looks right to me: mingw32 support in distutils should take the form of an external module that subclasses (when possible) or monkeypatches (on last resort) the standard distutils behavior.

The current patch makes too many changes in core distutils functions; it cannot be accepted in this form. I'm sure that most of the needed changes can be made in a subclass of the present Mingw32CCompiler.

f3d5ffa9-22a0-49e2-b73f-9da73147fe13 commented 13 years ago

I am not sure how we should do this, but here's my proposal for distutils2 at least:

make this new feature a standalone package that patches distutils

release it for 2.x

let's add your work in distutils2 as well, so it's back in the stdlib in 3.x

that would work very well: people doing new setup.py files, converting from python2 etc., will, duh, need to do _some_ conversion: they'll expect breakage and to be using distutils2 anyway.

thank you tarek.

f3d5ffa9-22a0-49e2-b73f-9da73147fe13 commented 13 years ago

The current patch makes too many changes in core distutils functions; it cannot be accepted in this form. I'm sure that most of the needed changes can be made in a subclass of the present Mingw32CCompiler.

that's what i did when creating the _other (yet another) mingw32 patch - however _some minimal changes to core distutils _are_ unfortunately required. and to sys.py and os.py - this is, after all, a new platform! it starts off with "sys.platform == 'mingw32'", requiring detection of gcc compiler type BUT and os type of win32, and goes from there.

right now, detection logic is:

"if gcc on win32 platform, platform MUST be cygwin"
"if msvc compiler, platform MUST be win32"

both of which are... well... wrong! :)

so, i found that it was necessary to start in os.py and sys.py, create a platform-type "mingw32" and _then it was easy to do a MingW32Compiler etc. with (mostly) additions not modifications to distutils, but _necessary to add in detection of the type.

if distutils was designed to do "exec import distutils.compiler.%s as compiler" % sys.platform and go from there then this would be an entirely non-issue: no modifications to distutils would be required, just a completely separate module containing the new compiler...

... but distutils isn't designed that way, is it? :)

l.

python / cpython

cross and native build of python for mingw* hosts #48121