Mingw-w64 and python on windows x64

[a1f769b2-b2b7-4237-92e2-33d87a4133db]

BPO	4709
Nosy	@loewis, @pfmoore, @giampaolo, @merwok, @bitdancer, @njsmith, @zware, @taschini, @zooba
Superseder	bpo-45538: MinGW is unsupported - close all open issues and list them here.
Files	mingw-w64.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/loewis' closed_at = created_at = labels = ['build'] title = 'Mingw-w64 and python on windows x64' updated_at = user = 'https://bugs.python.org/cdavid' ``` bugs.python.org fields: ```python activity = actor = 'iritkatriel' assignee = 'loewis' closed = True closed_date = closer = 'iritkatriel' components = ['Build'] creation = creator = 'cdavid' dependencies = [] files = ['12411'] hgrepos = [] issue_num = 4709 keywords = ['patch'] message_count = 55.0 messages = ['78140', '84177', '84209', '84252', '87294', '87307', '87310', '87314', '98616', '132798', '135708', '136144', '152694', '152702', '152729', '152743', '152783', '195541', '197108', '217487', '243511', '243513', '243531', '243532', '243533', '243534', '243535', '243539', '243563', '243566', '243569', '243589', '243605', '243607', '243624', '243625', '243630', '243647', '255649', '255650', '256112', '256788', '256799', '256801', '256810', '256828', '256834', '256836', '256852', '256860', '256861', '256897', '256900', '256901', '257336'] nosy_count = 27.0 nosy_names = ['loewis', 'paul.moore', 'jdpipe', 'casevh', 'giampaolo.rodola', 'donmez', 'scott.tsai', 'cdavid', 'eric.araujo', 'rpetrov', 'r.david.murray', 'njs', 'cgohlke', 'rubenvb', 'WhiteTiger', 'LRN', 'cournape', 'zach.ware', 'taschini', 'steve.dower', 'arbitraryvalue', 'ralf.gommers', 'simonzack', 'matham', 'carlkl', 'Henry Gomersall', 'Duncan McBryde'] pr_nums = [] priority = 'normal' resolution = 'wont fix' stage = 'resolved' status = 'closed' superseder = '45538' type = 'compile error' url = 'https://bugs.python.org/issue4709' versions = [] ```

[a1f769b2-b2b7-4237-92e2-33d87a4133db]

I believe the current pyport.h for windows x64 has some problems. It does not work for compilers which are not MS ones, because building against the official binary (python 2.6) relies on features which are not enabled unless MS_WIN64 is defined, and the later is not defined if an extension is not built with MS compiler.

As a simple example:

#include <Python.h>

int main()
{
    printf("sizeof(Py_intptr_t) = %d\n", sizeof(Py_intptr_t));
    return 0;
}

If you build this with MS compiler, you get the expected sizeof(Py_intptr_t) = 8, but with gcc, you get 4. Now, if I build the test like:

gcc -I C:\Python26\include -DMS_WIN64 main.c

Then I got 8 as well.

I believe the attached patch should fix the problem (I have not tested it, since building python on amd64).

[a1f769b2-b2b7-4237-92e2-33d87a4133db]

Is there any change to see this integrated soon ? The patch is only a couple of lines long,

thanks

[61337411-43fc-4a9c-b8d5-4060aede66d0]

Can you please provide some setup instructions for mingw-w64? What URLs should I install in what order, so that I can compile Python?

[a1f769b2-b2b7-4237-92e2-33d87a4133db]

I think compiling python itself with it would be quite difficult - I have never attempted it. My 'scenario' is building extensions against the official python for amd64 on windows.

The quickest way to test the patch would be as follows:

• take a native toolchain (by native, I mean runs on and target 64 bits windows subsystem - I have not tested cross compilation, or even using 32 bits toolchain on WoW). The one from equations.com is recent and runs well:

http://www.equation.com/servlet/equation.cmd?call=fortran

• builds a hello-world like python extension from the command line:

http://projects.scipy.org/numpy/wiki/MicrosoftToolchainSupport (I am sorry, the wiki page is bit messy, I will clean it up).

Since we use our own distutils extensions in numpy, I don't know how much is needed for support at the stdlib distutils level. If that's something which sounds interesting to python itself, I am willing to add support in python proper for mingw-w64.

[61337411-43fc-4a9c-b8d5-4060aede66d0]

Lowering the priority. It's too difficult to setup the environment to be able to reproduce the issue being fixed.

[a1f769b2-b2b7-4237-92e2-33d87a4133db]

The toolchain is difficult to build, but the patch is easy to review, it just moves the MS_WIN64 outside the MS compiler specific part. If it does not break the MS toolchain (the one used to build the official python), I don't see the problem.

I can make the toolchain available to you as a tarball, though, so that you can easily test from a windows command shell without having to install anything.

[a1f769b2-b2b7-4237-92e2-33d87a4133db]

Ok, it looks like following gcc 4.4.0 release, there is an installer for the whole toolchain:

http://www.equation.com/servlet/equation.cmd?call=fortran

This installs gcc (C+Fortran+C++ compilers, the download is ~ 40 Mb), and it can be used from the command line as conventional mingw32. Hopefully, this should make the patch much easier to test.

[61337411-43fc-4a9c-b8d5-4060aede66d0]

The toolchain is difficult to build, but the patch is easy to review, it just moves the MS_WIN64 outside the MS compiler specific part. If it does not break the MS toolchain (the one used to build the official python), I don't see the problem.

As a principle, I always try to reproduce a problem when reviewing a patch. In many cases, doing so reveals insights into the actual problem, and leads to a different patch.

That the patch is "harmless" is not a convincing reason to apply it.

I can make the toolchain available to you as a tarball, though, so that you can easily test from a windows command shell without having to install anything.

That would be nice.

[61337411-43fc-4a9c-b8d5-4060aede66d0]

I have now tried reproducing the problem, and still failed to. I downloaded, from http://sourceforge.net/projects/mingw-w64/files/, the distribution mingw-w32-bin_i686-mingw_20100123_sezero.zip.

With this, I get

c:\cygwin\mingw64\bin\gcc.exe -mno-cygwin -shared -s build\temp.win-amd64-2.6\Release\w64test.o build\temp.win-amd64-2.6\Release\w64test.def -LC:\Python26\libs -LC:\Python26\PCbuild\amd64 -lpython26 -lmsvcr90 -o build\lib.win-amd64-2.6\w64test.pyd c:/cygwin/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/4.4.3/../../../../x86_64-w64 -mingw32/bin/ld.exe: cannot find -lmsvcr90 collect2: ld returned 1 exit status

How am I supposed to link with this toolchain? If linking with -lmscvr90 is incorrect, what should I use instead? If it is correct, where do I get the proper import library from?

[2dcd154c-2b94-4a3d-83d0-353d59c98a2e]

Hi Martin,

It was nice meeting you at Pycon.

I finally took the time to set up a windows 64 bits environment, and here are the exact steps I needed to do to reproduce the issue, and fix it by hand:

• Start fresh from windows 7 64 bits
• clone python hg repository, and checkout the 2.6 branch (path referred as $PYTHON_ROOT from now on).
• build python with VS 2008 pro, 64 bits (release mode)
• install mingw-w64. I used this exact version: mingw-w64-1.0-bin_i686-mingw_20110328.zip (release 1.0, targetting win64 but running in 32 bits native mode), and unzipped it in C:¥
• I compiled a trivial C extension in foo.c as follows:

C:\mingw-w64\bin\x86_64-w64-mingw32-gcc.exe foo.c -I $PYTHON_ROOT\Include -I $PYTHON_ROOT\PC -shared -o foo.pyd $PYTHON_ROOT\PCbuild\amd64\python26.lib -DMS_WIN64

The patch removes the need for defining MS_WIN64. I don't know exactly the policies for which branch this could be applied - anything below 2.6 does not make much sense since most win64 users use 2.6 at least. Ideally, it could be backported to 2.6, 2.7 and 3.1 ?

[ee70db62-58c4-4ee1-bd9a-e1984d34a574]

Has anyone looked at this? I'm trying to build gdb with Python enabled with mingw-w64 (Python 2.7.1 with manually created import libraries), but have to manually define MS_WIN64 in the CFLAGS. The patch only does what's right (i.e. define a macro that should be defined). David has nicely explained what needs to be done to reproduce the issue.

Thanks!

[b510eab9-9bb7-4789-a06f-c0a1898e20d3]

I'm also using this patch successfully (together with http://tdm-gcc.tdragon.net/).

[9a2113a9-c5c2-4940-89fa-727c63758b8a]

Hi Martin

I have documented a build process for a Python package on MinGW-w64, including a requirement that this patch be applied. You might want to revisit it, given that your attempt to reproduce this bug earlier was unsuccessful.

http://ascend4.org/Building_ASCEND_for_64-bit_Windows

Please let me know if you're looking at it and if there's anything that needs to be clarified.

Cheers JP

[61337411-43fc-4a9c-b8d5-4060aede66d0]

John: in the current versions of the toolchain, Python's configure fails for me. I follow steps 1..3 of "Steps to date". Then running ./configure fails saying that it does not work. I then tried alternatively these three approaches:

1. set PATH to include /mingw/mingw/bin
2. set CC to /mingw/mingw/bin/gcc.exe
3. set CC to /c/mingw/mingw/bin/gcc.exe

Even though I can run gcc -v just fine, configure fails with

configure:3593: checking whether the C compiler works configure:3615: /mingw/mingw/bin/gcc conftest.c >&5 gcc.exe: error: CreateProcess: No such file or directory^ configure:3619: $? = 1 configure:3657: result: no

So apparently, mingw has some problem mapping the file name back to a Win32 path.

[b510eab9-9bb7-4789-a06f-c0a1898e20d3]

Marting, this issue is about building python extensions with mingw-w64 not about building python itself.

[9a2113a9-c5c2-4940-89fa-727c63758b8a]

Martin, Ralf is right and my as previously linked is about building a python extension. I should have been more explicit about that.

FWIW I found that the configure scripts on MinGW-w64 generally work fine if you add a "--build=x86_64-w64-mingw32" argument on the ./configure command line. Then you should only add /mingw/bin to your path (and edit /etc/fstab to map c:/mingw64 to /mingw). Adding those subdirectories to the PATH doesn't seem to be successful; I suspect that those binaries are off the standard path for some reason, eg internal use by GCC only, or something (GCC seems to do some clever tricks with relative paths, so I'm sure it's important that you use the correct starting executable).

[b510eab9-9bb7-4789-a06f-c0a1898e20d3]

There's no need to discuss or even run configure scripts. Martin, please reread the OPs original message. It's easy enough to reason about the issue instead of trying to reproduce it.

[9a2113a9-c5c2-4940-89fa-727c63758b8a]

This bug is still present in Python 2.7.5 on Windows 64-bit . I am currently providing the following instructions for MinGW-w64 users wanting to link to Python27.dll:

http://ascend4.org/Setting_up_a_MinGW-w64_build_environment#Setup_Python_for_compilation_of_extensions

[4c0bd4d3-02a8-45f7-a591-bce9e524070d]

I want to add that this bug led to bizarre behavior (described here: http://stackoverflow.com/questions/18646694/pass-pointer-from-c-to-python-w-boost-python) when using 64-bit Boost-Python compiled with Mingw-w64 in Windows 7. Boost-Python and programs linked to it compiled, but failed at run-time with segfaults. The solution described by jdpipe worked for me, but I only found it after a day of fruitless debugging attempts.

[cab0abe0-b98b-4df9-95b1-3bf3159047e2]

The problem is still present in python 3.4 with mingw gcc 4.8.2. I was having trouble with compiling radare2's python swig bindings. The solution described here: http://ascend4.org/Setting_up_a_MinGW-w64_build_environment#Setup_Python_for_compilation_of_extensions also fixes my problem when using generated dlls.

[aa2c5943-8264-4a78-97ed-7013d2cb52f6]

Still a problem with mingw-w64 gcc 5.1 and Python 3.4.3, time to fix this?

[b510eab9-9bb7-4789-a06f-c0a1898e20d3]

Well, the time to fix this would have been six years ago. The python core developers have shown a disinterest to fix problems with gcc on windows for a rather long time. I wouldn't expect this issue to be fixed.

[pfmoore]

Supporting mingw building of extensions has always been a complicated process, mainly because there appear to be many variants of mingw (and in particular, there seem to be multiple 64-bit builds). Add to this the fact that cygwin is sometimes used as a cross-compiler (often using the same code in distutils) and it's hard to know what to support, or what will break.

I would be very happy to see support for building of Python extensions using mingw (even though I use MSVC myself). However, in order for core Python to have a stable target to aim at, I believe the users of mingw need to work to provide an easy to use, unified platform that can be the target for core support.

I think the following would go a long way to giving the core developers (specifically me, I'd be willing to work on the core end of things to improve the situation) a chance of providing maintained support for using mingw as an alternative compiler for building Python extensions on Windows:

1. A well-documented and maintained process for setting up 32-bit and 64-bit mingw environments which can be used to build Python extensions. These need to use stable, well-maintained builds of mingw - ones that can be relied on to remain available in the long term (the links to equation.com and the Scipy instructions in this issue are now dead, for example, and both TDM and Mingw-W64 have been mentioned as toolchains, but it's not practical to try to support both - by support I mean at least that "some core developer installs them both and checks that things work"). Documentation patches to explain how to set up the official mingw build environment (once the community has agreed what it is) can be submitted to the packaging.python.org tracker.

2. Python 3.5 will be built using the MSVC 2015 "Universal CRT". From what I've seen on the mingw lists, there seem to be no plans for mingw to support this CRT in any form. Without that support, it's not clear to me how mingw builds for Python 3.5 will be possible, so the mingw community needs to work out how they will provide a solution for Python 3.5. Timely solutions for issues like this are needed if mingw is to be a supported build environment going forward. While it's understandable that the mingw community hasn't had much encouragement to work on things like this in the past, it is something that needs to change if full core Python support is to happen.

So that's the position, really. If the mingw community can come up with an "official" mingw distribution that supports building extensions for Python 32 and 64 bit, including for 3.5+, I will look at how we ensure that distutils supports that distribution going forward (that will likely mean *dropping* support for other mingw builds at some point). I know the Scientific Python users make a lot of use of mingw for builds, so maybe their standard build environment is a good target to aim for.

Bluntly, I'm not sure the community can achieve the above. It's a pretty hard goal, and lots of people have been working on it for a long time. But that's precisely why the core developers can't offer much in the way of support for mingw - it's too much of an unstable target as things stand. (Note for example, that I currently have *three mingw 64-bit installations on my PC, and they all work slightly differently - to the extent that I have projects that I can *only build with particular ones).

With regard to this specific patch, it seems that according to msg132798 it's easy enough to work around the issue by manually defining the symbol. The patch seems to apply cleanly, still, but I'm not aware if it has been tested on cygwin, for example. So, like Martin, I'm reluctant to apply it just because it *looks* harmless. Particularly this close to the 3.5 beta deadline.

[giampaolo]

For what it's worth I dropped support for mingw32 in psutil for exact the same reasons. As such I cannot imagine how hard could it be to add and maintain support for mingw in a much larger project as Python.

[83d2e70e-e599-4a04-b820-3814bbdb9bef]

Hunting around I found this on bpo-3871 https://mail.python.org/pipermail/python-dev/2013-January/123774.html.

From bpo-17590 upwards there are perhaps 25 issues with mingw in the title, so there's certainly work to be done. Please don't look at me, I'm simply not interested.

[bitdancer]

Please note that Paul is pretty new to the core team, and is a crossover with the packaging folks (which is mainly where the mingw issues lie). What this means to the mingw community is that with Paul on the core team and willing to work on the support, the mingw users have the best chance they've ever had of making progress on this.

[83d2e70e-e599-4a04-b820-3814bbdb9bef]

Does this mean scarce Windows resources being diverted off to what I consider a side show? Unless it's categorically stated that mingw is officially supported in which case fine, provided the experts index and everything else associated with official support is updated accordingly.

[pfmoore]

Not at all. Mingw support is important for the scientific community, as I understand it, and I'm willing to help there if I can. That won't be at the cost of other areas I can contribute to, but I consider packaging as much my area of expertise as Windows - and mingw support covers both of those areas.

To give some background, I was involved in adding mingw support for the MSVC 2010 builds of Python, which involved working with the mingw project on getting -lmsvcr100 support added. That was a battle, and I fully expect universal CRT support to be even harder[1]. I do *not* expect to get involved in that, but as I said, I do want it (along with a single mingw distribution blessed by the Python mingw user community) as a prerequisite for a higher level of core support.

That's (IMO) a *very* high bar, and I don't expect it to be easy for the mingw-using community to achieve it. But if they do, then the amount of effort involves deserves some recognition, and for my part I am offering some of my time improving core Python support on that basis.

[1] For example, AIUI, with the universal CRT, even the header definitions change - e.g., FILE is not based on an _iob symbol - so you have to know the target CRT at *compile time, not just at link time. That's an additional level of complexity not present in previous CRT releases.

[ee70db62-58c4-4ee1-bd9a-e1984d34a574]

Paul,

Thank you for your serious take on the issue.

I'm Ruben, a long-time contributor and 3-ish year toolchain builder for MinGW-w64. I originally helped patching Qt 4.5/4.6 so that it worked with MinGW-w64 on 64-bit Windows.

I can help liaison between you and the MinGW-w64 project, and point you towards stable, well-maintained MinGW-w64 builds. Perhaps we can discuss all the nitty gritty details elsewhere (or I can continue here of course), so we can get this straightened out.

In short, this is the story:

• MinGW.org is the "old" MinGW project, which has become pretty stale and is behind in soooo many aspects that it isn't even funny anymore (mostly new Windows APIs, DirectX support, and C++11 threading support is lacking). MinGW-w64 is a clean-room implementation that was released into the public domain, and by now included in all (I repeat, all) major Linux distro's as a cross compiler. Arch, Debian/Ubuntu, Fedora/Redhat etc. all provide a MinGW-w64 based cross-compiler. A complete list of MinGW-w64 features can be found on the web page: http://mingw-w64.yaxm.org/doku.php

• TDM is a famous name in the MinGW world because he provided a high quality toolchain with installer when MinGW.org lacked in providing the new GCC 4. Unfortunately, he applies (perhaps useful) patches which break binary compatibility with upstream GCC. Since my toolchains (first uploads in September of 2010, last one in June 2013), and later with the MinGW-builds toolchains which are now installable through the MinGW-w64 website directly (and shipped with Qt), there is really no reason to go look elsewhere for toolchains. The MSYS2 project also provides numerous binary packages and I think almost exactly the same toolchains within their environment. The official MinGW-w64 installer can be found here: http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/installer/mingw-w64-install.exe (it might occasionally complain it cannot download repo.txt, that's a Sourceforge redirect error thing that's proving might hard to fix, in any case you can also find the toolchains directly below) http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/ http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/

• There are several ABI incompatible variants, explained nicely on the Qt wiki: http://wiki.qt.io/MinGW-64-bit#Exception_handling:_SJLJ.2C_DWARF.2C_and_SEH http://wiki.qt.io/MinGW-64-bit#GCC_Threading_model_.28posix_vs_win32.29 This is a choice you'll have to make. The greatest compatibility is offered by the "posix threading" (which gives C++11 \<thread> support) and 32-bit sjlj and 64-bit seh variants. The 32-bit-dw2 provides a bit more juice in exception heavy code, but has the caveat that exceptions cannot be thrown across non-GCC-built code. The 32-bit dw2 variant is also what works with Clang, and is what is delivered in MSYS2 when installing the 32-bit toolchain. Since the 32-bit sturctured exception handling Borland Patents have expired, maybe a new, 32-bit seh version will emerge, but I have heard nothing concrete towards this direction.

• The MSYS2 project also provides GCC-built Python binaries, which of course needs quite heavy patching. These patches and build scripts can be found here: https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python3 https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python2

• the -mno-cygwin option is really a dinosaur, and should be thrown into a deep pit, buried, and forgotten. Cygwin provides MinGW-w64 cross-compilers ({x86_64,i686}-w64-mingw32-gcc) which work in exactly the same way as a Linux->Windows cross-compiler. The official Windows binaries are just "gcc" and work as any native compiler on Linux.

I hope this provides you with much of the information you need. If anything is unclear (I wrote this up in like ten minutes) or you need some concrete help figuring some things out, feel free to contact me directly or through the bugtracker. I sent an email to the MinGW-w64 public mailing list (subscription needed to post though, otherwise your message will be lost in the review queue) stating your and my intentions and current state of affairs: http://sourceforge.net/p/mingw-w64/mailman/mingw-w64-public/thread/CALc40c_p56PAg8mfPyoUvAKWsqn0PBYQmgQW_qFMcRupof8wGg%40mail.gmail.com/#msg34124287

Ruben

[pfmoore]

Ruben, Thanks for the detailed explanations. Just to be clear, I am *not* the person that will take this aspect of the process forward - that will be the community of people building (or wanting to build) extensions for Python with mingw. I don't know if that community has a spokesperson yet (or even if it's a well-defined "community") but they would be the ones to to engage with the mingw developers.

In particular, the choices around ABI incompatible variants that you mention are precisely the sort of thing the community needs to establish - which variant is compatible with Python, how to get a maintained build of that variant (there seems to be a lot of "get such-and-such's personal build from this URL" in the current crop of instructions for building Python extensions with mingw - that's not really sustainable).

The problem boils down to there needing to be a definitive, simple, and maintained set of instructions and software for "how to set up a mingw build environment for Python extensions". The core Python developers can't provide that (as we use MSVC). What we can do, when such a thing exists, is look at whether it's a toolchain that we can reasonably support.

At the moment mingw patch requests come in based on someone's custom build environment, that we can't easily reproduce, and we can't be sure is the same as anyone else's. That's not something we can support - hence the frustration from the mingw-using community, because we have partial support from the days when mingw.org and cygwin were the only two options for gcc-on-windows and we didn't really communicate the change in status (which admittedly would have been "we no longer support mingw", so wouldn't have helped much...)

Hopefully, the discussion on this issue clarifies the position a bit. Give us a well-defined "gcc on Windows" (mingw) platform definition, and we'll look at supporting it. Otherwise, we can maintain the status quo (what's there remains, but patches pretty much never go in, because reproducing issues and testing changes is too much effort to be viable) or formally drop support for mingw (which I'd be reluctant to do, but it may be worth it just to offer clarity).

[ee70db62-58c4-4ee1-bd9a-e1984d34a574]

Paul,

OK, I understand your point of view. As you say, there is no single "MinGW" community, nor a guiding body that takes these decisions. If you're not willing to choose one, all I can say is this: it will probably not matter which version you choose (all will work), only mixing them won't work.

A sound choice would be to follow the Qt Project (it's what they ship in their SDK): http://doc.qt.io/qt-5/supported-platforms.html#reference-configurations They chose the dw2/posix combo, which IMHO is the best choice all-round. For 64-bit, the obvious one is seh/posix. Incidentally, that's what MSYS2 people chose, and they regularly build all of Python with these toolchains (plus some other packages including numpy/scipy), see https://github.com/Alexpux/MINGW-packages/ and search for "-python-". These python builds are done from source, do not link to msvcr100, but just msvcrt.dll, which is the OS component MinGW GCC links to currently and in the foreseeable future.

As it stands, you can easily reproduce these builds by:

1. Installing MSYS2 (download and install, see http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/)
2. Installing GCC (i.e. "pacman -S mingw-w64-i686-gcc" for 32-bit or "pacman -S mingw-w64-x86_64-gcc" for 64-bit)
3. Installing all of Python's dependencies (see e.g. PKGBUILD: https://github.com/Alexpux/MINGW-packages/blob/master/mingw-w64-python2/PKGBUILD#L14)
4. running makepkg in the directory with the python PKGBUILD with the patches next to it. make sure to use the "MinGW-w64 Win64 Shell" or "MinGW-w64 Win32 Shell") that MSYS2 places in your start menu. This sets PATH to include the relevant toolchains and tools, much like the VS command prompts.

You can then extract the necessary dependency DLLs from the MSYS2's /mingw32/bin and /mingw64/bin directories so that a standalone MinGW-w64 Python installation can be created from that without issue.

If you feel this qualifies as an easy, maintainable, reproducible setup, perhaps work can be done to integrate the large amount of patches currently required. Note that these patches will work with any decent and/or official MinGW-w64 GCC build. The time of everyone needing to build their own toolchain is past. Even if some people still building all kinds of cludgy variants seem to have missed the memo.

Note that these builds differ from the official MSVC builds (which may not be a bad thing: it keeps MSVC\<->GCC compatibility issues arising from mixing the code to a minimum). Obviously, when MinGW-w64/GCC supports the UCRT stuff, this incompatibility can be harmoniously resolved. Until then, this seems to me like a good solution, and it is how everyone else is doing it (i.e. separate GCC builds on Windows).

If there is no interest in having a (community-supported, semi-official) GCC-built Python on Windows, I'm sure something else can also be worked out, which would include stripping the current dinosaur -mno-cygwin code which is what this bug was originally all about.

[pfmoore]

If there is no interest in having a (community-supported, semi- official) GCC-built Python on Windows, I'm sure something else can also be worked out, which would include stripping the current dinosaur -mno-cygwin code which is what this bug was originally all about.

The issue is about building Python *extensions* with gcc, not about building Python itself - that's not a supported approach at all (MSYS2 does it, but the patches used aren't going to be integrated in core Python).

The problem is that the people who build those extensions (which is not me, nor is it anyone on the core Python team) have never settled on a single version of the mingw toolchain as "what they want the distutils to support". So each bug report or patch is needs different "how to install mingw" instructions to be followed before a core developer can work on it.

I'm suggesting that the people raising distutils bugs about mingw support get together and agree on a common toolchain that they'll use as a basis for any future bugs/patches.

[9255d2b8-ed7c-4cd1-bf9c-ba91bbc36b00]

> If there is no interest in having a (community-supported, semi- > official) GCC-built Python on Windows, I'm sure something else > can also be worked out, which would include stripping the current > dinosaur -mno-cygwin code which is what this bug was originally > all about.

The issue is about building Python *extensions* with gcc, not about building Python itself - that's not a supported approach at all (MSYS2 does it, but the patches used aren't going to be integrated in core Python).

Let's say you have the official, upstream W32 CPython, built with MSVC and linking with, say, msvcr90.dll You build, say, libarchive-1.dll with MinGW-w64, because that's what you use to build stuff. Because it's MinGW-w64, libarchive-1.dll links to msvcrt.dll. Then you want to build, say, pyarchive extension for your Python and you do that with MinGW-w64, because that's what you use to build stuff. Because it's msvcrt.dll, libpyarchive.pyd link to msvcrt.dll. Then you run Python code that takes, say, sys.stdout file object and passes it to pyarchive. Python file object is backed by msvcr90 file descriptor. pyarchive expects a file object backed by msvcrt file descriptor. Boom.

Only three ways of avoiding this: 1) Use MSVC for everything. This is what upstream CPython does. 2) Use MinGW-w64 for everything (including CPython itself). This is what MSYS2 does. 3) Use MSVC for CPython and MinGW-w64 for the rest and somehow ensure that FDs are converted when passing the CRT barrier. And stat structs as well. And everything else that differs. This is a path of frustration that leads nowhere and does not seem sustainable to me.

This is why the discussion keeps coming back to building Python with MinGW-w64. This is why Universal CRT can be a solution (the absence of CRT incompatibility would resolve a lot of issues; the rest is manageable - remember MinGW-w64 has to use the same CRT/W32API DLLs that MSVC does, so binary compatibility is always achievable for anything with C interface).

[pfmoore]

On 19 May 2015 at 17:09, Руслан Ижбулатов \report@bugs.python.org\ wrote:

3) Use MSVC for CPython and MinGW-w64 for the rest and somehow ensure that FDs are converted when passing the CRT barrier. And stat structs as well. And everything else that differs. This is a path of frustration that leads nowhere and does not seem sustainable to me.

That is the one this issue is about. It *is* possible (mingw grew the -lmsvcr100 and similar flags, at least in part to support it). But it's not easy to set up, and the people asking for it to be supported have never really addressed all of the issues involved (at least not in a reproducible/supportable way).

Building Python with mingw, while out of scope for this particular issue, has always failed because nobody has been willing to step up and offer the long-term support commitment that would be required, AIUI.

[bitdancer]

My understanding matches yours, Paul. Core does not want to *distribute* a mingw built python, but if the mingw community came up with a support strategy, including one or more buildbots building using mingw, I believe that we would accept the patches. Basically, it has to meet the PEP-11 rules for supported platforms (including enough userbase to produce the people to maintain it long term :)

[3f481a25-d081-4bb7-ade9-aa917e2fce91]

A few comments from the perspective of what's needed for the scientific Python stack:

1. Of the three options mentioned in msg243605, it's definitely (3) that is of interest. We want to build extensions with MinGW-w64 that work with the standard MSVC Python builds. We've done this with mingw32 for a very long time (which works fine); not being able to do this for 64-bit extensions is the main reason why there are no official 64-bit Windows installers for Numpy, Scipy, etc.

2. There is work ongoing on a mingw-w64 toolchain that would work for the scientific Python stack: https://github.com/numpy/numpy/wiki/Mingw-static-toolchain. It actually works pretty well today, what needs to be sorted out is ensuring long-term maintainability. More details about what it's based on are provided in https://github.com/numpy/numpy/wiki/Mingw-w64-faq - I think it's consistent with what Ruben van Boxem recommends. Carl Kleffner, who has done a lot of the heavy lifting on this toolchain, is working with upstream mingw-w64 and with WinPython to ensure not creating yet another incompatible flavor mingw.

3. It's good to realize why making mingw-w64 work is especially important for the scientific Python stack: there's a lot of Fortran code in packages like Scipy, for which there is no free compiler that works with MSVC. So one could use MSVC + ifort + Intel MKL (which is what Enthought Canopy and Continuum Anaconda do), but that's quite expensive and therefore not a good solution for the many of contributors to the core scientific Python stack nor okay from the point of view of needing to provide official binaries that can be redistributed.

Paul's comments on picking a single mingw-w64 version, and that version not being a download from someone's personal homepage, make a lot of sense to me. We (Carl & several core numpy/scipy/scikit-learn devs) happened to have planned a call on this topic soon in order to move towards a long-term sustainable plan. I wouldn't expect everything to be sorted out in a couple of weeks (it's indeed a hard goal), but knowing that Paul is willing to review and merge patches definitely helps.

[pfmoore]

Ralf, thanks for the comments. The scientific community is definitely the key group that *need* mingw (as opposed to people who merely want to use it because they don't want to buy into MS compilers, or for similar personal reasons).

My personal view is that if the scientific community comes up with a mingw/gcc toolchain that they are happy with, and willing to support, then I would see that as a reasonable target to be "the" supported mingw toolchain for distutils.

I'd like to see a single-file "download, unzip, and use" distribution rather than the current rather intimidating set of instructions on how to set the toolchain up - but I'm sure that's part of what you're intending to cover under "ensuring long-term maintainability".

[3f481a25-d081-4bb7-ade9-aa917e2fce91]

Indeed, our idea of "easy to install" was/is a wheel or set of wheels so that "pip install mingw64py" does all you need. If necessary that can of course be repackaged as single download to unzip as well.

[8ab337ab-a4eb-4b8a-b8c0-09b9874c62a9]

I want to second Ralf's comments about both the need for this and it being easy to get.

What is required to make this happen (specifically the easy to install build chain - pip install...)? It would be good to enumerate the outstanding issues.

The current difficulty of building extensions on Windows should not be underestimated. Microsoft seem to change how their various tools work, with different updated SDKs and removing tools and changing things (even retrospectively) quite regularly.

I've wasted quite a bit of time setting up windows machines to build the various flavours (bits and python version), only to find that the same strategy doesn't for some reason beyond my comprehension doesn't work on a different machine. Throw in different windows versions and the problem is pretty insurmountable and unsustainable.

To be clear, the current situation surely cannot be worse than a MinGW situation.

[8ab337ab-a4eb-4b8a-b8c0-09b9874c62a9]

Of course, I mean:

*To be clear, the MinGW situation surely cannot be worse than the current situation.*

[01f443ed-975f-4d9c-825e-1328c16eb563]

The situation is not THAT bad. You can install a prerelease of mingwpy with pip:

pip install -i https://pypi.anaconda.org/carlkl/simple mingwpy

or with conda: (thanks to omnia-md)

conda install -c https://conda.anaconda.org/omnia mingwpy

It is not hosted on PYPI as long as there is no version for python-3.5.

[8ab337ab-a4eb-4b8a-b8c0-09b9874c62a9]

@carlkl right, but it's not really a seamless experience. I think my question is: What needs to still be done in order that a user with a fresh Python install in Windows (and no compiler installed) can do "pip install an_extension_that_needscompiling" and it _just works?

Is someone with a better understanding of the issues able to comment on this?

[zooba]

What needs to still be done in order that a user with a fresh Python install in Windows (and no compiler installed) can do "pip install an_extension_that_needscompiling" and it _just works?

The package developer takes the time to build a wheel on Windows (presumably they are already testing on Windows...?) and publishes it. Problem solved.

Most of our efforts are (or should be) aimed at making it easier for the developers to do this, rather than trying to make a seamless build toolchain for the end user.

[8ab337ab-a4eb-4b8a-b8c0-09b9874c62a9]

@Steve Great, so what needs to be done so that I as a package developer can do pip install windows-build-system, python setup.py bdist_wheel and it actually creates a wheel? (which AFAICT is the same problem).

My interest is precisely as a package developer. I've spent far far too much time fighting compilers on Windows (many days) and I don't want to do that any more. Every time I come across a new machine, I need to re-establish the current way to do things, which invariably doesn't work properly because I can't find the SDK to download, or the SDK installation doesn't include things any more, or some other reason which I can't work out. On Linux, everything is basically wonderful - I notice no difference between pure python packages and extensions packages.

[zooba]

As an occasional Linux user, I notice a huge difference between pure Python and extension packages there, but basically always have a compiler handy on my Windows machines. It's all about context and what you're used to :)

The advice has always been "Visual Studio X" is what is needed, and for 3.5 onwards that becomes "Visual Studio 2015 or later". Unfortunately, the story isn't so simple for legacy Python and 3.3/3.4 because those versions of VS are not so easy to get (unless you're a professional Windows developer with an MSDN subscription, which is pretty common). It is possible to use some other installers to get the old compilers, but Python was not designed to work with those and so there are issues that we cannot fix at this stage.

It also doesn't help that older versions of VC weren't as standards compliant, so people wrote code that doesn't compile when ported. There are also many dependencies that don't work directly with MSVC (for the same reason, but in this case it wasn't the package author's fault).

If you follow distutils-sig, where this occasionally comes up, you'll see the direction for packaging generally is to avoid needing to build. The hope is that even setuptools becomes nonessential enough that it can be dropped from a default install, but package developers will install it or another build manager to produce their packages (on Windows at least, though there's work ongoing to make this possible on many Linux distros too).

[3f481a25-d081-4bb7-ade9-aa917e2fce91]

@Steve Great, so what needs to be done so that I as a package developer can do pip install windows-build-system, python setup.py bdist_wheel and it actually creates a wheel? (which AFAICT is the same problem).

Hi Henry, I expect progress on the mingw-w64 front within the next few months. There'll be a status update with some more concrete plans soon. Also, http://mingwpy.github.io/ has appeared last week - a few wheels have been set in motion.

The advice has always been "Visual Studio X" is what is needed, and for 3.5 onwards that becomes "Visual Studio 2015 or later".

Hi Steve, that's actually not very useful advice for the scientific Python community. While things like C99 compliance are or could get better, there will always be a large Fortran-shaped hole in your suggestion. See my post above (from May 19) for more details.

[8ab337ab-a4eb-4b8a-b8c0-09b9874c62a9]

Thanks Ralf - I'm happy and keen to help so please feel free to poke me if you need assistance with anything. I'll keep an eye out too - is it actively being discussed on any list?

[zooba]

We are working on the Fortran issue in a few different ways, one of which is your work on MinGW compatibility so that gfortran can be used. But that isn't the only solution to this whole problem, and it's not even preferable for a lot of Python users, so expect other proposals to come out.

I'm not aware of any C99 limitations still present in VC14, so please let me know so I can file bugs against the team.

[njsmith]

We are working on the Fortran issue in a few different ways, one of which is your work on MinGW compatibility so that gfortran can be used. But that isn't the only solution to this whole problem, and it's not even preferable for a lot of Python users, so expect other proposals to come out.

Hey Steve, I'm a bit surprised to be hearing this now given all our off-list discussions about these issues this year. Can you clarify what you're talking about here? Who is "we", what other solutions do you see, and why would they be preferable? (If the compatibility issues are solved, then AFAIK gfortran is basically perfect for 99% of uses; the only alternatives are proprietary compilers with much nastier -- F/OSS-incompatible -- license terms. Note that by contrast gfortran itself is GPLed, but with a specific exemption added to clarify that it is totally okay to use for compiling proprietary code.)

[zooba]

"We" is a lot of different companies and individuals. Anyone distributing prebuilt binaries is helping here, a few people are working on the licensing concerns for some components, other people are working on C BLAS libraries.

I see the issue approximately as "it's hard to install the scipy stack", which is broader than "Windows does not have a Free Fortran compiler" and allows for more solutions (apologies for putting words in your mouth, which is not my intent, though I have certainly seen a fixation on this one particular solution to the exclusion of other possibilities).