Remove -mno-cygwin from distutils

[52e384d2-8525-4f37-9cb0-4e820faf100e]

BPO	12641
Nosy	@malemburg, @loewis, @doko42, @pfmoore, @pjeby, @pitrou, @tiran, @tarekziade, @jkloth, @jwilk, @ned-deily, @merwok
Files	pastie-2274486 issue12641-modernize_cygwin&mingw_compilers.tar.gz 0001-MINGW-issue12641-customize-mingw-cygwin-compilers.patch: customize compilers 0001-MINGW-issue12641-check-if-cygwin-mingw-compiler-supp.patch: restore -m{no-}cygwin if supported 0001-MINGW-issue12641-avoid-syntax-warning.patch check_mno_cywin_py27.patch check_mno_cywin_py3.patch test_mno_cygwin.tar.gz check_mno_cywin_py34.patch check_mno_cywin_py34_1.patch check_mno_cywin_py3_1.patch check_mno_cywin_py27_1.patch check_mno_cywin_py27_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/pitrou' closed_at = created_at = labels = ['type-bug', 'library'] title = 'Remove -mno-cygwin from distutils' updated_at = user = 'https://bugs.python.org/jonforums' ``` bugs.python.org fields: ```python activity = actor = 'ned.deily' assignee = 'pitrou' closed = True closed_date = closer = 'pitrou' components = ['Distutils', 'Distutils2'] creation = creator = 'jonforums' dependencies = [] files = ['22775', '29030', '29031', '29032', '29103', '30681', '30682', '30683', '30698', '30888', '30889', '30890', '31919'] hgrepos = [] issue_num = 12641 keywords = ['patch'] message_count = 86.0 messages = ['141172', '141173', '141230', '141231', '141265', '141271', '141277', '141296', '141301', '141613', '141614', '141661', '141662', '146384', '146499', '161514', '165178', '165181', '165182', '165184', '165210', '167620', '179175', '179176', '181830', '181831', '181834', '181865', '182090', '182277', '182726', '182737', '183032', '183048', '183077', '183081', '183213', '183224', '183227', '183240', '183241', '189748', '189756', '189769', '189770', '189786', '189805', '189813', '189814', '189871', '189874', '189878', '189887', '189906', '189941', '189943', '189946', '189947', '189948', '189958', '189962', '190060', '191714', '191735', '191749', '191753', '191755', '191845', '192758', '192763', '192764', '192766', '192877', '195576', '195809', '198650', '198690', '198691', '198732', '198733', '198734', '198763', '207498', '207499', '238062', '238067'] nosy_count = 35.0 nosy_names = ['lemburg', 'loewis', 'doko', 'paul.moore', 'pje', 'geertj', 'pitrou', 'christian.heimes', 'schmir', 'tarek', 'jkloth', 'jwilk', 'ned.deily', 'eric.araujo', 'rpetrov', 'cmcqueen1975', 'rubenvb', 'santoso.wijaya', 'alexis', 'python-dev', 'Seppo.Yli-Olli', 'jonforums', 'RubyTuesdayDONO', 'Jeffrey.Armstrong', 'danmbox', 'Martin.Fiers', 'oscarbenjamin', 'Pete.Forman', 'fratti', 'rivy', 'gaudyallure52', 'honorableinvasi', 'stakingrainbow2', 'lewisl', 'Michael.Clerx'] pr_nums = [] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue12641' versions = ['Python 2.7', 'Python 3.3', 'Python 3.4'] ```

[52e384d2-8525-4f37-9cb0-4e820faf100e]

The use of -mno-cygwin at http://bit.ly/qAIsZV causes build failures with versions of MinGW GCC. For example http://bit.ly/llK3f3

I recently ran into it trying to build a Mercurial snapshot using a MinGW GCC v4.6.2 flavor http://pastie.org/2274486

Why is -mno-cygwin still be used, and is there any reason why it can't be removed?

Thanks, Jon

[ee70db62-58c4-4ee1-bd9a-e1984d34a574]

I can confirm the option has been removed. For those that don't want to believe a random person's comment on a bugtracker, here's the commit:

http://repo.or.cz/w/official-gcc.git/commit/2637551aa9314c46cf4205d435ab5e8944e9ac8a

The changelog is a bit cryptic, but note the last two items "removing documentation from removed option".

[merwok]

Hi! Thanks for the report. This can’t be fixed in 2.5, 2.6 or 3.1, which are in security-fix only mode, but we can do something for the active branches.

A quick web search finds reference of this deprecation/removal as far as 2007. Does anyone have more official documentation? If we can find the first gcc version that deprecates this option, then I’ll be able to make a patch with a version check (in order not to change previously working code).

(In the future, it would be nice of you not to use URL obfuscators or pastebins. Such services offer no guarantee of continued availability of hosted resources or links, but it is useful to keep information on this bug tracker for future reference. Thank you.

The first link is this: http://hg.python.org/cpython/file/4feb889d3bed/Lib/distutils/cygwinccompiler.py#l296 The second: http://stackoverflow.com/questions/6034390/compiling-with-cython-and-mingw-produces-gcc-error-unrecognized-command-line-op The compile log from the third link is attached as a file.)

[merwok]

I forgot to answer this:

Why is -mno-cygwin still be used, Simply because distutils had no dedicated maintainer for a long time, and because nobody was aware of this change. I don’t think any Python core developer is using Cygwin.

and is there any reason why it can't be removed? For distutils, we have to work on eggshells when fixing bugs to make sure we don’t break third-party code that relies on known bugs or works around them. That’s why I propose adding a version check* instead of just removing the option.

• i.e. removing -mno-cygwin if self.gcc_version >= some version

[ee70db62-58c4-4ee1-bd9a-e1984d34a574]

Does anyone have more official documentation?

The commit I linked to has the full option removal at October 7 2010 (see the fifth item in the changelog entry). Any GCC (major) version released after that will have it completely removed.

Deprecation happened (according to the first doc change mentioning this option as being deprecated) on 2009-03-25: http://repo.or.cz/w/official-gcc.git/commit/6609be22531f447aeddbb3f670ad7883036cb23f

It looks like GCC 4.4 and up have this change in them, by looking at their commit logs. Anyone using GCC 4.3 on MinGW should be shot (but that's just my humble opinion, rather radical I admit ;-) )

Note this has really nothing to do with current Cygwin or the cygwin platform; MinGW was part of the Cygwin source for a long time, and every MinGW compiler was burdened to search the Cygwin header and lib paths. These commits fixed that problem.

MinGW(-w64) is a native compiler for Win32 just as MSVC is. Links to msvcrt.dll and has no POSIX translation/compatibility layer like the Cygwin DLL.

May I ask for a reconsideration to commit a fix for this for Python 2.7 at least? With the version check it doesn't hurt anyone, instead it helps prevent further confusion and support requests on the MinGW side. Distutils pop up everywhere, and the projects depending on them rely on them working correctly. Thanks

[52e384d2-8525-4f37-9cb0-4e820faf100e]

I have confirmed on Win7 that the following 32bit MinGW flavors still recognize -mno-cygwin option and build without error:

(4.5.2) http://tdm-gcc.tdragon.net/download

(4.5.4) http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Automated%20Builds/ using the mingw-w32-1.0-bin_i686-mingw_20110624.zip download

...by compiling with:

[i686-w64-mingw32-]gcc -Wall -mno-cygwin -o helloworld.exe helloworld.c

...where helloworld.c was:

#include <stdio.h>
int main(int argc, char *argv[]) { printf("Hello World!\n"); return 0; }

I have not yet checked the latest mingw.org download from the following, but I expect that it also recognizes -mno-cygwin. I will check later and report back if appropriate.

(4.5.2) http://sourceforge.net/projects/mingw/files/

Given Ruben's 4.4 comment, something is odd here. Looks like some gcc source spelunking as these doc links make no mention of the removal:

http://gcc.gnu.org/gcc-4.6/changes.html http://gcc.gnu.org/gcc-4.5/changes.html http://gcc.gnu.org/gcc-4.4/changes.html

and the gcc manual is inconsistent. This summary mentions the option

http://gcc.gnu.org/onlinedocs/gcc-4.6.1/gcc/Option-Summary.html#Option-Summary

but the details sections don't mention the option

http://gcc.gnu.org/onlinedocs/gcc-4.6.1/gcc/i386-and-x86_002d64-Windows-Options.html#i386-and-x86_002d64-Windows-Options

http://gcc.gnu.org/onlinedocs/gcc-4.6.1/gcc/i386-and-x86_002d64-Options.html#i386-and-x86_002d64-Options

Ah, our good friend grep.

[52e384d2-8525-4f37-9cb0-4e820faf100e]

should the question be "what's the first mingw gcc version that -mno-cygwin usage unnecessary" rather than finding the first version the option was removed?

and, does it matter whether you're building on win for win, or cross compiling for win from nix?

sadly, i don't know gcc internals but a post to the mingw and/or mingw-w64 ml's should get the right people weighing in.

btw, can distutils do a lightweight configure-like check for feature presence rather than going a gcc version check?

[merwok]

May I ask for a reconsideration to commit a fix for this for Python 2.7 at least? With the version check it doesn't hurt anyone There’s a misunderstanding: I explained why 2.5, 2.6 and 3.1 can’t be fixed, but if you look at the versions at the top of this page, you’ll see 2.7, 3.2 and 3.3 listed. This will get fixed in these versions.

Ah, our good friend grep. Thanks for the exploration. This looks complicated.

should the question be "what's the first mingw gcc version that -mno-cygwin usage unnecessary" rather than finding the first version the option was removed? If removing the option in these versions causes no change (i.e. doesn’t break any code), I agree.

and, does it matter whether you're building on win for win, or cross compiling for win from nix? I’m afraid I don’t know enough about Windows and MinGW to answer that. If we can’t be sure about versions and consequences here, I’ll go to the MinGW ML.

btw, can distutils do a lightweight configure-like check for feature presence rather than going a gcc version check? There is a config command that’s not very used, but it would be IMO overkill to use it in the compiler code.

[52e384d2-8525-4f37-9cb0-4e820faf100e]

> and, does it matter whether you're building on win for win, or cross > compiling for win from nix? I’m afraid I don’t know enough about Windows and MinGW to answer that. If we can’t be sure about versions and consequences here, I’ll go to the MinGW ML.

Ruben's on it :) http://sourceforge.net/mailarchive/message.php?msg_id=27864860

[52e384d2-8525-4f37-9cb0-4e820faf100e]

are you ok with a targeted patch similar to what's being discussed at

http://sourceforge.net/mailarchive/message.php?msg_id=27895558

assuming the regex search the output of gcc -dumpspecs idea is valid?

[d8d5aad8-e55b-4500-a3a0-9ea982d771ff]

it is save to remove -mno-cygwin from Mingw32CCompiler

[merwok]

The necessity of walking on eggs with the distutils codebase restrains me. I’ve read the thread on sourceforge (thanks Ruben) but don’t have enough information yet to decide whether to do a version check, call gcc -dumpspecs or remove the option altogether.

BTW, there’s no need to use re.search if you’re not using a regular expression: “'no-cygwin' in output” will work.

[52e384d2-8525-4f37-9cb0-4e820faf100e]

shortly after opening this issue i removed -mno-cygwin from my 2.7.2 install and have had no issues on win7 32bit. but i understand you're hesitation.

regardless what you decide, please consider placing a summary note in the source comments as a safety net.

if you'd like me to try building a limited set of extensions to help with your decision, let me know. i have the following mingw toolchains on my win7 32bit system:

• tdm gcc 4.5.2 - http://tdm-gcc.tdragon.net/
• mingw-w64 prefix-stripped gcc 4.5.4 - http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Automated%20Builds/
• Ruben's gcc 4.6.2 personal build - http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/rubenvb/4.6.2-1/

i'm not speaking for Ruben, but as he maintains https://github.com/rubenvb/MinGW-w64-build-scripts you might try cajoling him if you feel you need more test results ;)

re: re.search, understood, thanks...quick-testing.

[37f6a81f-86b0-4584-a6b5-eeccc376bdc3]

Would it be practical to have a trivial compilation test to see if we are capable of using GCC with -mno-cygwin and if so, use it, otherwise drop off? I think GNU autotools uses a similar strategy for detecting compiler capabilities.

[merwok]

Would it be practical to have a trivial compilation test to see if we are capable of using GCC with -mno-cygwin and if so, use it, otherwise drop off? I think GNU autotools uses a similar strategy for detecting compiler capabilities.

The config command does such things, but the compiler classes don’t. I’d prefer something more lightweight, like a version check (see msg141231).

[afcdb1a8-bc5d-4852-a2a5-71b4ae6361ae]

Just adding a data point: I use the "mingw32" compiler setting with the Cygwin GCC installation, to compile extensions for standard Windows Python (not Cygwin Python). It would be good if any change doesn't break this scenario -- assuming it's not already broken in newer Python versions.

(For this scenario, AFAIK, the no-cygwin option is required, or at least I believe it was the last time I intentionally compiled something.)

[f1e1d87a-4340-4801-8770-9e5b7119ec5a]

I've come across this issue when trying to build extensions for Python 3.3 on Windows, needing a recent enough MinGW to provide a library for msvcr100. See issue bpo-15315.

[61337411-43fc-4a9c-b8d5-4060aede66d0]

I have gcc 3.4.4 in my cygwin installation, and it still needs the -mno-cygwin option, else the resulting binary will link with cygwin1.dll (which is undesired).

[61337411-43fc-4a9c-b8d5-4060aede66d0]

I just installed the cygwin gcc4 package; this gives me gcc 4.5.3. In this version, -mno-cygwin is still recognized, and gives this error message:

gcc: The -mno-cygwin flag has been removed; use a mingw-targeted cross-compiler.

So it seems that removing the -mno-cygwin flag is definitely incorrect, as it will result in a cygwin binary.

I then installed the mingw-gcc-core package, which gave me the utility i686-pc-mingw32-gcc, which indeed is able to create a (nearly) correct binary.

So I think we should check whether i686-pc-mingw32-gcc exists in the path, and if so, use it (without a -mno-cygwin flag). If it doesn't exist, we should continue to use gcc -mno-cygwin.

My remaining concern with i686-pc-mingw32-gcc is that it still links with msvcrt.dll in addition to linking with msvcrXY.dll; this is incorrect.

There is another concern that this applies to 32-bit mode only; in 64-bit mode, i686-w64-mingw32-gcc should be used. However, this is bpo-4709.

[61337411-43fc-4a9c-b8d5-4060aede66d0]

The 64-bit compiler is actually called x86_64-w64-mingw32-gcc.

[f1e1d87a-4340-4801-8770-9e5b7119ec5a]

It would be great if this could be sorted out in time for Python 3.3. Otherwise I don't think we'll be able to use MinGW to build extensions in Windows. Unless there is a version of MinGW which supports the -mno-cygwin option, as well as libmsvcr100.

[ee70db62-58c4-4ee1-bd9a-e1984d34a574]

Checking for a compiler's file name is stupid. Native Windows gcc is just "gcc.exe", Cygwin native GCC is also "gcc". Some have a lot of toolchains in PATH at the same time. That's not the right way to handle this kind of thing.

[b2022d64-94bc-4cb2-9125-841eb8b64cb7]

*bump*

I just installed MinGW 2.6.2 32-bit on Windows XP. It doesn't accept -mnocygwin and there is no binary "i686-pc-mingw32-gcc" either.

It would be great if you could agree on an approach and get this fixed. This impacts a lot of users that want to build extensions on Windows. In the mean time users can find a hack to work around the issue here: https://gist.github.com/4466320

[61337411-43fc-4a9c-b8d5-4060aede66d0]

Geert: Someone will have to provide a patch, else the issue will go nowhere. I will insist that the patch continues to support "older" gcc installations (from 2012), at least if implemented as a bug fix to older Python releases.

[d8d5aad8-e55b-4500-a3a0-9ea982d771ff]

In scope of this issue I would like to propose following patch set. First step is remove checks for versions used in past millenium, i.e. to avoid checks for 15 year old binaries.

[d8d5aad8-e55b-4500-a3a0-9ea982d771ff]

Next step is to propose customization for cygwin&mingw compilers.

[d8d5aad8-e55b-4500-a3a0-9ea982d771ff]

And optionally If someone disagree options m{no-}cygwin to be removed I would like to propose a patch '..check if cygwin/mingw... -m{no-}cygwin' to restore it for GCC before 4.6x.

[doko42]

2.7, 3.3 and 3.4 now have AC_CANONICAL_HOST. So the correct compiler should be picked up. please could somebody validate this? or are explicit --host and --build options required for configure?

[61337411-43fc-4a9c-b8d5-4060aede66d0]

doko: this issue is about Windows; autoconf is not being used here (plus the "correct" compiler would be MSC).

[d8d5aad8-e55b-4500-a3a0-9ea982d771ff]

Hi Matthias,

This issue is only for windows. In scope autotool based builds compiler customization is used to 'transfer' some build settings (flags, options) to distutils. This include compiler set in make macro (variable) CC.

Transfer is not complete but this is distutils issue out of scope here.

P.S. (uploaded a patch to avoid syntax warning)

[9d895cda-1d24-4f05-8b54-d0e2a02f6773]

Guys, this looks really bad and inconveniences a lot of users. You install the latest MinGW and Distutils from their default location, try using them on **anything that requires compilation**, and get the cryptic gcc -mno-cygwin error (after having to edit the obscure distutils.cfg, of course).

Aren't Python / distutils supposed to be cross-platform? It's already hard enough to find distutils / pip setup instructions for Windows, shouldn't they at least **work**? After removing -mno-cygwin from cygwincompiler.py, I get another obscure -mdll error. This is ridiculous.

If you can't agree on a patch that detects both new and old compilers, can't you split cygwincompiler.py into several versions, or somehow provide separate mingw32-old and mingw32-new options?

[d8d5aad8-e55b-4500-a3a0-9ea982d771ff]

Dan added the comment:

Guys, this looks really bad and inconveniences a lot of users. You install the latest MinGW and Distutils from their default location, try using them on **anything that requires compilation**, and get the cryptic gcc -mno-cygwin error (after having to edit the obscure distutils.cfg, of course).

Aren't Python / distutils supposed to be cross-platform? It's already hard enough to find distutils / pip setup instructions for Windows, shouldn't they at least **work**? After removing -mno-cygwin from cygwincompiler.py, I get another obscure -mdll error. This is ridiculous. Yes . This is reason to pack many changes in one archive "bpo-12641-modernize_cygwin&mingw_compilers.tar.gz \http://bugs.python.org/file29030/issue12641-modernize_cygwin%26mingw_compilers.tar.gz\", i.e. to remove all checks for tools used in previous millеnium.

My oldest compilers are : a) i386-mingw32msvc-gcc (GCC) 3.4.5 (mingw special) Copyright (C) 2004 Free Software Foundation, Inc. b) gcc.exe (GCC) 3.4.5 (mingw-vista special r3) Copyright (C) 2004 Free Software Foundation, Inc.

Check for -m{no-}cygwin flags is optional. I can not found reason this patch to be applied, as with implementation of compiler customization this is for developer guide.

Roumen

[05ddb1fc-98be-4c1a-a11e-4a6c9c9d41f6]

This also affects our software. I agree with Dan (danmbox): I don't understand; so many people depend on it and yet an out-of-the-box solution doesn't work.

I don't want to break the distutils package of our users because we use mingw. Within one Python script, I managed to fix it using this before the setup call:

if isWindows():
    """
    Fix bug in cygwinccompiler: removed -mno-cygwin. This is fixed in cygwinccompiler_new.
    We hacked the distutils.ccompiler : def new_compiler : It uses sys.modules to fetch the compiler
    By modifying the sys.modules, we can choose our own compiler version.
    (this is a bug that's out there for quite some time)
    """
    import cygwinccompiler_new
    import distutils.cygwinccompiler
    import sys
    sys.modules["distutils.cygwinccompiler"] = cygwinccompiler_new

..if I then later run setup(...), it will use my new cygwinccompiler_new, that has the '-mno-cygwin' line removed.

However, when you want to install new packages using pip from the command-line, I cannot find a suitable fix (except if I would replace the distutils.cygwinccompiler before pip'ing, then put it back). For afaik, distutils cannot be virtualenv'ed, right? So we cannot even fix the issue in a virtual environment.

If it is not possible to find out what version of gcc implemented it first; can't you simply use a pragmatic solution and run "gcc -mno-cygwin": if it gives an error, then remove the option. That would need the least testing and would fix the issue.

[9d895cda-1d24-4f05-8b54-d0e2a02f6773]

Nice partial work-around.

I think it's quite clear that the decision makers for this bug have not been making rational decisions for a year and a half, so we can't really expect change. This being the open-source world, the only recourse is publicizing the issue (or is there some bounty system for python?). So I suggest you blog about it.

[merwok]

I don't understand; so many people depend on it and yet an out-of-the-box solution doesn't work.

It is because of the combinations of some facts:

• Python contributors are volunteers;
• distutils is used a lot and quite brittle, which is a combination that makes us extremely cautious with changes;
• I am not a Windows user or developer;
• Martin knows Windows but apparently not Cygwin;
• nobody submitted patches until this month;
• nobody reviewed patches after they were attached.

You can choose to believe that we are unresponsible, stupid, etc. or accept that in an open, volunteer-based project, subtle changes in a niche part of the code are not committed right away. Whatever we do, we get blamed for moving too slowly and for changing Python too fast (not by the same people :).

The good news is that Roumev submitted clear patches, and that Trent Nelson put together Snakebite, a set of computers for Python developers that run various OSes including Windows. Using these machines to test the patches, we can now make progress.

[05ddb1fc-98be-4c1a-a11e-4a6c9c9d41f6]

Dear Eric,

I never said that anyone of these volunteers is unresponsible/stupid/whatsoever. It was also never my intention to express myself in this way, so I apologize if you felt harmed in any way. I was just suprised that this issue exists for so long. And since I only recently started developing on Windows, I was a bit knocked over by the additional effort that was needed to get things working (for which I blame Windows, not Python!). I've been developing for some time now and I understand completely that it's far from trivial; given the complexity of the software and fragility of distutils. So kudos to the developers that put their hours into this!

Anyway, I have a workaround using some distutils manipulations now (see my previous post), which works for now, and I'm looking forward to a permanent fix!

With best regards, Martin

[9d895cda-1d24-4f05-8b54-d0e2a02f6773]

That's bull, Eric. This is not about a corner case in cygwin. This is about mingw, which is the **main free software that builds executables on Windows**. You know, for when you don't want to require your users to install Visual Studio.

Additionally, both you and Matthias imposed artificial conditions that made it unlikely for patches to be created (search for "will insist").

Now, I have to agree that the larger python community (and not an under-resourced team like your good selves) should be involved in distutils (or choose and **support** a different package manager). But I'm not sure where I could file a bug for that (again, blogging may be the best choice).

[merwok]

[Martin] Thanks for the message. My previous message actually quoted Dan, not you, so no apology is necessary.

[Dan]

That's bull, Eric. Could you rephrase this? English is not my native language.

This is not about a corner case in cygwin. Oops I confused cygwin and mingw, apologies.

This is about mingw, which is the **main free software that builds executables on Windows**. You know, for when you don't want to require your users to install Visual Studio. Let me state that I fully sympathize with people who use Windows and still want to use free software as much as possible, even though some version of Visual Studio is now available at no cost. Again, the fact is that many core developers don’t use Windows, and the official CPython builds use MSVC, so from this viewpoint, MinGW is a niche platform for CPython. It does not mean that patches are rejected, just that it’s harder for me to assess them.

Additionally, both you and Matthias imposed artificial conditions that made it unlikely for patches to be created (search for "will insist"). Martin used that in a comment about backward compatibility, which we do take seriously. Roumen’s patches don’t break backward compat if I read them correctly.

Now, I have to agree that the larger python community (and not an under-resourced team like your good selves) should be involved in distutils Yes, more feedback, bug reports, patches and reviews from the community would help.

(or choose and **support** a different package manager). But I'm not sure where I could file a bug for that An effort is ongoing to define the successors of distutils. One problem is that it lacks features (like dependencies), and at the same time it tries to be too much (build, upload, install). The community (Python core developers, packaging tools developers, users) is working on these things.

(again, blogging may be the best choice). If you want to get involved in the packaging discussions, you can join the distutils-sig mailing list. If you want to push this specific issue forward, testing the patches on your system would help. If you want to support Python on MinGW in general, you can contribute a buildbot to our continuous integration system, so that we can see when things break.

[52e384d2-8525-4f37-9cb0-4e820faf100e]

[Eric]

> This is about mingw, which is the **main free software that builds > executables on Windows**. You know, for when you don't want to require > your users to install Visual Studio. Let me state that I fully sympathize with people who use Windows and still want to use free software as much as possible, even though some version of Visual Studio is now available at no cost. Again, the fact is that many core developers don’t use Windows, and the official CPython builds use MSVC, so from this viewpoint, MinGW is a niche platform for CPython. It does not mean that patches are rejected, just that it’s harder for me to assess them.

I'm going to advocate why mingw-based toolchains (mingw.org and mingw-w64) should be considered important platforms for CPython by giving some background info.

An important part of this issue has nothing to do with the politics of "free software" and everything to do with the technical benefits of using mingw-based toolchains on Windows. While you may be aware of the technical benefits, the following are important to me:

1a) MSVC forces you to link against a specific CRT version, and I don't get to decide which CRT version via a build setting. To the best of my knowledge the only way to link against another CRT version is to use another MSVC version. Many times it's important for all parts of a codebase to link against the same CRT version to prevent well-known cross-CRT object usage failures. While Mingw toolchains typically link to msvcrt.dll by default, they allow you to specify a CRT version via spec file tweaks.

1b) Often the most practical/stable way to build mixed codebases using CRT functionality is to link everything against msvcrt.dll. Using MSVC, the only way I'm aware of is to use old-and-slow VC6 which is not easily available. Updated and performant mingw toolchains enable one to link against msvcrt.dll and other CRT versions via spec file hackery.

2) While VC++ is a good IDE, many of us would like to simply use the command line compiler tools and not clutter our environment with a huge IDE we never use. MSFT has a poor record on this front, and after many fine years of making the command line tools available as part of the Windows SDK, they've regressed back to their lame ways in Windows 8 by removing the cmd line compiler tools from the SDK and embedding into the IDE similar to the old VC6 days. Enticing as it may be, I'll stop here rather than lurch into sailor-speak rage rant.

There are other benefits, but (1a) and (1b) are very important for many real-world usage scenarios on Windows. For example, easily building mercurial with mingw on Windows that work with the official MSVC-built Python. Extend this to other widely used Python and non-Python libraries. Extend this to those building binary libraries on Linux/OSX via mingw for their Windows users.

Also, there are other build automation awesomeness things you can do with mingw toolchains. For example, wouldn't it be great if you could download your source software, a complete mingw toolchain, and build the software on Windows in a sandbox using the mingw toolchain? You can with mingw and mingw-w64 toolchains. Nothing to pre-install and highly automated with minimal user intervention.

[9d895cda-1d24-4f05-8b54-d0e2a02f6773]

There's an additional aspect. Even though I don't use Windows as a development platform, I care about being able to propose Python to clients as a cross-platform technology (instead of, say, Java). Having an essential piece of Python infrastructure fail miserably on Windows impacts Python's appeal / credentials.

To clarify another point, I think that Python should pick and support an "official" package manager (be it distutils, pip or whatever) not eventually, but ASAP. For the time being it looks like distutils is a keystone, therefore it needs to work.

[52e384d2-8525-4f37-9cb0-4e820faf100e]

Eric, I'm assuming you are the key decision maker for any fix. As I want to see this issue stay razor focused, where specifically do you need community help?

Depending upon your needs, I can test on my win7 32bit notebook with official python 2.7.3. It also has arch 3.7.9 and ubuntu server 12.10 virtualbox vm's. I don't build python from source with mingw and my win8 64bit box isn't ready yet. The win7 box has mingw gcc 4.6.2 32bit and mingw-w64 32bit toolchains, both of which have msys and autohell goodies. I'm swamped these days but will carve out time from somewhere.

[a35477f8-71c1-46d8-a00e-b2f89ce712bf]

I'd really like to get a resolution on this issue so I've tried to gather some more information about this problem by asking some questions in the mingw-users mailing list. The resulting thread can be found here: http://comments.gmane.org/gmane.comp.gnu.mingw.user/42092

This issue concerns users of distutils --compiler=mingw32 mode. Normally this is used to build with mingw gcc in which case it is currently broken because of -mno-cygwin. It has been suggested above that some may be using it to build with cygwin's gcc. The -mno-cygwin option is there for those using cygwin's gcc.

To summarise the points (see the thread on mingw-users for more info): 1) -mno-cygwin has never had a meaningful effect in mingw. 2) -mno-cygwin produces an error in recent (~2011 onwards) mingw releases and will do for all future releases. This prevents distutils from building extensions with mingw. 3) -mno-cygwin only ever had a meaningful effect for cygwin's gcc 3.x where it could be used to build binaries that did not depend in cygwin.dll. 4) -mno-cygwin was always considered an experimental feature and its use is discouraged. 5) -mno-cygwin was removed from cygwin's gcc in the transition from 3.x to 4.x without any deprecation period (as it was an experimental feature). In gcc 4.x it produces an error preventing build. 6) The recommended way to replace the -mno-cygwin option is either to use mingw or to use cygwin's cross-compilers.

So there are two types of breakage affected by -mno-cygwin:

A: Anyone trying to use recent and future mingw versions to build extensions with distutils in the way that is described in the distutils docs. For this group distutils has been broken for 2 years and will continue to be until the -mno-cygwin option is removed.

B: Anyone who is using distutils with --compiler=mingw32 but using cygwin's gcc 3.x instead of mingw's gcc to build an extension for a non-cygwin Python. For this group removing the -mno-cygwin option would result in unusable extension modules. (the resulting .pyd requires cygwin.dll but is to be used in a non-cygwin Python).

Firstly, note that users in group B must surely be a group that diminishes with time since they are using the legacy gcc 3.x cygwin compiler. Similarly, since neither of mingw or cygwin will ever bring back -mno-cygwin users in group A will only increase with time. (I am such a user and have been manually removing all reference to -mno-cygwin from distutils for 2 years now.) This means that the balance of breakage will only move towards group A over time.

Secondly, any users in group B will suffer the same problem as users in group A if they try to use gcc 4.x. However this has not been reported on the tracker (I read through all matches in a search for '-mno-cygwin'). I think this serves as an indication of how many people are actually using this setup.

Thirdly, the -mno-cygwin option is a now-abandoned, experimental feature of a legacy compiler. Its creators did not feel the need to give it a deprecation period and its use is discouraged by both mingw and cygwin.

Bringing these points together: not removing -mno-cygwin from distutils trades the possible breakage for possibly non-existent users of the obscure, legacy, and generally considered broken setup B against the definite, known breakage for users of the appropriate documented setup A. I think this should be enough to say that the fix for the next Python version should simply remove all reference to '-mno-cygwin' as I have been doing for 2 years now without problem. The only users who can be adversely affected by this are those in group B who decide to upgrade to non-cygwin Python 3.4 (while still using an ancient cygwin gcc to build extensions). The suggested fix can be either to use mingw or to setup the cross-compilers in their cygwin installation.

For Python 2.7, 3.2 and 3.3 I think that this should be considered a bug that can be fixed in a bugfix release. However in that case it may be considered inappropriate to risk the small possibility of users in group B experiencing breakage. Since such users must be using cygwin's gcc 3.x, I propose that distutils check the gcc version and only add '-mno-cygwin' if the major version is 3. This will not adversely affect users in group B and will fix the problem for users in group A. Users in group B who attempt to use gcc 4.x will find that they get a different error message (at import time instead of build time) but that their setup will still be just as broken as it was before this change.

Thanks, Oscar

[pfmoore]

+1 for Oscar's proposed fix. It sounds like a sensible approach.

[4a48c1a3-d390-4d6f-967c-42eaeef5e2a4]

Another +1 for Oscar. I've just done an install of Python 2.7.5 and had to hack cygwinccompiler.py again. I'm using mingw with gcc 4.6.2 on Windows 7.

[61337411-43fc-4a9c-b8d5-4060aede66d0]

Oscar: I agree with your analysis, but it is incomplete. There is a group

C: Users who have only cygwin gcc 4.x installed

For those, the current setup will produce an error message, essentially telling them that the need to fix something (specifically: edit distutils, install mingw). With the proposed change, --compiler=mingw32 will produce a binary, but the binary will incorrectly depend on cygwin. They may not notice on their local system (since cygwin.dll is available), but only on customer systems.

That said: which of Roumen's patches (if any) would you recommend for inclusion?

[a35477f8-71c1-46d8-a00e-b2f89ce712bf]

On 21 May 2013 17:21, Martin v. Löwis \report@bugs.python.org\ wrote:

C: Users who have only cygwin gcc 4.x installed

For those, the current setup will produce an error message, essentially telling them that the need to fix something (specifically: edit distutils, install mingw). With the proposed change, --compiler=mingw32 will produce a binary, but the binary will incorrectly depend on cygwin. They may not notice on their local system (since cygwin.dll is available), but only on customer systems.

Well there cannot be anyone in group C who currently has a functioning setup. But I agree that it's better to have a good error message. It may be possible to check in some way that the gcc used is from cygwin and add an error message specifically for this case. I'll have a look at this when I'm next on Windows.

More generally I think that compiling non-cygwin extensions with cygwin gcc should be altogether deprecated (for Python 3.4 at least). It should be discouraged in the docs and unsupported in the future. It can only work with -mno-cygwin which in turn only works with gcc 3.x, has never documented as being a stable gcc feature, is now abandoned and is referred to disparagingly on both the mingw and cygwin mailing lists:

To quote Dave Korn from cygwin http://cygwin.com/ml/cygwin/2009-03/msg00802.html ''' gcc-3 -mno-cygwin still works just as well (or badly!) as it ever has done, and will be retained for ever.

gcc-4 series releases will not support it at all. As the whole thing is (still) experimental and explicitly warned to be unstable I don't see the need to go for a deprecation period. '''

Or Earnie Boyd from mingw-users http://permalink.gmane.org/gmane.comp.gnu.mingw.user/42111 ''' On Mon, May 20, 2013 at 9:13 AM, Paul Moore wrote:

So building an extension using --compiler=mingw in Python could pick up a cygwin gcc if that was on PATH, and this will work as long as -mno-cygwin is passed on the command line. But it won't work (it will build a DLL with a dependency on the cygwin DLL) if -mno-cygwin is omitted. I'd argue that people should just install and use mingw rather than cygwin, but that may not be what everyone does in practice.

No!!! The -mno-cygwin abomination is dead. If you want to build a native Python using Cygwin you would do it the cross compiler way and state the --host you're configuring for. Python's distutil needs to remove the -mno-cygwin option. '''

However no-cygwin mode is currently a documented feature: http://docs.python.org/3.4/install/index.html#gnu-c-cygwin-mingw

So it can't simply be deprecated in already released Pythons but I do want to fix the mingw bug there if possible. The suggestion to make -mno-cygwin conditional on gcc major version may lead to some users who attempt to use a setup that did not previously work not seeing the appropriate error message. However it does, I believe, come with the following two guarantees:

1) Mingw setups that are used, wanted and currently broken will be fixed. 2) No currently functional setups will be broken.

That may be the best that is possible given the tight constraints on changes to distutils.

That said: which of Roumen's patches (if any) would you recommend for inclusion?

None. I may have misread them but my impression is that they are not particularly intended to be used as individual patches. I can't see one that just makes the relevant changes and collectively they make up a more pervasive change than I was proposing.

The patch that I was proposing for 3.4 would simply remove -mno-cygwin on these 5 lines: http://hg.python.org/cpython/file/7fce9186accb/Lib/distutils/cygwinccompiler.py#l322

For 2.7, 3.2 and 3.3 I would do the same but conditional on self.gcc_version.

I think Roumen has identified many different issues but I would try and keep it focussed on the one -mno-cygwin issue.

Oscar

[61337411-43fc-4a9c-b8d5-4060aede66d0]

Am 21.05.13 23:14, schrieb Oscar Benjamin:

More generally I think that compiling non-cygwin extensions with cygwin gcc should be altogether deprecated (for Python 3.4 at least). It should be discouraged in the docs and unsupported in the future.

I agree with that, although I find it sad that the Cygwin project apparently abandoned support for building Mingw binaries.

It can only work with -mno-cygwin

This is factually incorrect. It also works with the i686-pc-mingw32-gcc executable, which (IIUC) is still available for Cygwin.

[a35477f8-71c1-46d8-a00e-b2f89ce712bf]

On 22 May 2013 12:43, Martin v. Löwis \report@bugs.python.org\ wrote:

Am 21.05.13 23:14, schrieb Oscar Benjamin: > More generally I think that compiling non-cygwin extensions with > cygwin gcc should be altogether deprecated (for Python 3.4 at least). > It should be discouraged in the docs and unsupported in the future.

I agree with that,

Excellent.

although I find it sad that the Cygwin project apparently abandoned support for building Mingw binaries.

I don't understand their reasoning but given the scorn poured on to -mno-cygwin from at least some people I trust that they had some good reason :)

Also they have replaced it with something that they consider more appropriate (the cross-compilers).

> It can only work with -mno-cygwin

This is factually incorrect. It also works with the i686-pc-mingw32-gcc executable, which (IIUC) is still available for Cygwin.

I should have been slightly clearer. It can only currently work in distutils with -mno-cygwin. The executable you refer to is part of cygwin gcc's cross-compiler toolchain. This is their recommended replacement for -mno-cygwin (if not mingw) but is AFAICT unsupported by distutils.

I think there's a case for saying that distutils should support these but it should only be done with a new UnixCCompiler subclass and a new --compiler entry point. It should also perhaps provide a way to specify the --host since I think that facility is part of the purpose of the new toolchain.

In any case cygwin cross-compiler support should not be conflated in the codebase with distutils' mingw support and if it is to be added that should be discussed in a separate issue. I personally don't think I would use it and would not push for the support to be added.

Going back to the group C users: I think that it should be possible to create an iscygwingcc() function that would parse the output of 'gcc --version'. Then Mingw32CCompiler.\_init__ could do:

    if is_cygwingcc() and self.gcc_version >= '4':
        raise RuntimeError('No cygwin mode only works with gcc-3. Use
gcc-3 or mingw')

The is_cygwingcc() function can be conservative since false positives or more of a problem than false negatives. I think this should address your concern.

However on further reflection I'm a little reluctant to force an error if I can't *prove* that the setup is broken. I'm no stranger to monkey-patching distutils and it's possible that someone has already monkey-patched it to make some bizarre setup just about work. I would be a little peeved if my setup broke in a bugfix release simply because someone else who didn't understand it decided that it wasn't viable. (The same monkey-patching concerns apply to the other changes but I think that fixing the non-monkey-patched setup for mingw trumps in that case.) So perhaps the best place to deal with the gcc-4/no-cygwin issue is in the distutils docs.

My updated proposal is (I'll write patches if this is acceptable):

Python 3.4: Remove '-mno-cygwin'. This breaks the no-cygwin mode and fixes the mingw mode. The distutils docs are updated with something like: ''' Note: Previous Python versions supported another 'no-cygwin' mode that could use cygwin gcc to build extensions without a dependency on cygwin.dll. This is no longer supported.

New in Python 3.4: No-cygwin mode is no longer supported. '''

Python 2.7, 3.2 and 3.3: Only use '-mno-cygwin' if self.gcc_version \< '4'. This should not break any currently functioning setups (barring serious monkey-patching). The distutils docs are updated with something like: ''' Note: The no-cygwin mode only works with cygwin's gcc-3. For gcc-4 it may produce .pyd files with dependencies on cygwin.dll that are not fully redistributable. The use of no-cygwin mode is deprecated by cygwin and support for it is removed in Python 3.4. '''

If you would rather have the is_cygwingcc() check I'm happy to put that in also if it gets this issue moving but I'm personally cautious about it.

Thanks, Oscar

[a35477f8-71c1-46d8-a00e-b2f89ce712bf]

On 22 May 2013 13:40, Oscar Benjamin \report@bugs.python.org\ wrote:

However on further reflection I'm a little reluctant to force an error if I can't *prove* that the setup is broken.

After a little more reflection I realise that we could just do:

    if self.gcc_version < '4' or is_cygwingcc():
        # use -mno-cygwin

This way the cygwin/gcc-4 error is still emitted only if gcc emits it. If the is_cygwingcc() function is conservative then there could be cases where it mistakenly does not use -mno-cygwin but that would have to be a broken cygwin/gcc-4 setup anyway.

[a35477f8-71c1-46d8-a00e-b2f89ce712bf]

I have written a function that can be used to determine if the gcc that distutils will use is from Cygwin or MinGW:

def is_cygwingcc():
    '''Try to determine if the gcc that would be used is from cygwin.'''
    out = Popen(['gcc', '-dumpmachine'], shell=True, stdout=PIPE).stdout
    try:
        out_string = out.read()
    finally:
        out.close()
    # out_string is the target triplet cpu-vendor-os
    # Cygwin's gcc sets the os to 'cygwin'
    return out_string.strip().endswith('cygwin')

The idea is that 'gcc -dumpmachine' emits a string that always ends in 'cygwin' for the Cygwin gcc (please let me know if I'm wrong about that). Earnie Boyd at mingw-users described this method for distinguishing MinGW and Cygwin gcc as not being a bad idea: http://permalink.gmane.org/gmane.comp.gnu.mingw.user/42137

With this the Mingw32CCompiler.__init__ method can be modified to do:

if self.gcc_version < '4' or is_cygwingcc():
    no_cygwin = ' -mno-cygwin'
else:
    no_cygwin = ''

self.set_executables(compiler='gcc%s -O -Wall' % no_cygwin,
                     compiler_so='gcc%s -mdll -O -Wall' % no_cygwin,
                     compiler_cxx='g++%s -O -Wall' % no_cygwin,
                     linker_exe='gcc%s' % no_cygwin,
                     linker_so='%s%s %s %s'
                            % (self.linker_dll, no_cygwin,
                               shared_option, entry_point))

This will fix the problem for MinGW, should not break existing no-cygwin/gcc 3.x setups and preserves the error message currently seen for no-cygwin with gcc 4.x. In other words it should satisfy users in all three groups A, B and C referred to above. In particular the is_cygwingcc() function hopefully addresses Martin's concern for users in group C.

Is this approach acceptable?

Thanks, Oscar