Using the MSYS2 tools to build conda packages

[jakirkham]

@mingwandroid has done an amazing job of packaging MSYS2 so that is works within conda. See his channel here. Also, see his comment about the work he has done and what still remains for more details.

Given how this greatly reduces the gap between Windows and the NIXes, this makes Windows packaging very accessible to NIX devs. This should not be undervalued as it has the potential to really increase our bandwidth on Windows by basically the size of our community. Packages that couldn't be built due to CMake files missing or other problems also dissolve away. This has the potential to substantially reduce the effort required to get a package working on all platforms, which makes us more likely to do exactly that.

Also, it has the potential to free this quite painful constraint of compiler and CPython version. That has raised such issues as making custom patched versions of CPython ( https://github.com/conda-forge/conda-forge.github.io/issues/64 ) just to have C++11 features when using Python 2.7. Further this can free us from thinking about VC features entirely, which have had various issues.

It is honestly making me seriously considering switching to MSYS2 with everything.

Would be very curious to hear perspectives from a wide range of people on this. Please feel free to share your thoughts and also bring other people over that I have missed. I have put this on our docket for next weeks meeting.

cc @ocefpaf @pelson @ChrisBarker-NOAA @patricksnape @gillins @msarahan @mcg1969 @isuruf @ngoldbaum @ukoethe @jasongrout @SylvainCorlay

[jankatins]

Re having MSYS2 as a package and use it in the builds: +10000

Re patched CPython:

as making custom patched versions of CPython ( #64 ) just to have C++11 features when using Python 2.7

This shouldn't break using pip with wheels. And also not break any of the "how to compile python with native libs on windows"-guides, which users turn to when they have to build stuff for themself (e.g. the patched CPython should still be able to build and afterwards work with a pandas version which is compile the "normal way").

My guess (which is based on compiling a few python native packages and fix a few errors on the way, but no deep knowledge on how compiler and ABI work) is that it will take more time to make it work (e.g. patch out WIN32 #ifdefs which then assume a certain compiler or things like mpl which when run on windows sets up the VS compilers) than what you gain by being able to use a compiler unconstrained by the python version.

[patricksnape]

Two things, in brief as I'm just heading out:

1. @mingwandroid has packaged many msys packages on the msys2 channel. This is great, but it involves a load of changes to conda-build etc to natively support it. This brings in the larger question of the scope of this project. It could really start to become enormous if we have builds for perl and all perl packages and msys and all msys packages and R and Python etc... Maybe we might think about how we interact with other channels? Perhaps, since this is a build requirement, we could almost run the channels in parallel?
2. I'm not sure that moving to msys actually 'replaces' VS. This is only the case if we don't link to VS, which would mean we lose compatibility with wheels/the rest of the Python ecosystem. If we do link against VS then we still have to build 3 versions on Windows and sort of the features headache we currently have.

[jakirkham]

Per point 2, @patricksnape, see this comment from @mingwandroid.

[jakirkham]

Per point 1, @patricksnape, we could investigate using this pacondaman and some sort of automated maintenance strategy like ( https://github.com/conda-forge/conda-forge.github.io/issues/51 ).

Though I don't know if we will want to do that or if it will be that bad. Right now we build things like autoconf, libtool, pkg-config, etc. It shouldn't be a big deal to add MSYS style builds to these repos. It looks like the recipes needed already exist.

[mingwandroid]

I expect to add all of the build tools as a matter of priority. So far only those needed for R have been added.

[jakirkham]

If you would like to try PRs to those feedstocks, that would be great. I think it will be @ocefpaf and I working with you for the most part as we are maintainers on these sorts of things. Definitely agree those are high priority. That was our mentality here (at conda-forge) too when addressing Mac and Linux.

[jakirkham]

2.7 and 3.4 do need special handling after that yes, there is an idea floating around for this.

Could you please clarify on exactly what idea you are thinking about here? Even if it is hazy, it would be useful to get some insight.

I figure fixing 3.5 (and with it, linking GCC and MSVC) is more important though.

That is certainly the easiest place to start. If you haven't had a chance, I'd really recommend taking a look at our proposed recipe for CPython 3.5 ( https://github.com/conda-forge/staged-recipes/pull/370 ). This is something that @jjhelmus has been working on for a bit and he just recently got Windows support with it. Though I believe it still needs a couple of things. @pelson has worked pretty substantially on this effort too most of which occurred a previous PR. It would be interesting to see what you think would need to happen to this recipe to move it in that direction. This is actually a great time to explore this as we are still working on that recipe.

Some things I missed from earlier: m2-mingw-w64-toolchain supports gfortran (and ada and objective-c). I didn't actually write any of the packages . They were created automatically from the MSYS2 packages by pacondaman. I did add some patches to one of them (m2-msys2-filesystem) and had to rewrite the post-link scripts for two? of them (this can be automated though).

Sure. I'm betting there will be a bit of patching on the first few passes, which is fine. Normally we have been pretty eager to see patches move upstream. Though given you are a core developer of MSYS2 and there may be a fair bit of patching required, we might need to be more lax just so that we can keep this effort moving at a good pace. Also we may need to come up with a better system for tracking these patches.

[mingwandroid]

The idea was to do a solution for DLLs and Python specifically where it looks at the C-Runtime already loaded by the process and stubs through to the functions there.

The patching I refer to is specifically to patch the differences between MSYS2 upstream and Conda. You can see what I mean if you study https://github.com/mingwandroid/conda-recipes/tree/MSYS2-pkg-conversion/msys2/m2-msys2-filesystem

After unpacking the MSYS2 package, the patches are applied which allows us to make such modifications as we need to. It means we don't need MSYS2 upstream to put switches in for Conda (nor should they).

[ukoethe]

@patricksnape

I'm not sure that moving to msys actually 'replaces' VS.

I seriously consider an MSYS-based Python ecosystem as a possible future alternative to VC-based builds. My hope (or, rather, expectation) is that recipes for MSYS will reuse much of the existing Linux scripts. In fact, there should be only one build.sh script that works for Linux, Mac and MSYS, with just occasional ifs to account for the remaining differences. For this to work, conda should treat the MSYS-based ecosystem as an entirely new platform.

[jakirkham]

For this to work, conda should treat the MSYS-based ecosystem as an entirely new platform.

Yeah, I imagine that this transition is not happening over night, but I am seriously hoping we can phase out any direct dependence on VS in the long run. Things like CPython will still need this to behave well with wheels so we can't entirely eliminate it, but we should be able to disentangle compiler features from Python version which has been a huge strain on the community.

In fact, there should be only one build.sh script that works for Linux, Mac and MSYS, with just occasional ifs to account for the remaining differences.

Yes, this is exactly my hope. Ideally this will lower the threshold for getting productive packagers that are comfortable on *NIX systems trying things on Windows too. It should also significantly lower the maintenance burden with Windows packages.

[ChrisBarker-NOAA]

Sorry to get a bit confused here, but, IIUC:

MSYS is a port of the *nix command line tools (a shell and common utilities) to run semi-nativly on Windows.

that gives us the regular make and all that, which is nice.

But it also essentially requires using MInGW for the compiler, yes? Probably not strictly, but we wouldn't gain much without that.

So this is really a question about using the MInGW compiler, yes? So I have these questions:

Back in the day, you could use MinGW to build extensions compatible with the ("standard", i.e. VS compiled) 32 bit Python (2.6 anyway) on Windows, it required a bit of hacking, and resulted in slower binaries than VS, but worked.

However, my understanding is that MInGW never got "properly" ported and working for 64 bit -- so not sure the status of all that for 64 bit Windows. I do know for sure that I dropped MinGW years ago -- using VS was so much easier (provided I could find the Windows builds of the libs I needed)

However, as of recently, the numpy folks have sponsored work on MinGW (or SOMETHING) that has gotten stable? now for 64 bit Windows, OpenBLAS, Fortran and all -- so maybe it's ready to rock. Is this what you're talking about using?

And does this use the MS runtime that Python is already built against? -- I'm pretty sure you NEED that to get full compatibility.

As for:

Rr patched CPython:

as making custom patched versions of CPython ( #64 https://github.com/conda-forge/conda-forge.github.io/issues/64 ) just to have C++11 features when using Python 2.7

This shouldn't break using pip with wheels.

not sure what the "this" is -- but providing a new build of CPython with a different runtime would break binary wheels unless it was tagged in a different way (as a different platform) so that pip would know not to try to use a binary wheel built for the "standard" Windows Python.

But I take it that this is about using MSYS/ MingGW for extensions and libs -- NOT python itself, yes?

-CHB

On Tue, Apr 19, 2016 at 9:52 AM, jakirkham notifications@github.com wrote:

@mingwandroid https://github.com/mingwandroid has done an amazing job of packaging MSYS2 so that is works within conda. See his channel https://anaconda.org/msys2 here. Also, see his comment https://github.com/conda-forge/staged-recipes/pull/364#issuecomment-211830313 about the work he has done and what still remains for more details.

Given how this greatly reduces the gap between Windows and the NIXes, this makes Windows packaging very accessible to NIX devs. This should not be undervalued as it has the potential to really increase our bandwidth on Windows by basically the size of our community. Packages that couldn't be built due to CMake files missing or other problems also dissolve away. This has the potential to substantially reduce the effort required to get a package working on all platforms, which makes us more likely to do exactly that.

Also, it has the potential to free this quite painful constraint of compiler and CPython version. That has raised such issues as making custom patched versions of CPython ( #64 https://github.com/conda-forge/conda-forge.github.io/issues/64 ) just to have C++11 features when using Python 2.7. Further this can free us from thinking about VC features entirely, which have had various issues.

It is honestly making me seriously considering switching to MSYS2 with everything.

Would be very curious to hear perspectives from a wide range of people on this. Please feel free to share your thoughts and also bring other people over that I have missed. I have put this on our docket for next weeks meeting.

cc @ocefpaf https://github.com/ocefpaf @pelson https://github.com/pelson @ChrisBarker-NOAA https://github.com/ChrisBarker-NOAA @patricksnape https://github.com/patricksnape @gillins https://github.com/gillins @msarahan https://github.com/msarahan @mcg1969 https://github.com/mcg1969 @isuruf https://github.com/isuruf @ngoldbaum https://github.com/ngoldbaum @ukoethe https://github.com/ukoethe @jasongrout https://github.com/jasongrout @SylvainCorlay https://github.com/SylvainCorlay

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/conda-forge/conda-forge.github.io/issues/112

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

[ChrisBarker-NOAA]

On Tue, Apr 19, 2016 at 11:38 AM, Ullrich Koethe notifications@github.com wrote:

I seriously consider an MSYS-based Python ecosystem as a possible future alternative to VC-based builds.

That's a whole other ball of wax -- take a look at Pyton-dev from a couple months back on this thread:

https://mail.python.org/pipermail/python-dev/2010-August/102909.html

It goes on a for a bit, but the upshot is that making a MInGW build is non-trivial and not going to get support form the core devs -- at least not right away.

If someone does want to do that -- build a conda-distro based on fully MInGW builds -- then that is very much a separate project from conda-forge, and it would have to get pretty mature and tested before we would somehow merge it into conda-forge (Not that it's my decision)

For this to work, conda should treat the MSYS-based ecosystem as an entirely new platform.

Exactly.

-CHB

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

[mingwandroid]

However, my understanding is that MInGW never got "properly" ported and working for 64 bit

mingw.org and mingw-w64 are two different endeavors. There was a disagreement about whether you could use MSDN for reference, roughly speaking.

mingw-w64 has had 64-bit from the start. MSYS2 is a software distro for Windows that's build on mingw-w64 and a fork of Cygwin (which is built on top of mingw-w64 these days and also provides a 64-bit variant).

In Conda, our intention is to use it to provide a consistent POSIX build environment that interoperates with native tools, a native GNU toolchain and some GNU libraries without duplicating effort or further splitting the already small GNU-based Open Source on Windows developer community.

[jankatins]

For this to work, conda should treat the MSYS-based ecosystem as an entirely new platform.

Can someone explain the gains from such a move?

This is a deviation from upstream, so this means that conda-forge has the complete burden to maintain forks to basically all tools and libs which assume "win = VSC" , starting with python and conda-build itself (setup.py must find the right compiler). Basically every if sys.platform == 'win32' in any setup.py code or any #if !defined(_WIN32) is actually a if the compiler is VSC, which would need to be patched (or at least confirmed that this is still right). IMO there is no incentive for upstreams to do such porting, because only conda-fore would have such a port. And there would still be a "normal" windows port, because otherwise conda-forge on windows would become irrelevant (unless this msys2 port would be compatible to the VS builds).

examples for porting work:

• Matplotlib: https://github.com/matplotlib/matplotlib/blob/master/setupext.py#L201
• pandas: https://github.com/pydata/pandas/blob/f71537ab2561ab5727008095e9685966619fa7b9/pandas/src/msgpack/sysdep.h#L39, https://github.com/pydata/pandas/blob/f71537ab2561ab5727008095e9685966619fa7b9/pandas/src/parser/io.h#L32

So basically you would have all the efforts to support VC windows in addition to even more effort to support a msys2 port. So what are the gains?

[ukoethe]

@ChrisBarker-NOAA

However, my understanding is that MInGW never got "properly" ported and working for 64 bit -- so not sure the status of all that for 64 bit Windows.

I'm currently using MinGW as part of my VS builds in two situations:

• for C packages that don't have a VS build system (e.g. openblas)
• for Fortran compilation (e.g. parts of scipy)

I consider MinGW (64-bit version) a mature compiler that doesn't cause any headaches -- it feels like its Linux counterpart. Linking of MinGW C/Fortran binaries into VS modules works mostly out of the box with two minor tweaks:

• one must create link libraries from MinGW's runtime libs using gendef and lib
• one must provide a few missing C99 functions (such as asinh or csqrt) in a little helper lib (note: this problem may have been solved in the meantime, I haven't checked recently)

If MinGW worked for the entire Python ecosystem, one could mostly reuse the effort that was already spent on creating Linux recipes. People who are primarily working on Linux (i.e. many in the scientific computing community) might prefer (or rather: will love) this approach to porting their work to Windows.

Of course, whether this gain is worth the effort is a matter of judgement. At the very least, some experiments have to be done to determine exactly how big the effort would be. This certainly isn't something I want to pursue now -- I just wanted to point out the possibility.

[jakirkham]

Interesting that you mention OpenBLAS and needing a Fortran compiler. That's part of what sparked this discussion ( see PR https://github.com/conda-forge/staged-recipes/pull/364 ).

A bit of an aside: It is worth noting that the BLAS part of OpenBLAS can be built with CMake (started near the end of 2014/beginning of 2015). However, that doesn't necessarily help people that want the LAPACK part too, which requires a Fortran compiler. Also, it means the loss of the assembly optimizations, which really puts the value in question IMHO. Though for using something easily on Windows, I suppose it still has value.

[ChrisBarker-NOAA]

On Tue, Apr 19, 2016 at 1:08 PM, jakirkham notifications@github.com wrote:

A bit of an aside: It is worth noting that the BLAS part of OpenBLAS can be built with CMake (started near the end of 2014/beginning of 2015). However, that doesn't necessarily help people that want the LAPACK part too, which requires a Fortran compiler. Also, it means the loss of the assembly optimizations, which really puts the value in question IMHO. Though for using something easily on Windows, I suppose it still has value.

the numpy/scipy folks have been working hard on Mingw and OpenBLAS, and .... I've lost track of where they are at, but we should be sure to leverage what they've done.

-CHB

[ChrisBarker-NOAA]

On Tue, Apr 19, 2016 at 12:33 PM, Ullrich Koethe notifications@github.com wrote:

@ChrisBarker-NOAA https://github.com/ChrisBarker-NOAA

However, my understanding is that MInGW never got "properly" ported and working for 64 bit -- so not sure the status of all that for 64 bit Windows.

I'm currently using MinGW as part of my VS builds in two situations:

• for C packages that don't have a VS build system (e.g. openblas)
• for Fortran compilation (e.g. parts of scipy)

I consider MinGW (64-bit version) a mature compiler that doesn't cause any headaches

I guess it took a while, but if it's there now, great.

-- it feels like its Linux counterpart. Linking of MinGW C/Fortran binaries into VS modules works mostly out of the box with two minor tweaks:

• one must create link libraries from MinGW's runtime libs using gendef and lib
• one must provide a few missing C99 functions (such as asinh or csqrt) in a little helper lib (note: this problem may have been solved in the meantime, I haven't checked recently)

So these are then fully compatible with the VS2009 (is that the one?) lib that Windows Python 2.7 is built with? If so, then great.

And does it all work with py3.5 and VS2015?

If MinGW worked for the entire Python ecosystem, one could mostly reuse the effort that was already spent on creating Linux recipes.

yup -- that would be nice.

In practice, if it's easy to build stuff on Windows at the command line, then it's not hard to build a recipe -- but there are all too many libs that aren't in that state...

-CHB

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

[gillins]

Sorry I've been silent on this. +1 to investigating using MSYS2 and seeing how hard it is to build everything using it before committing to using it. I'm happy to help with this.

[jjhelmus]

the numpy/scipy folks have been working hard on Mingw and OpenBLAS, and .... I've lost track of where they are at

The The Mingwpy project page list some information on this. Still seems to be a work in progress. Worth keeping an eye on though.

[jakirkham]

I was about to ask where to follow this progress on OpenBLAS, @ChrisBarker-NOAA, but it looks like @jjhelmus has the answer already. Or are there more places to look?

[jakirkham]

As you referred to the more general problem of packaging and working with MinGW-64 compilers, I am cc'ing you here @mikofski. Also, I am cc'ing the NumFocus grant authors ( @carlkl @rgommers @matthew-brett @njsmith @ogrisel ) here, as well.

Here we are discussing how best to explore the interaction between conda-forge and MSYS2. We are interested in the ability of working with Windows in nearly the same way we do with any *NIX platform (many of whose builds tools are packaged already). While the idea has been floating around, our attempt of building OpenBLAS at conda-forge has raised this more general point of how we work with MinGW-64 compilers and MSYS2. As there now are quite a few packages available to start exploring this problem and some tools to aid in their automated addition, we are very eager to see where and how we can put them to use. Also, related is how this system is maintained. We are planning to have a meeting where we discuss this and other issue at conda-forge. Anyone who is interested is welcome to come.

[patricksnape]

I think there are sort of two separate issues:

1. We need an msys2/mingw-w64 toolchain if we want to support certain builds such as OpenBLAS and ffmpeg. We almost certainly need this toolchain to correctly link to the appropriate VS versions for each Python version. This is a practical issue that I believe @mingwandroid is very close to having solved (if not already solved).
2. We may want to swap all builds to this toolchain as it removes the burden of writing Windows batch files or having to deal with quirks in the older VS compilers. This, IMHO, is a much larger project and perhaps out of the scope of conda-forge. Which is not to say I wouldn't be interested in helping see how far we could push it, but I feel like it would really require cooperation with the Python Core devs.

The Intel Fortran Compiler is not free or available to us, so if we want to compile Fortran we need the msys2 toolchain. I think we should focus on how we enable the use of that toolchain for projects that already require it, like ffmpeg and OpenBLAS - rather than patching all projects to build with msys2.

[njsmith]

We almost certainly need this toolchain to correctly link to the appropriate VS versions for each Python version. This is a practical issue that I believe @mingwandroid is very close to having solved (if not already solved).

Last I heard, @mingwandroid wasn't actively working on this. (Correct me if I'm wrong?) The problem is on its way to being solved for python 2.6-3.4 (that's Carl Kleffner's mingwpy work). The problem for 3.5 is that it requires deeper surgery on the guts of mingw-w64, and we don't yet have a real plan for how to make that happen. The most likely prospect atm is indeed getting @mingwandroid to implement this, since he understands the issues and is familiar with mingw-w64's guts; the blocker to this is that higher-ups at Continuum don't consider it a priority, so the best we've managed to get out of them is that they might let him work on it for a few hours a week starting a few months from now if we figure out how to get someone else to pay for it. If you think this should be higher priority then I guess you should tell them that :-).

We may want to swap all builds to this toolchain as it removes the burden of writing Windows batch files or having to deal with quirks in the older VS compilers.

An intermediate step would be to use msys2 for scripting but continue to call msvc. This would at least let you get rid of the .bat files

[patricksnape]

I thought that only issue with Python 3.5 was if we want to link in a 'clever' way to the new versionless MSVCRT? If we just want to do the old way and link directly to msvcrt14 that should be possible, no? Seems like a reasonable stop gap - since we have that issue with the other VS versions anyway, so for the time being I don't see the problem with continuing it into Python 3.5 until mingw is sorted.

[mingwandroid]

@njsmith You are correct, I'm not actively working on it. MSYS2 is coming to Conda to address some of the other issues of building software on Windows (POSIX shell, Autotools, GNU toolchain with gfortran, git, CMake, consistency over all those things - we'll end up with more consistency here than the Unixes ironically since there's only one MSYS2).

@patricksnape UCRT is quite a lot different to the other CRTs and it'll require a reasonable amount of effort above what was required for e.g. to upgrade to msvc2012. @njsmith and @carlkl are far more knowledgeable about the ins and outs of it since my involvement so far has mostly been trying to figure out when I might get time to look into it properly. Linking directly to the ucrt14 is exactly what you shouldn't do AFAIUI as your executables will break on future updates to the operating system.

[njsmith]

Right, starting with msvc 2015 there is no msvcrt at all :-). They changed the names and semantics of a bunch of symbols, they removed a few (or rather: moved them to a private helper library, vcruntime140) that we will want to replace with our own versions, and in general all symbols got rearranged into a whole collection of new libraries. AFAIK there's nothing deeply difficult here, but it's complicated, especially since we would like to do the work upstream in mingw-w64 rather than make a fork that we will have to maintain forever. (mingwpy is effectively a fork, but that work is much less intrusive so tracking upstream is relatively easy, and for several reasons upstream is less interested in helping with mingwpy and more interested in helping with the msvc 2015 stuff, so the calculation is different.)

[njsmith]

OTOH if someone wanted to try just wading in and hacking mingw-w64 to work with the UCRT then we might at least learn something. Maybe it will turn out to not be that hard :-). I'm happy to share what I know and give hints (and I guess @mingwandroid and others are as well).

[njsmith]

BTW are these meeting details available anywhere? (Like, when is it?)

[jakirkham]

Yes, those are important details. :wink:

We have been putting our agendas (with dates) on them about when we meet and what we will discuss in hackpad. Time is 1400 UTC. We have been doing them on Google Chat. Normally @pelson sends out a link before so that we can join the same chat room.

[jakirkham]

Also, anyone is welcome to add items to the agenda.

[ChrisBarker-NOAA]

The most likely prospect atm is indeed getting @mingwandroid https://github.com/mingwandroid to implement this, since he understands the issues and is familiar with mingw-w64's guts; the blocker to this is that > higher-ups at Continuum don't consider it a priority,

So how is continuum doing Fortran on Win64 for python 3.5? The intel compiler? We really need SOME way to build Fortran extensions.

and while I"m at it what's the best practice for Fortran with Python2.7 an Anaconda on Win64...

-CHB

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

[msarahan]

I'm not completely certain, but I'm pretty sure we use the Intel compiler.

[jakirkham]

That would make sense. FWICT the Intel compiler is pretty nicely compatible with VC. Though I have no experience using it. Also, I don't know what would go into getting that into the VM(s).

[jakirkham]

@njsmith @mingwandroid and others interested in coming to the meeting next week the date has been corrected from Thursday to Friday. The time is still the same (1400 UTC).

[mingwandroid]

Ok thanks, I'll await the link to the Hangout then.

[pelson]

Hangout link can be found at https://conda-forge.hackpad.com

[jakirkham]

Thanks @pelson.

[tkelman]

I'm a little surprised no one has mentioned this here, but it is possible to build CPython 3.5 using mingw-w64 GCC (and 2.7 as well with a somewhat different set of patches). If you install mingw-w64-python3 in MSYS2, this is what you get. However it requires 90-odd patches that upstream CPython-dev has been very resistant to even consider reviewing the first handful, much less merge and work towards getting this to work out of the box. See https://github.com/Alexpux/MINGW-packages/blob/master/mingw-w64-python3/PKGBUILD for all the gory details.

This is the extreme case of forking a whole new platform for ease of building scientific extensions that need things MSVC doesn't support like Fortran, inline assembly, building with autotools, etc. But it has the advantage that it works today, unlike getting mingw-w64 to link against the VC 2015 runtime. In Julia's case we choose to completely ignore MSVC as it's a lousy compiler that doesn't meet our needs, and it's way too foreign to expect open-source developers who mostly use Linux and Mac to work with.

[njsmith]

@tkelman: unfortunately, a Python distribution for Windows that can neither build nor use pypi wheels is unlikely to see much uptake. Also, conda-forge has no control over what compiler is used to build Python itself, unless they abandon anaconda compatibility entirely.

[tkelman]

Depends where you'd rather invest your effort. Building a custom python ecosystem of your own, or building a custom compiler toolchain of your own for (partial) compatibility with a different compiler that you don't actually want to use. You're letting the way python.org chooses to build Python binaries drive the decisions here, and it's not a foregone conclusion that it has to stay that way.

[ukoethe]

@njsmith

unfortunately, a Python distribution for Windows that can neither build nor use pypi wheels is unlikely to see much uptake

I don't think so. If you want to do serious scientific computing in Windows, pypi is just not good enough, and I'm skeptical that its design can be fixed. conda is the only package manager on Windows whose approach is sufficiently powerful to make this work (although it's still quite a way to go). If conda lives up to our expectations, I believe it will replace pypi on Windows entirely. So, let's make conda and conda-forge as good as possible and not constrain ourselves by upstream's prejudice.

[tkelman]

Pip and the wheels situation and the pynativelib proposal are all rapidly improving and I'm sure everyone appreciates the work Nathaniel and others have done to make that possible. And mingwpy will hopefully reach its goals sooner or later w.r.t. MSVC 2015 compatibility, and once it's there the mingw-w64-using community will evaluate whether it's something that will be more broadly useful for things like MSYS2, Julia, CRAN, etc where currently MSVC is just ignored.

In the meantime, it seems there are more people capable of working on rebuild-the-world packaging infrastructure than hacking on the guts of compiler runtime compatibility issues. It's at least worth experimenting with, since there are examples of other ecosystems that rebuild everything on their own and do fine without worrying about ABI compatibility with what's posted on pypi. Cygwin has peacefully been building its own set of python packages for years and it works decently for its users.

[mingwandroid]

I've not mentioned MSYS2's mingw-w64-python ports yet since they're not something that we plan to add to Conda as it would cause confusion for our users and also add significantly to my workload, ending up with another Conda Python ecosystem that bears no resemblance to the Linux and OS X ones. That time would be better spent on the effort to make the toolchains compatible.

Guido said that if we identified mingw-w64 Python as different from MSVC Python from pip and wheels perspective) then the idea of a mingw-w64 Python wouldn't be rejected outright, but there's still the problem that CPython developers have mostly displayed reactions ranging from disinterest to contempt and fear at the notion of compiling Python on Windows using anything but MSVC.

The patches will continue to be maintained, as time allows, by the MSYS2 project and interested contributors are welcome to lend a hand there..

We should discuss this at the conda-forge meeting tomorrow I think.

[carlkl]

@mingwandroid, can you make a summary of features/properties you would expect from mingwpy to be included into conda?

• static vs. canonical linking of gcc runtime code
• posix vs. win32 threads model
• exception model for 32 bit code: sjlj vs. dwarf
• long double default implementation: extended precision vs. double precision like MSVC (can be changed at compile time btw.)
• things I forgot to mention

[tkelman]

My personal preferences are:

• defaulting to dynamic linking (adding -static within any build systems that absolutely need it)
• posix threads since LLVM and other libraries are making use of C++11 threads these days (and winpthreads is a small liberally licensed dll, and the performance has improved with recent mingw-w64 runtime versions though still worth benchmarking carefully to see if it's noticeably worse than win32 thread model), or maybe building both and providing a choice like Debian's recent cross compiler packages do
• sjlj if you ever want to throw, unwind, or debug through closed-source binaries (also worth benchmarking the perf difference of this)
• could go either way, extended precision long doubles can be useful once in a while but they're tough to make portable
• I'll make a note that -mincoming-stack-boundary=2 is needed once in a while even if you stick to the conventional msvcrt.dll used by mingw-w64. I've seen segfaults without this flag on i686, usually when threading is involved. https://github.com/Alexpux/MINGW-packages/pull/720 / https://github.com/libgit2/libgit2/issues/3342 is one concrete example though it has come and gone with different libgit2 versions

[mingwandroid]

@carlk: conda isn't conda-forge. If the conda-forge guys want to use mingwpy then they will. Getting things into conda is something that Continuum try to do so much, instead we try to host our recipes in conda-forge. An exception to this was R on Windows, where we use MSYS2's mingw-w64 libraries directly. I un-package the Pacman tar.xzs and create conda tar.bz2s. The reasons were:

1. Lack of time to rebuild the libraries we need.
2. I didn't want to split the GCC-on-Windows efforts further, work flows between now.

static vs. canonical linking of gcc runtime code

I'm not sure about the "gcc runtime code" qualification. Shared linking is a bit of a requirement for me, otherwise I'm forever building software. Large scale software distributions with static linking don't err, scale. We'd need to track what libraries got statically linked to each package (which conda doesn't do yet) just so we know that when we build library A we also need to rebuild package B, C, D and E. You can't mix -static-libgcc and non -static-libgcc. With MSYS2's mingw-w64 packages we never use -static-libgcc because you can't fix issues in your toolchain (or other gcc libraries, like the CRLF bug in gfortran recently) without rebuilding everything.

posix vs. win32 threads model

This doesn't matter so much in practice. winpthreads is built on top of win32 threads so they're just native threads anyway. You can't share the sharing/exclusion primitives between the two directly, but that's not unexpected. A program can happily run with threads spawned by each though, if it linked to libraries configured each way.

exception model for 32 bit code: sjlj vs. dwarf

dw2 is faster, esp. for exception heavy C++ so we picked that. Thiago from the Qt project did benchmarks a while ago. sjlj adds a significant register-dumping overhead to every try, whether exceptions are taken or not.

long double default implementation: extended precision vs. double precision like MSVC (can be changed at compile time btw.)

Internal x87 80bit precision has never struck me as anything of interest. Like you say, those bits get thrown away finally, and more precision can only help while the calculations are being done in the x87 registers. That it's different from MSVC is expected anyway.

things I forgot to mention

I can't see much problem with changing MSYS2's i686 mingw-w64-gcc to use -mincoming-stack-boundary=2.

[ocefpaf]

@carlk: conda isn't conda-forge. If the conda-forge guys want to use mingwpy then they will.

If fact we already do :smile:

https://github.com/conda-forge/pyspharm-feedstock/blob/master/recipe/meta.yaml#L21

mingwpy is my preferred option for Fortran on Windows so far.

[carlkl]

Just 2 links for all developers expecting extra precision from long double arguments with FPU utilization:

• https://randomascii.wordpress.com/2014/10/09/intel-underestimates-error-bounds-by-1-3-quintillion
• http://notabs.org/fpuaccuracy/index.htm

Unfortunately Intel FPU trigonometric functions offers much less precision than expected if used with specific argument domains or used with argument range reduction. This affects long double precision values as well.

Today you are better of using SSE2 math code libraries or use dedicated multiprecision libraries if double precision is not sufficient.

[ChrisBarker-NOAA]

Sorry, being really lazy here: does this have implications for which gcc libc is used? I.e is the old one in CentOS 5 still using the hardware fsin()?

CHB

On May 6, 2016, at 4:29 AM, carlkl notifications@github.com wrote:

Just 2 links for all developers expecting extra precision from long double arguments with FPU utilization:

- https://randomascii.wordpress.com/2014/10/09/intel-underestimates-error-bounds-by-1-3-quintillion

• http://notabs.org/fpuaccuracy/index.htm

Unfortunately Intel FPU trigonometric functions offers much less precision than expected if used with specific argument domains or used with argument reduction. This affects long double precision values as well.

Today you are better of using SSE2 math code libraries or use dedicated multiprecision libraries if double precision is not enough.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/conda-forge/conda-forge.github.io/issues/112#issuecomment-217416840