Document the MSYS2 stack

jaimergp commented 1 year ago

Where should the content be added?

Probably in the knowledge base

What should be added?

The MSYS2 stack is built in non-conventional ways and its not well documented. E.g. TIL how m2-* vs m2w64-* packages must be used in the recipe, but I don't know why.

There are some issues with some background, but the picture is incomplete and it should be preferably clarified and discussed in the documentation.

Additional information

zklaus commented 1 year ago

Thanks for the initiative, @jaimergp! One question I would like to see addressed is that of data model: Unix generally uses LP64, whereas Windows uses LLP64, the main difference being that long int is 64bit, and 32bit, respectively. This gives me some headaches with the Windows builds of esmf, netcdf, etc. Cygwin on Windows uses LP64, precisely to simplify porting things from the Unix world. This requires greater care when calling into the Windows API, see here.

If we think the main reason for the win-64 subdir is the porting of Unix toolchains, it may be better to just go with the Cygwin approach?

pkgw commented 9 months ago

It looks like the update to Python 3.12 is going to interact with this issue. In https://github.com/conda-forge/xcb-proto-feedstock/pull/24 , the Windows / Python 3.12 build is failing because the most recent available MSYS2 automake package, m2-automake1.15, provides a very old version of a support script (py-compile) that relies on the imp module, which was removed in 3.12. This problem can definitely be worked around at the feedstock level, but it would be nice to not have to do that.

isuruf commented 9 months ago

If we think the main reason for the win-64 subdir is the porting of Unix toolchains, it may be better to just go with the Cygwin approach?

@zklaus, m2- packages uses the MSYS2 runtime which is a fork of the Cygwin runtime whereas m2w64- packages link to msvcrt (or ucrt). If you compile netcdf etc with Cygwin, then you would have to compile python with Cygwin too, then you won't be able to use any of the pip wheels because it will be a different Python ABI.

isuruf commented 9 months ago

@pkgw, we will update the packages in the near future.

pkgw commented 9 months ago

@isuruf That is great to hear! Do you have a sense of how near "near" is? Depending on the timeline it might make sense to deploy some workarounds to help advance the Python 3.12 migration.

isuruf commented 9 months ago

A month or two I guess

FaustinCarter commented 3 months ago

A month or two I guess

Any update on this? Would be awesome to have an update to the old 2016 packages.

isuruf commented 3 months ago

The m2 packages have been updated, but not the m2w64 packages.

ehfd commented 2 months ago

As of today, how is it possible to use MSYS2 to build newer packages which have various issues impacting its usage with Visual Studio (such as https://mesonbuild.com/FAQ.html#my-project-works-fine-on-linux-and-mingw-but-fails-to-link-with-msvc-due-to-a-missing-lib-file-fatal-error-lnk1181-why)?

For instance, I need to use m2-automake1.16 which is required for autoconf > 2.70, and there is the following error with the m2w64_c toolchain. Would like to know if it's possible to utilize this now or not.

Could not solve for environment specs
The following packages are incompatible
\u251c\u2500 m2-automake1.16 is installable and it requires
\u2502  \u2514\u2500 m2-conda-epoch 20230914 *_x86_64, which requires
\u2502     \u2514\u2500 msys2-conda-epoch <0.0a0 , which can be installed;
\u2514\u2500 m2w64-toolchain_win-64 is not installable because it requires
   \u2514\u2500 m2w64-toolchain, which requires
      \u2514\u2500 msys2-conda-epoch 20160418 , which conflicts with any installable versions previously reported.

Edit: For an answer to this, refer to https://github.com/conda-forge/xorg-makedepend-feedstock/pull/9#issuecomment-1930098269 (copied below)

I'll duplicate one of my commit messages here in the PR discussion. For posterity, here are the changes to update the package to work with the updated MSYS2 stack:

Remove very old MSYS2_ARG_CONV_EXCL workaround in bld.bat
Remove very old IFS workaround in build.sh
Update Automake to 1.16 (in both meta.yaml and build.sh)
Update aclocal include path given to autoreconf
Remove posix dep; addm2-{grep,m4} (on Windows only, of course)
Change make dep to m2-make (Windows only)
Change m2w64-pkg-config dep to m2-pkg-config (Windows only)
Switch to MSVC compilers: change compiler('m2w64_c') to compiler('c'), etc

Other X.org packages that are migrating should also use the opportunity to start depending on the unified xorg-xorgproto protocol package, which supersedes the individual xorg-*proto packages that we have used historically.

In this particular package, we also need to patch the source to work on a native MSVC build. I believe that most other X.org packages should be able to build with MSVC without needing (additional) patching.

ehfd commented 2 months ago

Moreover, is it possible to use code containing fork() with Windows MSYS2 builds without major patches? If so, how?

pkgw commented 2 months ago

@ehfd I think your kinds of questions are a bit too fine-grained for this issue, which is more targeted at the higher-level MSYS2 situation. I've replied in the xorg-libxmu PR thread with some pointers.

conda-forge / conda-forge.github.io