Build with Mingw-w64 - Githubissues

ghost commented 6 years ago

Crystal is in the process of supporting Visual Studio. However Mingw-w64 should be supported as well.

sam0x17 commented 6 years ago

I'm all for this because I like the idea of being able to build windows binaries without windows, but not to the detriment / timeliness of the current windows development being done. That said, maybe this will make windows CI easier? or no?

straight-shoota commented 6 years ago

Of course, Mingw64 should be supported. It doesn't feel like a pressing issue to me, though.

I don't no much about the practical differences between MSVC and Mingw64 in regards to Crystal supporting it. But, as I alread said in https://github.com/crystal-lang/crystal/issues/5430#issuecomment-395958620, the current effort should focus on porting the stdlib to Win32 API. That needs to be done anyway and still requires a lot of work. Supporting an alternate tool chain doesn't gain anything until that is at least somewhat matured.

this will make windows CI easier?

Windows CI needs to run on windows anyway, so there is no point in cross compiling from a different platform.

faustinoaq commented 6 years ago

Windows CI needs to run on windows anyway, so there is no point in cross compiling from a different platform.

Azure CI looks very interesting. It supports Windows, Linux and Mac and is free fpr Open Source Projects: https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/ @bcardiff @RX14 :eyes:

straight-shoota commented 6 years ago

travis-ci has also announced windows support: https://blog.travis-ci.com/2018-10-11-windows-early-release

That should be fairly easy to integrate into the CI setup

mominshaikhdevs commented 1 year ago

Since Crystal is maturing support towards MSVC and Win32 APIs directly instead of mingw-w64 (https://github.com/crystal-lang/crystal/issues/26#issuecomment-999358790), Is this still relevant?

straight-shoota commented 1 year ago

Technically yes. Mingw-w64 is a different target than msvc and supporting it is still a goal I think. Although it's not actively driven at the moment due to low importance. So I guess we could also close this issue because it's not directly actionable, with the option to re-open once there's activity towards supporting this platform. I expect it shouldn't be much work, but also probably not many people care about it due to native Win32 support and the overall decline of Mingw due to WSL.

mominshaikhdevs commented 1 year ago

Exactly. The Crystal team can always re-open the issue if there is enough demand for it.

ghost commented 1 year ago

Mingw-w64 is a different target than msvc and supporting it is still a goal I think.

correct. which is the reason language like Rust offer a MSVC and GNU build for Windows, the latter being MinGW-w64.

So I guess we could also close this issue because it's not directly actionable

how is it not actionable? wouldn't adding support for it be an action?

re-open once there's activity towards supporting this platform

please dont do that. you have over 1000 issues open, no harm in leaving this open. closing by a maintainer sends a strong signal that this is essentially WONTFIX, regardless of any reason given.

not many people care about it due to native Win32 support and the overall decline of Mingw due to WSL.

both of these items are false. Native Win32 support is only marginally useful, as developing with that environment requires an install of Visual Studio, which is at least 100 times the install size of MSYS2 and similar options. secondly, WSL has no impact on MinGW/MSYS2, because WSL is a Linux emulator, so you take a huge performance hit using that option, whereas MSYS2 offers native Windows output.

straight-shoota commented 1 year ago

how is it not actionable? wouldn't adding support for it be an action?

Actionable means to have a clear path what's necessary to achieve this, or at least the intention to discuss that and determine a path.

you have over 1000 issues open, no harm in leaving this open. closing by a maintainer sends a strong signal that this is essentially WONTFIX, regardless of any reason given.

Yes, we'd like to reduce that number. There's not much value in having lots of stale issues. We know that Mingw is not supported and that support for it would be welcome. But no need to keep an issue around that does nothing. There are dozens if not hundreds of somewhat reasonably relevant platforms which Crystal does not currently support. It doesn't make sense to keep open issues just to remind of that.

both of these items are false. Native Win32 support is only marginally useful, as developing with that environment requires an install of Visual Studio, which is at least 100 times the install size of MSYS2 and similar options. secondly, WSL has no impact on MinGW/MSYS2, because WSL is a Linux emulator, so you take a huge performance hit using that option, whereas MSYS2 offers native Windows output.

Sorry, I have a hard time making any sense of "Native Win32 support is only marginally useful". This has been one of the most requested features since almost the beginning of Crystal (#26). And even though we're not yet fully there, it's already used by many people. I fail to see any marginality of the most popular desktop operating system's native environment.

I also beg to differ that "WSL has no impact on MinGW/MSYS2". MSYS2 is a tool for using Unix programs on Windows. I used that myself back in the day. But I've long moved to WSL because it's a much more complete Unix environment with an actual Linux kernel interface that offers a lot more possibilities. Sure, WSL is not a replacement for all use cases of MSYS2. But for me, it is a much better one. And I'm pretty sure for many others as well.

ghost commented 1 year ago

Yes, we'd like to reduce that number. There's not much value in having lots of stale issues. We know that Mingw is not supported and that support for it would be welcome. But no need to keep an issue around that does nothing. There are dozens if not hundreds of somewhat reasonably relevant platforms which Crystal does not currently support. It doesn't make sense to keep open issues just to remind of that.

I get the desire to want to close issues, but closing them doesn't fix anything, it just makes you feel better. they should be closed when they are actually fixed, or when they decision is made that they will never be fixed.

Sorry, I have a hard time making any sense of "Native Win32 support is only marginally useful". This has been one of the most requested features since almost the beginning of Crystal (#26).

sorry I was just using your language. a better comment would be "MSVC support is only marginally useful". Windows developers use MSVC when its the only option. Ideally MSYS2 support would also exist, as it has several benefits over MSVC.

I also beg to differ that "WSL has no impact on MinGW/MSYS2". MSYS2 is a tool for using Unix programs on Windows.

MSYS2 is able to produce native Windows programs:

https://packages.msys2.org/base/mingw-w64-gcc

Sure, WSL is not a replacement for all use cases of MSYS2.

its not a replacement at all. WSL "programs" are linux programs that work on Windows via an emulator. MSYS2 produces native Windows programs that rely on the Windows API. the two couldn't be more different.

straight-shoota commented 1 year ago

I get the desire to want to close issues, but closing them doesn't fix anything, it just makes you feel better. they should be closed when they are actually fixed, or when they decision is made that they will never be fixed.

Keeping it open also does nothing towards getting it resolved. Closing issues with no forseeable activity helps to keep the amount of currently relevant issues manageable. And I think it's more honest to close the issue to make clear that it's not an active goal. The Core Team won't pursue Mingw support, so perhaps it might be more accurate to describe this as "wontfix". Although it's not a hard one. We're accepting contributions from the community. But until someone picks that up and starts actively working on it, this issue is just a zombie.

HertzDevil commented 9 months ago

To be clear, there are a few different toolchains possible on Windows:

Toolchain	Compiler	Triple	Flags	Additional runtimes
Microsoft Visual C++	`cl.exe`	`x86_64-pc-windows-msvc`	`windows` `msvc` `win32`	VCRedist, UCRT
MinGW-w64 (native)	`gcc.exe`	`x86_64-w64-windows-gnu`	`windows` `gnu` `win32`	VCRedist, (UCRT or MSVCRT)
MinGW-w64 (MSYS)	`gcc.exe`	`x86_64-w64-windows-gnu`	`windows` `gnu` `unix`	`msys-2.0.dll`
Cygwin	`gcc.exe`	`x86_64-unknown-cygwin` `x86_64-pc-windows-cygnus`	`windows` `gnu` `unix`	`cygwin1.dll`

MSYS is the "MSYS" environment of MSYS2, as indicated by $MSYSTEM; native is any other MSYS2 environment (e.g. UCRT64), or a stand-alone installation like WinLibs. Clang and LLVM are unmaintained and extremely outdated on Cygwin's repository, so we will forget about that for the moment.

While the majority of our MSVC toolchain work is completed, some deserves extra attention regarding what happens if other Windows toolchains are going to be supported, such as the meaning of "default shell". There is also the issue that MinGW-w64's license will always be more permissive than MSVC's, and that could be a deal breaker for some people. So I think now is a good time to revisit those alternative toolchains.

ysbaddaden commented 9 months ago

I see little value in supporting Cygwin or MSYS-2.0 since we support the official Windows API: there shouldn't be any portability issues of crystal code.

I can see value in MinGW-w64 as an alternative to MSVC, especially if we could link an EXE from linux or macos (though I'm not sure about the runtime in that case).

beta-ziliani commented 9 months ago

I see little value in supporting Cygwin or MSYS-2.0 since we support the official Windows API: there shouldn't be any portability issues of crystal code.

+1

especially if we could link an EXE from linux or macos

What? 😮

straight-shoota commented 9 months ago

especially if we could link an EXE from linux or macos

What? 😮

Sure. Linking against msys we wouldn't need any Windows system libraries, so it should be entirely feasible to cross-link executables.

beta-ziliani commented 9 months ago

Ah, right, thanks for clarifying 🙇

HertzDevil commented 9 months ago

You can cross-link Windows executables straight from non-Windows systems using x86_64-w64-mingw32-gcc, as long as all the third-party dependencies are also cross-linked. MinGW-w64 maintains its own import libraries for the Win32 API (which are always dynamically linked), and that's why you don't need anything from a Visual Studio installation for this to work.

x86_64-w64-windows-gnu is not merely about switching the compiler, by the way; the build environment should also answer questions like whether programs like sh.exe or make.exe or pkg-config.exe are expected to exist at build time, which makes a difference for things like library version detection. There is a reason why RubyInstaller comes with the MSYS2 UCRT64 environment or requires you to bring your own.

Anyway, here is a worked example of using MinGW-w64 stand-alone on Windows:

Install WinLibs, choosing Win64 + UCRT runtime + POSIX threads, to a location like %USERPROFILE%\mingw64;

Compile and install PCRE2 for MinGW-w64: (do not use the MSVC libs that come with Crystal)

git clone https://github.com/PCRE2Project/pcre2.git
cd pcre2
mkdir build
cd build
set PATH=%USERPROFILE%\mingw64\bin;%PATH%
cmake .. -G "MinGW Makefiles" -DBUILD_SHARED_LIBS=ON
cmake --build .
cmake -DCMAKE_INSTALL_PREFIX=%USERPROFILE%\crystal-libs-mingw64 -P cmake_install.cmake

Cross-compile the following code to get a test.obj file:

class String
def bytesize
  @bytesize
end

def to_unsafe
  pointerof(@c)
end
end

@[Link("kernel32")]
lib LibC
alias HANDLE = Void*
alias DWORD = UInt32
alias BOOL = Int32
alias SizeT = UInt64

STD_OUTPUT_HANDLE = 0xFFFFFFF5_u32

fun GetStdHandle(nStdHandle : DWORD) : HANDLE
fun WriteFile(hFile : HANDLE, lpBuffer : Void*, nNumberOfBytesToWrite : DWORD, lpNumberOfBytesWritten : DWORD*, lpOverlapped : Void*) : BOOL

fun malloc(size : SizeT) : Void*
end

@[Link("pcre2-8")]
lib LibPCRE2
alias Int = Int32

CONFIG_VERSION = 11

fun config = pcre2_config_8(what : UInt32, where : Void*) : Int
end

def print(msg)
LibC.WriteFile(LibC.GetStdHandle(LibC::STD_OUTPUT_HANDLE), msg, msg.bytesize, out _, nil)
end

print("Hello world on ")
print(Crystal::TARGET_TRIPLE)
print("\nUsing PCRE2 version ")

len = LibPCRE2.config(LibPCRE2::CONFIG_VERSION, nil)
buf = LibC.malloc(len).as(UInt8*)
LibPCRE2.config(LibPCRE2::CONFIG_VERSION, buf)
LibC.WriteFile(LibC.GetStdHandle(LibC::STD_OUTPUT_HANDLE), buf, len, out _, nil)

crystal build --prelude=empty --cross-compile --target=x86_64-w64-windows-gnu test.cr

Link the object file using the MinGW-w64 GCC:

%USERPROFILE%\mingw64\bin\x86_64-w64-mingw32-gcc.exe test.obj -lpcre2-8 -lkernel32 -lmsvcrt -L%USERPROFILE%\crystal-libs-mingw64\lib

Copy %USERPROFILE%\crystal-libs-mingw64\bin\libpcre2-8.dll to the current directory; (note the lib prefix which is absent in our MSVC build, so if we are not cross-compiling we need to adjust the dll: parameter for @[Link])

Run a.exe:

Hello world on x86_64-w64-windows-gnu
Using PCRE2 version 10.43 2024-02-16

We can supposedly symlink src/lib_c/x86_64-windows-msvc/ to src/lib_c/x86_64-windows-gnu/, but Crystal needs to know whether it is inside an MSYS2 environment or not, because an MSYS / Cygwin port would have used that same directory.

crysbot commented 4 months ago

This issue has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/the-easiest-way-to-prepare-a-ready-to-use-windows-executable-on-linux/7040/2

HertzDevil commented 1 month ago

Apart from cross-compilation, it is also possible to build and distribute Crystal itself using x86_64-windows-gnu as the default triple. Now is the time to consider whether such a distribution would be portable, like the MSVC one, or it would rely on an entire POSIX environment. The main difference is in how the standard library detects the version information of third-party libraries like OpenSSL:

https://github.com/crystal-lang/crystal/blob/991f9d0036e1e80297f2d934a8a3dc6dda1c36ce/src/openssl/lib_crypto.cr#L3-L20

On MSYS2, assuming a non-portable Crystal installation, there is probably no need for an openssl_VERSION file, so we could change the flag?(:win32) check into flag?(:msvc). But then the backticks on the other branch do not work: (this is of course related to #9030)

In src\openssl\lib_crypto.cr:16:27

 16 | {% from_libressl = (`hash pkg-config 2> /dev/null || printf %s false` != "false") &&
                          ^
Error: error executing command: hash pkg-config 2> /dev/null || printf %s false: The system cannot find the file specified.

They could actually be fixed by wrapping them in sh -c '...', because then the whole command string parses like a single process invocation:

      {% from_libressl = (`sh -c 'hash pkg-config 2> /dev/null || printf %s false'` != "false") &&
                         (`sh -c 'test -f $(pkg-config --silence-errors --variable=includedir libcrypto)/openssl/opensslv.h || printf %s false'` != "false") &&
                         (`sh -c 'printf "#include <openssl/opensslv.h>\nLIBRESSL_VERSION_NUMBER" | ${CC:-cc} $(pkg-config --cflags --silence-errors libcrypto || true) -E -'`.chomp.split('\n').last != "LIBRESSL_VERSION_NUMBER") %}
      {% ssl_version = `sh -c 'hash pkg-config 2> /dev/null && pkg-config --silence-errors --modversion libcrypto || printf %s 0.0.0'`.split.last.gsub(/[^0-9.]/, "") %}

Yet now the initial cross-compilation to x86_64-windows-gnu fails, because it tries to execute these shell commands on an MSVC toolchain host, with or without sh -c. (See also #14376.)

HertzDevil commented 1 month ago

LLVM is similar, if we assume that a MinGW-w64-based compiler is only portable within an MSYS2 shell. First the flag?(:win32) guard for the llvm_VERSION-specific macro code can be turned into flag?(:msvc). Then, since the macro backtick still delegates to LibC.CreateProcessW, POSIX shell scripts aren't considered executable on Windows, and now we must attach a shell here:

    {% llvm_config = env("LLVM_CONFIG") || `sh #{__DIR__}/ext/find-llvm-config`.stringify %}

This is somewhat related to #9030 and shows that we probably shouldn't assume /bin/sh to be the default shell on x86_64-windows-gnu.

If $LLVM_CONFIG isn't set, the next line will normally fail:

    {% llvm_version = `#{llvm_config.id} --version`.stringify %}

This is not a parsing issue, but simply because LibC.CreateProcessW cannot locate /ucrt64/bin/llvm-config.exe. (This is a root-relative path, so it might find something else actually.) We have to use cygpath inside src/llvm/ext/find-llvm-config itself:

if [ "$LLVM_CONFIG" ]; then
  case "$(uname -s)" in
    MSYS_NT*|MINGW32_NT*|MINGW64_NT*)
      printf "%s" "$(cygpath -w "$LLVM_CONFIG")"
      ;;
    *)
      printf "%s" "$LLVM_CONFIG"
      ;;
  esac
else
  # ...
fi

or alternatively, as described above, wrap each macro backtick invocation within sh -c, since MSYS2 understands /ucrt64/bin. The first option also makes Makefile work under MSYS2.

HertzDevil commented 1 month ago

Currently, MSYS2 only provides a Cygwin-based Git, which unlike Git for Windows doesn't handle Windows symlinks correctly. That means when attempting to check out this repository, one of the following things will happen:

core.symlinks=false: All symbolic links become plain text files, in particular the src/lib_c/x86_64-windows-gnu directory.
core.symlinks=true: ln will fail since lib/markd/lib and lib/reply/lib are recursive and symlinks become hard copies by default.
MSYS=winsymlinks:nativestrict core.symlinks=true: Some symlinks will still fail to create, although MSYS='winsymlinks:nativestrict' git restore --source=HEAD :/ at the repository root will successfully restore everything.

This could be a problem for Shards as well.

straight-shoota commented 1 month ago

I suppose we could work with core.symlinks=false. It requires the compiler to resolve symlink text files which shouldn't be too hard to implement.

HertzDevil commented 1 month ago

While we have been using the UCRT64 environment so far, the same cross-compilation also works for CLANG64 and MINGW64, but an even faster alternative is to download the "portable" artifact and do CRYSTAL=.../x86_64-mingw-w64-crystal/bin/crystal.exe make in that new environment. Their differences are:

cc and ld in CLANG64 are actually Clang and LLD, so all compiled binaries (including the compiler itself) depend on libc++.dll and libunwind.dll, rather than libstdc++-6.dll and libgcc_s_seh-1.dll.
MINGW64 links programs against the older MSVCRT runtime msvcrt.dll (not to be confused with msvcrt.lib, the MSVC toolchain's library for the C runtime startup code), instead of the UCRT. This older runtime's implementation for LibC.strtof and LibC.strtod is not C99-compliant, and fails to handle infinities and NaNs, affecting String#to_f and #to_f32. Ideally this will be fixed as part of #11952 though. Additionally, the @[Link] annotation in src/lib_c.cr needs to be commented out, otherwise C symbols will conflict.

The CLANGARM64 environment is only meaningful on an ARM64 Windows host, but it should make porting to ARM64 relatively straightforward.

The MSYS environment is a bit different. LLVM never officially supports MSYS, so if you ever see an x86_64-windows-msys triple, it originates from the numerous unmaintained patches exclusive to the MSYS2 package repository. I don't think there is much interest in getting Crystal to work there, now that WSL exists.

crystal-lang / crystal

Build with Mingw-w64 #6170