Please support MSYS2 MINGW64

[Razakhel]

Hi and thanks for your issue!

I mostly use MinGW-w64 myself, although not through MSYS2. What do you need to build RaZ this way, what's blocking for you?

[ghost]

What do you need to build RaZ this way, what's blocking for you?

I'm mostly use the MSYS2 MINGW64 Shell as it has everything needed for me.

[Razakhel]

After trying it out, I can make the engine compile with a few additions, which are on the way. However, as I can't find anywhere if there is a specific definition available in code to check if using MSYS, I'm not able to properly make the code compatible with it at the moment. I can't really consider it supported (and I will most definitely not test it frequently, if at all).

If anything, I would advise dropping MSYS2 entirely. MinGW64 works fine on its own; I usually download mine at https://winlibs.com/, but I've also heard about https://github.com/mstorsjo/llvm-mingw, which would be fine too. If it works for you, let me know how this goes!

[ghost]

After trying it out, I can make the engine compile with a few additions, which are on the way. However, as I can't find anywhere if there is a specific definition available in code to check if using MSYS, I'm not able to properly make the code compatible with it at the moment. I can't really consider it supported (and I will most definitely not test it frequently, if at all).

If anything, I would advise dropping MSYS2 entirely. MinGW64 works fine on its own; I usually download mine at https://winlibs.com/, but I've also heard about https://github.com/mstorsjo/llvm-mingw, which would be fine too. If it works for you, let me know how this goes!

MSYS2 is the Linux-like build environment. MINGW64 is the MinGW64 compiler. MSYS2 MINGW64 is the MinGW64 compiler with a Linux-like build environment. If it's working on the standalone MinGW64, I should work on MSYS2 MINGW64 too.

[ghost]

cmake configuration failed with:

$ cmake ..
-- Building for: Ninja
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/msys64/mingw64/bin/cc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/msys64/mingw64/bin/c++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
[OpenAL] Found:
  - Include directory: C:/msys64/mingw64/include
  - Library: C:/msys64/mingw64/lib/libopenal.dll.a
  - DLL: OPENAL_DLL-NOTFOUND
CMake Error at examples/CMakeLists.txt:102 (file):
  file COPY_FILE failed to copy

    C:/msys64/home/Administrator/RaZ/examples/OPENAL_DLL-NOTFOUND

  to

    C:/msys64/home/Administrator/RaZ/build/examples/OpenAL32.dll

  because: No such file or directory (input)

CMake Error at tests/CMakeLists.txt:78 (file):
  file COPY_FILE failed to copy

    C:/msys64/home/Administrator/RaZ/tests/OPENAL_DLL-NOTFOUND

  to

    C:/msys64/home/Administrator/RaZ/build/tests/OpenAL32.dll

  because: No such file or directory (input)

-- Found Doxygen: C:/msys64/mingw64/bin/doxygen.exe (found version "1.9.6") found components: doxygen dot
-- Configuring incomplete, errors occurred!

This is the same problem as https://github.com/SpartanJ/eepp/issues/21#issuecomment-1508009391

[Razakhel]

If it's working on the standalone MinGW64, I should work on MSYS2 MINGW64 too.

It's not so simple. As you say, MSYS is a whole environment (you could say a platform on its own which makes no real difference here), while MinGW is just a compiler. Should you use the standalone MinGW, your environment would be Windows itself, without anything else in the way. The issues I had yesterday with MSYS is related to the environment, not the compiler (which now accepts my code just fine). The fact that OpenAL is not usable as-is is a proof of that: it is directly related to the environment, as MSYS doesn't have everything expected; MinGW has nothing to do with it.

I've been using MinGW (as a standalone compiler) for several years, and I can guarantee RaZ (and all my projects for that matter) compiles just fine using a recent enough version (9.0+ if I recall correctly, as prior versions have some internal issues). MinGW is not tied to MSYS and neither of them requires the other. If you absolutely want a Linux-like environment, you should use WSL to have a real one. Cygwin is technically supported on my side (at least with Clang), although I haven't tried for a few years now so I guarantee nothing. The fact that you use Ninja even with MSYS is also something I cannot really comprehend; if you don't use MSYS Makefiles, you should just get Ninja & MinGW under Windows. You will literally have no difference in usage, just less troubles.

Concerning your OpenAL issue, I made it so that the DLL is not mandatory for it to be considered found. It may indeed be an issue, although it should not happen in a common environment. As it is not required to build RaZ, you can disable it giving the option -DRAZ_USE_AUDIO=OFF when calling CMake. However, with my OpenAL-Soft installation under Windows and without ever having installed it with MSYS, it is found as expected (see the dedicated wiki page to setup it if needed):

[OpenAL] Found:
  - Include directory: C:/Libraries/OpenAL/include
  - Library: C:/Libraries/OpenAL/libs/Win64/libOpenAL32.dll.a
  - DLL: C:/Libraries/OpenAL/bin/Win64/soft_oal.dll

Either way, for now you will run into other issues though, as just like Cygwin, MSYS removes the _WIN32 definition which breaks the code in a few places. This is the change that is on the way.

Also, be aware that with MSYS's MinGW (in Debug mode only), due to the template-heavy Lua scripting, the library takes around ~20 minutes to link, the script demo another ~10 minutes, and the tests another ~20 minutes. You should disable scripting giving -DRAZ_USE_SOL2=OFF to CMake (FYI, this does not occur with a recent standalone MinGW version, given the right compiler/linker arguments, which takes "only" around 3 or 5 minutes. It does not happen at all with other compilers, which are able to do it within seconds).

[ghost]

Concerning your OpenAL issue, I made it so that the DLL is not mandatory for it to be considered found. It may indeed be an issue, although it should not happen in a common environment. As it is not required to build RaZ, you can disable it giving the option -DRAZ_USE_AUDIO=OFF when calling CMake. However, with my OpenAL-Soft installation under Windows and without ever having installed it with MSYS, it is found as expected (see the dedicated wiki page to setup it if needed):

[OpenAL] Found:
  - Include directory: C:/Libraries/OpenAL/include
  - Library: C:/Libraries/OpenAL/libs/Win64/libOpenAL32.dll.a
  - DLL: C:/Libraries/OpenAL/bin/Win64/soft_oal.dll

Disabling features is not an option. And did you read the resource I linked?

Go here and see: https://github.com/SpartanJ/eepp/issues/21#issuecomment-1508009391

This particular comment if you want something straight forward: https://github.com/SpartanJ/eepp/issues/21#issuecomment-1509967354

To find the description of any MSYS2 package, just Google with software name msys2, e.g: openal msys2 and it will give you the result immediately.

The name of the OpenAL DLL on MSYS2 MINGW64 is libopenal-1.dll even though the import library's name is still libopenal.dll.a. See: https://packages.msys2.org/package/mingw-w64-x86_64-openal (scroll to the Files section at the bottom of the page).

You said you support Cygwin. Yeah, MSYS2 is a fork of Cygwin, the naming convention of DLL is the same and you could see Cygwin is worse as it replaces lib with cyg so the DLL name is cygopenal-1.dll.

https://www.cygwin.com/packages/x86_64/libopenal1/libopenal1-1.18.2-1

Cygwin is not an option anymore as people all embraced MSYS2. There are so many packages become unmaintained on Cygwin. Clang and LLVM are also unmaintained on Cygwin. It's really that bad!

Either way, for now you will run into other issues though, as just like Cygwin, MSYS removes the _WIN32 definition which breaks the code in a few places. This is the change that is on the way.

This information is wrong. I have build so many software on MSYS2 MINGW64 and I'm sure _WIN32 is always defined. You are confusing the MSYS2 Shell (which is basically Cygwin and provides a POSIX compatibility layer) with the MSYS2 MINGW64 Shell we are talking about here (which doesn't need any POSIX compatibility layer as the binaries generated by this environment are native Win32 binaries). Please have a look:

https://www.msys2.org/docs/environments/

Also, be aware that with MSYS's MinGW (in Debug mode only), due to the template-heavy Lua scripting, the library takes around ~20 minutes to link, the script demo another ~10 minutes, and the tests another ~20 minutes. You should disable scripting giving -DRAZ_USE_SOL2=OFF to CMake (FYI, this does not occur with a recent standalone MinGW version, given the right compiler/linker arguments, which takes "only" around 3 or 5 minutes. It does not happen at all with other compilers, which are able to do it within seconds).

That is not a problem. I could use the MSYS2 CLANG64 Shell as it's using LLVM's LLD instead of GNU LD. It's basically the same compiler as https://github.com/mstorsjo/llvm-mingw is used plus the MSYS2 build environment.

See: https://github.com/brechtsanders/winlibs_mingw/issues/118

I'm familiar with both of the projects you linked (winlibs and llvm-mingw). I prefer to use MSYS2 than using them directly.

[Razakhel]

Disabling features is not an option.

Can I ask why? What do you intend to use RaZ for?

The name of the OpenAL DLL on MSYS2 MINGW64 is libopenal-1.dll even though the import library's name is still libopenal.dll.a.

I've added to the upcoming changes another DLL name to be searched, which now works:

[OpenAL] Found:
  - Include directory: C:/MSYS64/mingw64/include
  - Library: C:/MSYS64/mingw64/lib/libopenal.dll.a
  - DLL: C:/MSYS64/mingw64/bin/libopenal-1.dll

you could see Cygwin is worse as it replaces lib with cyg so the DLL name is cygopenal-1.dll.

I tested Cygwin way before adding OpenAL, so it may be an issue now indeed.

This information is wrong. I have build so many software on MSYS2 MINGW64 and I'm sure _WIN32 is always defined. You are confusing the MSYS2 Shell (which is basically Cygwin and provides a POSIX compatibility layer) with the MSYS2 MINGW64 Shell we are talking about here (which doesn't need any POSIX compatibility layer as the binaries generated by this environment are native Win32 binaries).

I'll let you try this in your MSYS MinGW64 shell:

...@Razakhel MINGW64 ~
$ echo "#include <iostream>
#ifdef _WIN32
#pragma message(\"Windows\")
#else
#pragma message(\"Not Windows\")
#endif
int main() {
    std::cout <<
#ifndef _WIN32
    \"Not \"
#endif
    \"Windows\n\";
}" > main.cpp && g++ main.cpp -o msys_win32 && ./msys_win32

which gives me:

main.cpp:5:30: note: ‘#pragma message: Not Windows’
    5 | #pragma message("Not Windows")
      |                              ^
Not Windows

Given this, I have to insist: _WIN32 is not defined here. The errors I had came from this exact issue, and forcing it to be defined "fixed" them. There are even still issues in ImGui that originates from this too. Please let me know if I missed something.

That is not a problem. I could use the MSYS2 CLANG64 Shell as it's using LLVM's LLD instead of GNU LD.

This is what I meant by "given the right compiler/linker arguments". I already give -Wa,-mbig-obj -fuse-ld=lld, and it takes forever even with these options. But well, the specific Clang64 environment may be different, although I doubt it; let me know if it is.

[ghost]

So my MSYS2 MINGW64 is different from your MSYS2 MINGW64? I don't think so!

$ echo "#include <iostream>
#ifdef _WIN32
#pragma message(\"Windows\")
#else
#pragma message(\"Not Windows\")
#endif
int main() {
    std::cout <<
#ifndef _WIN32
    \"Not \"
#endif
    \"Windows\n\";
}" > main.cpp && g++ main.cpp -o msys_win32 && ./msys_win32

main.cpp:3:26: note: '#pragma message: Windows'
    3 | #pragma message("Windows")
      |                          ^
Windows

[ghost]

Well, I think I know the problem. Please post the result of gcc -v of you here. Is it something like this?

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-msys/11.3.0/lto-wrapper.exe
Target: x86_64-pc-msys
Configured with: /c/S/gcc/src/gcc-11.3.0/configure --build=x86_64-pc-msys --prefix=/usr --libexecdir=/usr/lib --enable-bootstrap --enable-shared --enable-shared-libgcc --enable-static --enable-version-specific-runtime-libs --with-arch=x86-64 --with-tune=generic --disable-multilib --enable-__cxa_atexit --with-dwarf2 --enable-languages=c,c++,fortran,lto --enable-graphite --enable-threads=posix --enable-libatomic --enable-libgomp --disable-libitm --enable-libquadmath --enable-libquadmath-support --disable-libssp --disable-win32-registry --disable-symvers --with-gnu-ld --with-gnu-as --disable-isl-version-check --enable-checking=release --without-libiconv-prefix --without-libintl-prefix --with-system-zlib --enable-linker-build-id --with-default-libstdcxx-abi=gcc4-compatible --enable-libstdcxx-filesystem-ts
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 11.3.0 (GCC)

Or like this?

$ gcc -v
Using built-in specs.
COLLECT_GCC=C:\msys64\mingw64\bin\gcc.exe
COLLECT_LTO_WRAPPER=C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-12.2.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/include --libexecdir=/mingw64/lib --enable-bootstrap --enable-checking=release --with-arch=nocona --with-tune=generic --enable-languages=c,lto,c++,fortran,ada,objc,obj-c++,jit --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts --enable-libstdcxx-time --disable-libstdcxx-pch --enable-lto --enable-libgomp --disable-multilib --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev10, Built by MSYS2 project' --with-bugurl=https://github.com/msys2/MINGW-packages/issues --with-gnu-as --with-gnu-ld --disable-libstdcxx-debug --with-boot-ldflags=-static-libstdc++ --with-stage1-ldflags=-static-libstdc++
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 12.2.0 (Rev10, Built by MSYS2 project)

which gcc gives /mingw64/bin/gcc or /usr/bin/gcc?

I'm pretty sure you have not yet installed the MINGW64 compiler. What you are using is the MSYS2 compiler aka Cygwin compiler. Please use this command to install the MINGW64 compiler: pacman -S mingw-w64-x86_64-toolchain and remember any packages for MSYS2 MINGW64 environment has the prefix mingw-w64-x86_64-.

You could think the MSYS2 Shell is the base shell. MSYS2 MINGW64 or MSYS2 CLANG64 are just environment variable trick by putting /mingw64/bin or /clang64/bin before /usr/bin in PATH so /mingw64/bin/gcc or /clang64/bin/gcc will be found before /usr/bin/gcc. If you didn't have /mingw64/bin/gcc or /clang64/bin/gcc then it's naturally for /usr/bin/gcc to be picked.

[Razakhel]

You're right, it seems that I had installed the ucrt version; it works as expected with the proper one. It now compiles just fine, but there seems to be a mismatch with my standalone version, as Dependencies indicates that MinGW's DLLs (libgcc_s_seh-1.dll, libwinpthread-1.dll & libstdc++6.dll) are found from my other installation. You may not have this problem on your side, so I won't mind it right now.

Also, another necessary flag to dramatically reduce linking times in Debug was -Og. It is almost mandatory if you don't want to wait for ages on every recompilation.

Changes are available on the msys2_support branch. If you want to checkout and test it out, let me know the result!

[ghost]

You're right, it seems that I had installed the ucrt version; it works as expected with the proper one. It now compiles just fine, but there seems to be a mismatch with my standalone version, as Dependencies indicates that MinGW's DLLs (libgcc_s_seh-1.dll, libwinpthread-1.dll & libstdc++6.dll) are found from my other installation. You may not have this problem on your side, so I won't mind it right now.

On UCRT64 Shell, packages have the prefix mingw-w64-ucrt-x86_64-, so the MinGW compiler is mingw-w64-ucrt-x86_64-toolchain. The DLLs are picked up from PATH (Windows' PATH, not MSYS2's PATH). You could remove your standalone MinGW from Windows' PATH and add C:\msys64\ucrt64\bin to PATH instead, it will pick up the correct DLLs.

p/s: this is the very reason why I prefer MSYS2 over standalone MinGW, as MSYS2 Shells are isolated from each other and each Shell has environment variables correctly setup for you so you don't have to mess with PATH, this way you could use multiple MinGW based toolchains on your computer without conflict, without having to handle PATH manually like you currently have to.

[ghost]

Also, another necessary flag to dramatically reduce linking times in Debug was -Og. It is almost mandatory if you don't want to wait for ages on every recompilation.

I experienced no such problem. The linking speed is normal without needing any flags. Maybe this only apply to recompiling? I don't know, because I deleted the build directory and started from scratch so I didn't do any recompiling.

Changes are available on the msys2_support branch. If you want to checkout and test it out, let me know the result!

Confirmed it's building fine now. Thank you.

[Razakhel]

Merged on master (696ede5).

[brechtsanders]

There were additional optimizations possible when building the https://winlibs.com/ standalone builds. The following releases have them enabled:

• https://github.com/brechtsanders/winlibs_mingw/releases/tag/13.1.0-16.0.5-11.0.0-msvcrt-r5
• https://github.com/brechtsanders/winlibs_mingw/releases/tag/13.1.0-16.0.5-11.0.0-ucrt-r5

[Razakhel]

Thanks for reaching out! Unfortunately, although it does seem to have improved slightly, it still takes around 13 minutes to compile & link (the latter step taking most of the time) the whole project in Debug mode, even when using lld. When using -Og instead of the default (supposedly -O0), it goes down to 3 minutes. In Release mode, it takes around 2 minutes. For comparison, MSVC takes around 30 seconds to do the same. I should try with Clang someday, to check if it works better!