search

brechtsanders / winlibs_mingw

winlibs standalone build of GCC compiler and MinGW-w64
961 stars 44 forks source link

Some problems with LLVM #39

Closed Keithcat1 closed 3 years ago

Keithcat1 commented 3 years ago

It seems that for what ever reason, building projects with CMAKE and Ninja will fail if you use llvm-ar, which can't seem to create a static library. Also, I notice that you're linking LLVM dynamically. This seems to cause a bunch of weird things to happen like where ld.lld.exe will just sit there and never finish.

brechtsanders commented 3 years ago

Do you have a minimal reproducible scenario so I can reproduce those issues?

If the shared version is causing issues you should check your PATH environment variable for any paths before the winlibs binary folder that may contain DLLs with the same name.

Or is there an actual recommendation to build LLVM statically?

Keithcat1 commented 3 years ago

I'm trying to build the following repository: https://github.com/synthizer/synthizer To reproduce, git clone it and build it with CMAKE using: set CC=clang set CXX=clang++ cmake -G Ninja -DCMAKE_BUILD_TYPE=release -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=on .. ninja synthizer From within the build directory. I personally think it'd be a good idea linking LLVM statically. The only bad side effect is that the builds get quite a bit bigger.

brechtsanders commented 3 years ago

I tried to build that library with winlibs.com GCC and that worked. Then I tried with winlibs.com LLVM/CLANG and it showed a bunch of errors, starting with these:

FAILED: CMakeFiles/synthizer.dir/src/effects/fdn_reverb.cpp.obj 
D:\Prog\winlibs64-10.2.0-8.0.0\mingw64\bin\c++.exe -DBUILDING_SYNTHIZER -DWDL_RESAMPLE_TYPE=float -I../include -I../third_party/miniaudio -I../third_party/dr_libs -I../third_party/stb -I../third_party/cpp11-on-multicore/common -I../third_party/wdl -I../third_party/plf_colony -I../third_party/concurrentqueue -O3 -DNDEBUG -std=gnu++17 -MD -MT CMakeFiles/synthizer.dir/src/effects/fdn_reverb.cpp.obj -MF CMakeFiles\synthizer.dir\src\effects\fdn_reverb.cpp.obj.d -o CMakeFiles/synthizer.dir/src/effects/fdn_reverb.cpp.obj -c ../src/effects/fdn_reverb.cpp
In file included from ../include/synthizer/property_internals.hpp:6,
                 from ../include/synthizer/effects/fdn_reverb.hpp:13,
                 from ../src/effects/fdn_reverb.cpp:4:
../include/synthizer/property_xmacros.hpp:75:10: error: 'SYZ_P_MEAN_FREE_PATH' is not a type
   75 | DOUBLE_P(SYZ_P_MEAN_FREE_PATH, mean_free_path, MeanFreePath, 0.0, 0.5, 0.02) \
      |          ^~~~~~~~~~~~~~~~~~~~
../include/synthizer/effects/fdn_reverb.hpp:62:24: note: in expansion of macro 'FDN_REVERB_EFFECT_PROPERTIES'
   62 |  #define PROPERTY_LIST FDN_REVERB_EFFECT_PROPERTIES
      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
../include/synthizer/property_impl.hpp:105:2: note: in expansion of macro 'PROPERTY_LIST'
  105 |  PROPERTY_LIST
      |  ^~~~~~~~~~~~~
../include/synthizer/property_xmacros.hpp:75:32: error: 'mean_free_path' has not been declared
   75 | DOUBLE_P(SYZ_P_MEAN_FREE_PATH, mean_free_path, MeanFreePath, 0.0, 0.5, 0.02) \
      |                                ^~~~~~~~~~~~~~
../include/synthizer/effects/fdn_reverb.hpp:62:24: note: in expansion of macro 'FDN_REVERB_EFFECT_PROPERTIES'
   62 |  #define PROPERTY_LIST FDN_REVERB_EFFECT_PROPERTIES
      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
../include/synthizer/property_impl.hpp:105:2: note: in expansion of macro 'PROPERTY_LIST'
  105 |  PROPERTY_LIST
      |  ^~~~~~~~~~~~~
../include/synthizer/property_xmacros.hpp:75:48: error: 'MeanFreePath' has not been declared
   75 | DOUBLE_P(SYZ_P_MEAN_FREE_PATH, mean_free_path, MeanFreePath, 0.0, 0.5, 0.02) \
      |                                                ^~~~~~~~~~~~
../include/synthizer/effects/fdn_reverb.hpp:62:24: note: in expansion of macro 'FDN_REVERB_EFFECT_PROPERTIES'
   62 |  #define PROPERTY_LIST FDN_REVERB_EFFECT_PROPERTIES
      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
../include/synthizer/property_impl.hpp:105:2: note: in expansion of macro 'PROPERTY_LIST'
  105 |  PROPERTY_LIST
      |  ^~~~~~~~~~~~~

So somehow the code isn't working well with clang. Does it expect a certain C++ standard?

Keithcat1 commented 3 years ago

Did you do: set CC=path/to/clang.exe set CXX=path/to/clang++.exe You also need to enable LTO or this won't happen for some reason.

luisdallos commented 3 years ago

It seems that for what ever reason, building projects with CMAKE and Ninja will fail if you use llvm-ar, which can't seem to create a static library.

Same here.

/mingw64/bin/llvm-ar.exe rcS libtest.a *.o
C:\msys64\mingw64\bin\llvm-ar.exe: error: unable to open 'libtest.a': No such file or directory

I had to resort to the llvm-ar binary from the official installer at LLVM site (LLVM-11.0.0-win64.exe).

Also, I notice that you're linking LLVM dynamically. This seems to cause a bunch of weird things to happen like where ld.lld.exe will just sit there and never finish.

This looks like some sort of multi-threading issue. Turning off multi-threading support in LLD by passing -fuse-ld=lld -Wl,--Xlink=-threads:1 to GCC/Clang works around this issue.

brechtsanders commented 3 years ago

Maybe it's related to the combination of LTO and multithreading. I tried to reproduce again, but now synthizer now longer builds with gcc, so it looks like new issues were introduced in its updates. I could really do with a more minimal example to reproduce this.

luisdallos commented 3 years ago

Maybe it's related to the combination of LTO and multithreading.

Even with LTO turned off this issue shows up. I could really do with a more minimal example to reproduce this.

A small test (e.g. "hello world") program is enough to reproduce the issue with LLD.

Test (try 50 times to compile the test program):

for i in `seq 1 1 50`
do
    echo "==> Try $i"
    clang -fuse-ld=lld test.c -o test
done

After a variable number of tries the linker gets stuck, so the only alternative is proceeding to kill the linker process manually as typing Ctrl+C kills the compiler process (gcc.exe/clang.exe) but not the underlying linker process (ld.lld.exe).

Only by disabling multi-threading in LLD (i.e. passing -Wl,--Xlink=-threads:1 to GCC/Clang) this issue goes away.

brechtsanders commented 3 years ago

Can you please try to reproduce this issue with LLVM 12 from the latest release and let me know if it's solved now?

Keithcat1 commented 3 years ago

I updated to LLVM 12 / GCC 11 and now all 32-bit binaries using LLVM (llvm-nm, clang, lldb) ETC crash on startup (GCC works fine). There is no error message. I type a command, it sits there for a while and then windows tells me that this clang.exe isn't responding. I don't have access to my 64-bit computer for now.

brechtsanders commented 3 years ago

I'm not able to reproduce your issue. Make sure the location of clang.exe and the other tools is specified first in the PATH variable to make sure you're not pulling DLL files from other locations.

Here's a quick test I did in a Command Prompt:

SET PATH=D:\Prog\winlibs32-11.1.0\mingw32\bin;%PATH%
ECHO #include "stdio.h" > _.c
ECHO int main() { printf("Hello world!"); return 0; } >> _.c
clang -c -o _.o _.c
clang -o _.exe _.o
_.exe

And the result is:

Hello world!

You could try to check the EXE files that you have issues with using DependancyWalker to see if there are issues loading certain DLLs.

Another thing you should try is to disable antivirus as this may also cause issues. After all, you are running a program that is generating executable code and writing that to disk...

Keithcat1 commented 3 years ago

At my 64-bit computer now. The llvm-ar error seems to have been fixed, but the ld.lld hanging still happens once.

luisdallos commented 3 years ago

I updated to LLVM 12 / GCC 11 and now all 32-bit binaries using LLVM (llvm-nm, clang, lldb) ETC crash on startup (GCC works fine). There is no error message. I type a command, it sits there for a while and then windows tells me that this clang.exe isn't responding.

When debugged, this issue turns out to be an "illegal instruction" exception thrown from libz3.dll

$ gdb --args J:\mingw32\bin\clang.exe -v
...
This GDB was configured as "i686-w64-mingw32".
...

...
Reading symbols from J:\mingw32\bin\clang.exe...
(No debugging symbols found in J:\mingw32\bin\clang.exe)
(gdb) run
Starting program: J:\mingw32\bin\clang.exe -v

Program received signal SIGILL, Illegal instruction.
0x699e277f in libz3!_ZN11ast_manager15raise_exceptionEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE () from J:\mingw32\bin\libz3.dll
(gdb)

And the offending instruction is:

(gdb) disassemble /r 0x699e277f,+1
Dump of assembler code from 0x699e277f to 0x699e2780:
=> 0x699e277f <libz3!_ZN11ast_manager15raise_exceptionEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+681835>:   c5 f9 ef c0     vpxor  %xmm0,%xmm0,%xmm0
End of assembler dump.
(gdb)

Note that the VPXOR instruction (with xmm registers as operands) requires AVX instruction set support, which is not available in X86/AMD64 processor families such as Intel Core2.

leduyquang753 commented 3 years ago

Having the same problem as luisdallos described above. Using Intel Core i5 M560 on Dell Latitude E6510.

brechtsanders commented 3 years ago

@Keithcat1 @luisdallos @leduyquang753 Thanks to each of you for pinpointing the issue. I have rebuilt the z3 with compiler flag -march=pentium4 for 32-bit and -march=core2 for 64-bit Windows. That way SSE instructions are still allowed but AVX will not be used. Can you please dowbload http://winlibs.com/z3.dll.zip and copy the right libz3.dll over the one that comes with https://github.com/brechtsanders/winlibs_mingw/releases/tag/11.1.0-8.0.0-r2 and let me know if that fixes it?

leduyquang753 commented 3 years ago

The same problem is still present unfortunately.

PS D:\Temporary files > gdb d:\gcc\bin\clang.exe
GNU gdb (GDB for MinGW-W64 x86_64, built by Brecht Sanders) 10.2
Reading symbols from d:\gcc\bin\clang.exe...
(No debugging symbols found in d:\gcc\bin\clang.exe)
(gdb) r --version
Starting program: d:\gcc\bin\clang.exe --version
[New Thread 2936.0x3544]
[New Thread 2936.0x13ac]
[New Thread 2936.0x3178]

Thread 1 received signal SIGILL, Illegal instruction.
0x00007fff7d7a0dec in libz3!_ZN11ast_manager15raise_exceptionEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE ()
   from d:\gcc\bin\libz3.dll
(gdb) disassemble /r 0x00007fff7d7a0dec,+1
Dump of assembler code from 0x7fff7d7a0dec to 0x7fff7d7a0ded:
=> 0x00007fff7d7a0dec <libz3!_ZN11ast_manager15raise_exceptionEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+826542>:  c5 f9 ef c0      vpxor  %xmm0,%xmm0,%xmm0
End of assembler dump.
luisdallos commented 3 years ago

The 32-bit libz3.dll works well:

$ /j/mingw32/bin/gdb.exe --args /j/mingw32/bin/clang.exe -v
GNU gdb (GDB for MinGW-W64 i686, built by Brecht Sanders) 10.2
...
This GDB was configured as "i686-w64-mingw32".
...

...
Reading symbols from J:/mingw32/bin/clang.exe...
(No debugging symbols found in J:/mingw32/bin/clang.exe)
(gdb) run
Starting program: J:\mingw32\bin\clang.exe -v
(built by Brecht Sanders) clang version 12.0.0
Target: i686-w64-windows-gnu
Thread model: posix
InstalledDir: J:\mingw32\bin
[Inferior 1 (process 4132) exited normally]

About the 64-bit libz3.dll, same problem as leduyquang753 described above.

Both tests have been done on a PC with an Intel Core 2 Duo E4700.

brechtsanders commented 3 years ago

@Keithcat1 @luisdallos @leduyquang753 Second attempt, this time built z3 with -march=i686/x86_64 -mtune=generic -msse2. Could you please download http://winlibs.com/z3.dll.zip again and test both 32-bit and 64-bit Windows?

leduyquang753 commented 3 years ago

The instruction is still there (on 64-bit), nothing changed.

(gdb) r
Starting program: D:\GCC\bin\clang.exe
[New Thread 13576.0x1ce4]
[New Thread 13576.0x3758]
[New Thread 13576.0x620]

Thread 1 received signal SIGILL, Illegal instruction.
0x00007fff84f645bc in libz3!_ZN11ast_manager15raise_exceptionEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE ()
   from D:\GCC\bin\libz3.dll
(gdb) disassemble /r 0x00007fff84f645bc,+1
Dump of assembler code from 0x7fff84f645bc to 0x7fff84f645bd:
=> 0x00007fff84f645bc <libz3!_ZN11ast_manager15raise_exceptionEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+830110>:  c5 f9 ef c0      vpxor  %xmm0,%xmm0,%xmm0
End of assembler dump.
luisdallos commented 3 years ago

Second attempt, this time built z3 with -march=i686/x86_64 -mtune=generic -msse2. Could you please download http://winlibs.com/z3.dll.zip again and test both 32-bit and 64-bit Windows?

The problem is still present on 64-bit and this time on 32-bit too:

32-bit test:

$ /j/mingw32/bin/gdb.exe --args /j/mingw32/bin/clang.exe -v
GNU gdb (GDB for MinGW-W64 i686, built by Brecht Sanders) 10.2
...
This GDB was configured as "i686-w64-mingw32".
...

...
Reading symbols from J:/mingw32/bin/clang.exe...
(No debugging symbols found in J:/mingw32/bin/clang.exe)
(gdb) run
Starting program: J:\mingw32\bin\clang.exe -v

Program received signal SIGILL, Illegal instruction.
0x6172277f in libz3!_ZN11ast_manager15raise_exceptionEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE () from J:\mingw32\bin\libz3.dll
(gdb) disassemble /r 0x6172277f,+1
Dump of assembler code from 0x6172277f to 0x61722780:
=> 0x6172277f <libz3!_ZN11ast_manager15raise_exceptionEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+681835>:   c5 f9 ef c0     vpxor  %xmm0,%xmm0,%xmm0
End of assembler dump.

64-bit test:

$ /j/mingw64/bin/gdb.exe --args /j/mingw64/bin/clang.exe -v
GNU gdb (GDB for MinGW-W64 x86_64, built by Brecht Sanders) 10.2
...
This GDB was configured as "x86_64-w64-mingw32".
...

...
Reading symbols from J:/mingw64/bin/clang.exe...
(No debugging symbols found in J:/mingw64/bin/clang.exe)
(gdb) run
Starting program: J:\mingw64\bin\clang.exe -v

Program received signal SIGILL, Illegal instruction.
0x00007ffc25b545bc in libz3!_ZN11ast_manager15raise_exceptionEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE () from J:\mingw64\bin\libz3.dll
(gdb) disassemble /r 0x00007ffc25b545bc,+1
Dump of assembler code from 0x7ffc25b545bc to 0x7ffc25b545bd:
=> 0x00007ffc25b545bc <libz3!_ZN11ast_manager15raise_exceptionEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+830110>:   c5 f9 ef c0     vpxor  %xmm0,%xmm0,%xmm0
End of assembler dump.
luisdallos commented 3 years ago

Second attempt, this time built z3 with -march=i686/x86_64 -mtune=generic -msse2. Could you please download http://winlibs.com/z3.dll.zip again and test both 32-bit and 64-bit Windows?

And both variants have same file size as their original counterparts and differ in only 5 bytes.

$ cksfv.exe /j/mingw32/bin/libz3.dll{.orig,}
;     29498894  07:10.57 2021-05-02 J:/mingw32/bin/libz3.dll.orig
;     29498894  12:16.38 2021-05-07 J:/mingw32/bin/libz3.dll
J:/mingw32/bin/libz3.dll.orig CC43F771
J:/mingw32/bin/libz3.dll 1A10D9F4

$ cmp -bl /j/mingw32/bin/libz3.dll{.orig,}
     217 340 M-`   31 ^Y
     218 114 L     54 ,
26697221 214 M-^L 276 M->
26697222 210 M-^H 147 g
26697223 216 M-^N 225 M-^U
$ cksfv.exe /j/mingw64/bin/libz3.dll{.orig,}
;     27818510  06:19.06 2021-05-02 J:/mingw64/bin/libz3.dll.orig
;     27818510  12:16.06 2021-05-07 J:/mingw64/bin/libz3.dll
J:/mingw64/bin/libz3.dll.orig DEE0C05D
J:/mingw64/bin/libz3.dll C4122B7F

$ cmp -bl /j/mingw64/bin/libz3.dll{.orig,}
     217 204 M-^D 325 M-U
     218 235 M-^] 210 M-^H
25434117 113 K    225 M-^U
25434118 174 |    147 g
25434119 216 M-^N 225 M-^U
brechtsanders commented 3 years ago

@Keithcat1 @luisdallos @leduyquang753

Third attempt of z3 built in 2 different ways:

Can you please download http://winlibs.com/z3.dll.zip and try again?

leduyquang753 commented 3 years ago

Still the same instruction.

brechtsanders commented 3 years ago

On 64-bit only?

leduyquang753 commented 3 years ago

On 64-bit only?

Yeah I only run 64-bit. Things have been kinda repetitive so I gradually omitted the details.

brechtsanders commented 3 years ago

Tough nut to crack, this one...

Maybe I should build the dll with debugging info, so we can find out where exactly the offending instruction is located...

brechtsanders commented 3 years ago

Unfortunately z3 exports too many symbols when building debug version (linker error: export ordinal too large: 103489) :-(

brechtsanders commented 3 years ago

This time attached z3's CMakeLists.txt and changed if (HAS_SSE2) to if (HAS_SSE2 AND NOT MINGW) to make sure that stuff is disabled (as AVX comes with it). I have also added z3.exe itself and made sure the binaries are not stripped. Can you please download http://winlibs.com/z3_attempt4.zip and try again? I'm also curious to know if z3.exe crashes the same way.

leduyquang753 commented 3 years ago

Good news this time, they run now.

PS D:\Temporary files > z3 --version
Z3 version 4.8.10 - 64 bit
PS D:\Temporary files > clang --version
(built by Brecht Sanders) clang version 12.0.0
Target: x86_64-w64-windows-gnu
Thread model: posix
InstalledDir: D:\GCC\bin
luisdallos commented 3 years ago

This time attached z3's CMakeLists.txt and changed if (HAS_SSE2) to if (HAS_SSE2 AND NOT MINGW) to make sure that stuff is disabled (as AVX comes with it). I have also added z3.exe itself and made sure the binaries are not stripped. Can you please download http://winlibs.com/z3_attempt4.zip and try again? I'm also curious to know if z3.exe crashes the same way.

Both variants work fine:

$ /j/mingw32/bin/clang.exe --version (built by Brecht Sanders) clang version 12.0.0 Target: i686-w64-windows-gnu Thread model: posix InstalledDir: J:\mingw32\bin


- 64-bit:

$ /j/mingw64/bin/z3.exe --version Z3 version 4.8.10 - 64 bit

$ /j/mingw64/bin/clang.exe --version (built by Brecht Sanders) clang version 12.0.0 Target: x86_64-w64-windows-gnu Thread model: posix InstalledDir: J:\mingw64\bin

brechtsanders commented 3 years ago

Ok, then I think we found the solution. This should be my final attempt: http://winlibs.com/z3_attempt5.zip This one is not stripped and I have removed the -march=core2 flag for 64-bit. I had to keep -march=pentium4 for 32-bit in order to be able to build without SSE2. If you can confirm this works then this is how I will build it for the next release.

leduyquang753 commented 3 years ago

They also work normally this time. We are good now it seems.

brechtsanders commented 3 years ago

Excellent! I have added an extra download for the latest release, and I will build z3 like this for the future releases.

Keithcat1 commented 3 years ago

The problem where ld.lld hangs forever is still not solved yet, should this issue be closed?

brechtsanders commented 3 years ago

Is there any more information about that? For example, can you run the ld.lld process to reproduce the hanging state and check with a debugger (GDB) where it hangs (in which DLL or even in which function)?