stockfish.exe hangs in Cutechess and loses random games due to 'connection stalls'

hero2017 commented 5 years ago

Normally I compile SF on Windows 2012 R2 with gcc (9.1 now) but it seems to hang at random points during games in my cutefish tournaments although it never hangs when I use it in infinite analysis.

So I wanted to try compiling with clang for the first time but so far no luck. I downloaded & installed LLVM 8.0.1 (pre-built x64 binaries). This is on a dual Xeon E5-2696v3 system.

I'm using the following make command:

make build ARCH=x86-64-bmi2 COMP=clang

but I quickly get an error:

Testing config sanity. If this fails, try 'make help' ...

make ARCH=x86-64-bmi2 COMP=clang all
make[1]: Entering directory '/c/Users/user1/Desktop/Stockfish-master/src'
clang++ -Wall -Wcast-qual -fno-exceptions -std=c++11  -pedantic -Wextra -Wshadow
 -m64 -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -DUSE_PEXT -mss
e4 -mbmi2 -flto   -c -o benchmark.o benchmark.cpp
benchmark.cpp:21:10: fatal error: 'fstream' file not found
#include <fstream>
         ^~~~~~~~~
1 error generated.
make[1]: *** [<builtin>: benchmark.o] Error 1

snicolet commented 5 years ago

Normally I compile SF on Windows 2012 R2 with gcc (9.1 now)

Hi @hero2017, thanks for the report. Two questions:

a) Could you give us more details about the compiler environment you use normally? b) Do you think your issue is related (for exemple using the same compiling suite) to the issues https://github.com/official-stockfish/Stockfish/issues/2229 and https://github.com/official-stockfish/Stockfish/issues/2291 ?

erbsenzaehler commented 5 years ago

I had the same problem when I tried clang on Windows. The problem is that clang alone does not detect detect the location of the mingw64 headers/libraries. I had to add --target=x86_64-w64-mingw32 to the Makefile. Can you provide us with the output of clang -v? https://clang.llvm.org/docs/CrossCompilation.html

mstembera commented 5 years ago

@hero2017 I would like to see if I can reproduce your hangs from the gcc 9.1 compile but am not sure where to install gcc 9.1 for Windows from. Can you advise?

hero2017 commented 5 years ago

@mstembera: Please use this link to get gcc 9.1 and compile method: https://github.com/glinscott/fishtest/wiki/Building-stockfish-on-Windows

MSYS2 MinGW 64-bit shell is what I normally use to compile my Windows builds and I never had any hang issues prior to gcc 9.1.

I set my windows path with:

path=E:\msys64\mingw64\bin;E:\msys64\usr\bin;C:\Program Files\LLVM\bin then I use my little batch script to compile:

make.exe profile-build ARCH=x86-64-bmi2 COMP=gcc -j 36

After running strip.exe on it I have my LTO version of SF that's ready to go.

I've been using this method for years and never had any issues with SF in cutefish. I noticed it only after SF was beaten badly in a few tournaments due to time forfeits.

@snicolet: My issue is not the same as #2291 because I never have problems during compiling when using mingw64. I only have problems using it in cutefish gui on windows where SF just keeps hanging at random positions and losing on time. This started after compiling in MSYS MingW64 using gcc 9.1

@erbsenzaehler: Is that all you did to compile SF using clang on windows? You just added --target=x86_64-w64-mingw32 ? Where in the makefile did you add it?

EDIT: Thanks @erbsenzaehler: I added it like so on line 212 and it almost worked with:

CXXFLAGS += -pedantic -Wextra -Wshadow --target=x86_64-w64-mingw32

But then it errored with:

benchmark.o: file not recognized: File format not recognized clang++: error: linker command failed with exit code 1 (use -v to see invocation ) make[1]: *** [Makefile:500: stockfish] Error 1 Here's my clang version info:

C:\Program Files\LLVM\bin>clang -v clang version 8.0.1 (tags/RELEASE_801/final) Target: x86_64-pc-windows-msvc Thread model: posix InstalledDir: C:\Program Files\LLVM\bin

I'm using the pre-built version from here: (https://github.com/llvm/llvm-project/releases/download/llvmorg-8.0.1/LLVM-8.0.1-win64.exe)

erbsenzaehler commented 5 years ago

add the target stuff to the CXXFLAGS and add -fuse-ld=lld to the linker flag. Profile-guided optimization did not work with clang on windows when I last tried

CoffeeOne commented 5 years ago

@hero2017 2 Questions: The first question is not very important, but I am curious: You write, that you are using MSYS2 and have gcc9.1. So you do not update your MSYS2 installation? I see:

$ g++ --version
g++.exe (Rev2, Built by MSYS2 project) 9.2.0

The second point is essential and important: Can you downgrade your winpthread library and test again? Here is the detailed instruction: 1) Go to http://repo.msys2.org/mingw/x86_64/ Download 2 files from this site:

http://repo.msys2.org/mingw/x86_64/mingw-w64-x86_64-winpthreads-git-7.0.0.5325.11a5459d-1-any.pkg.tar.xz http://repo.msys2.org/mingw/x86_64/mingw-w64-x86_64-libwinpthread-git-7.0.0.5325.11a5459d-1-any.pkg.tar.xz 2) Copy those 2 files with the windows explorer to your MSYS2 home drive. 3) Downgrade the 2 packages in MSYS by opening your MSYS2 shell (you are automatically in your homedrive then) and running the command: pacman -U mingw-w64-x86_64-libwinpthread-git-7.0.0.5325.11a5459d-1-any.pkg.tar.xz mingw-w64-x86_64-winpthreads-git-7.0.0.5325.11a5459d-1-any.pkg.tar.xz 4) Then build a new binary as usual.

Does this change something in the problem "in cutefish gui on windows where SF just keeps hanging at random positions and losing on time" ?

hero2017 commented 5 years ago

@CoffeeOne Thanks. Your instructions worked perfectly but unfortunately the issue persists and my very first game hung at the first position out of book (usually happens later but it's random as I said).

Also, I just used the instructions on the page I provided to install and update MSYS2 and then I don't do anything unless I hear a new gcc version was released. So perhaps those instructions aren't correct or up to date or I'm doing something wrong. What do you suggest I do to check if there's a new gcc/g++ update and the commands to run them after opening the MSYS2 shell?

hero2017 commented 5 years ago

@erbsenzaehler Awesome! I was able to successfully compile my very first clang version of SF. I thought it didn't worked as I couldn't find stockfish.exe. I found a file called 'stockfish' (no extension) so I just renamed it to stockfish.exe and ran strip on it and it worked.

Many thanks erbsenzaehler!! Time to test if clang is better than gcc/mingw. Hopefully this version doesn't hang like the gcc version.

CoffeeOne commented 5 years ago

@hero2017 pacman -Syuu Of course you will upgrade the winpthread library, too. I recommend to downgrade it afterwards (after having upgraded everything else).

hero2017 commented 5 years ago

Ok that's what I've been doing: pacman -Syuu which I just ran and updated my gcc to 9.2.0. Btw, I don't think I was using g++ 9.2.0 before.

In any case, after every upgrade of gcc I have to downgrade the winpthread library with the the commands you provided earlier? And I guess I also have to update my C:\Windows\System32 folder by copying:

libgcc_s_seh-1.dll libstdc++-6.dll libwinpthread-1.dll

for LTO builds?

Anyway, I've upgraded to gcc 9.2.0 now but SF continues to hang. I then downgraded winpthread as you suggested and still SF hangs.

Anything else I can try or should check?

erbsenzaehler commented 5 years ago

Clang links against the libs in your E:\msys64\mingw64\bin folder, but the problem should be in the libwinpthread of msys2 if understand the discussions correctly

hero2017 commented 5 years ago

@erbsenzaehler I think that's what they're saying but I've already tried upgrading and downgrading winpthread as suggested and that's not resolving the engine freezing in Cutefish gui. I'm forced to always download the pre-built exe's from abrok to check SF strentgh in cutefish. I never had to do that before gcc 9.1 or higher though.

Good news is that whenever I test my gcc compiled SF vs clang SF it's always gcc SF that freezes not clang SF. At least so far (since it doesn't happen in every game).

erbsenzaehler commented 5 years ago

@hero2017 Do you build your gcc builds from the msys2-shell? Does Stockfish still crash if you use your old mingw64 setup with gcc via cmd?

hero2017 commented 5 years ago

No, I build them from my .bat file which simply issues the make command I mentioned earlier. That's how I've always compiled earlier versions and it was fine.

make.exe profile-build ARCH=x86-64-bmi2 COMP=gcc -j 36

After more testing with only SF clang I see the clang versions also freeze in Cutefish so now I'm at a loss as to what to do.

CoffeeOne commented 5 years ago

Can you provide a log file of cutechess? Interesting would be the hang right after the opening book. Maybe we can read out something .... Compiling with clang makes no sense, because it's slower than the gcc build. Before you switch the toolchain, can you try to build with COMP=mingw with up to date MSYS2, but with the downgraded libwinpthread(!)? Doing so will make sure, that the exe is not depending on any external dll.

EDIT: I suggested that, because you wrote: "And I guess I also have to update my C:\Windows\System32 folder by copying: ..." That's frightening, so we don't know which dll was used, maybe you are still using the problematic dll. Better copy the dlls to the folder, where the stockfish.exe is located, then you have better control about it.

hero2017 commented 5 years ago

@CoffeeOne Putting the libraries with SF makes no difference. SF wouldn't even run in infinite analysis if that was the problem. And the libraries are updated in system32 too so it's something else. I tried it anyway but no dice. Also, I could not find any cutefish log file in its folder, nor is there an option to enable the log in Cutefish. I'll try to search for the log in other paths later.

I did try using COMP=mingw instead of gcc but same problem. Maybe doesn't happen as fequently but nevertheless the problem persists. So what does this mean?

On another note, can the SF team give us the option to use -march=native by issuing a parameter with Make.exe like for cFish during compilation? Meaning, I'd just have to add that to the make.exe command in my batch file rather than always manually adding it to CXXFLAGS in Makefile. In cFish I just compile with the following this way we never have to touch Makefile:

make profile-build ARCH=x86-64-bmi2 bits=64 optimize=yes **native=yes** numa=yes pext=yes extra=yes COMP=gcc -j 36

NOTE: With or without adding -march=native manually to Makefile SF still freezes.

CoffeeOne commented 5 years ago

That would mean that your problem is not caused by the libwinpthread library, so I do not know.

You can compile stockfish without making any changes in the makefile with native: Use: make profile-build ARCH=x86-64-bmi2 COMPCXX='g++ -march=native' -j

hero2017 commented 5 years ago

Just a minute. I'm trying LB (LittleBlitzer) since it can also run multiple simultaneous games. I can't watch the games live the way I can in Cutefish but looking at the score SF is leading for a change so I don't think it's forfeiting/freezing. I'm letting it run longer to make sure.

If it turns out to be Cutefish then it'll be to my surprise. I've always compiled cutefish myself also and didn't have this problem but perhaps I had a problem with MSYS2 and didn't even know it the last time I compiled it.

hero2017 commented 5 years ago

It looks like it was the gui afterall since SF vs Raubfisch won 54 to 47.5. I will have to see what's up with Cutefish.

Sorry for wasting everyone's time. I really appreciate all your efforts in trying to resolve my issue. I should've thought of trying LB before.

But one thing doesn't make sense. If it was the gui then why do the abrok versions of SF not have the same issue? Perhaps SF won because the # of wins were higher than the # of timeouts due to freezing? I will try a pre-built release of cutechess to find out.

hero2017 commented 5 years ago

I don't get it. I tried the pre-built cutefish version and my SF compiles still freeze. So the LB score may have only been good for SF because SF doesn't freeze frequently enough.

Anyone have any other ideas?

hero2017 commented 5 years ago

I just tried to compile directly in the MSYS2 shell using:

make profile-build ARCH=x86-64-bmi2 COMP=gcc -j 36

but I get:

make: *** No rule to make target 'profile-build'. Stop.

But the same command works if I run it from a windows .bat file. Could this have something to do with my SF freezing issue?

CoffeeOne commented 5 years ago

Go to the src folder

hero2017 commented 5 years ago

Ahh yes of course. It's obvious I've spent too much time on why SF is hanging and am too tired to figure out the simplest issues. Thanks.

snicolet commented 5 years ago

@hero2017 Is the issue still there for you?

hero2017 commented 5 years ago

I'd have to say yes but not 100% sure now. When I compiled another engine today, BrainLearn, it was having the same hanging problem. But yesterday I compiled the latest SF (Raise stack size to 8MB for pthreads) and so far it seems ok.

For BrainLearn 4.1a I had to use their pre-built exe for it to work in cutechess. I do know it's not the gui but either something with the compiler (starting from g++ 9.1.0) or there was a bug in previous versions of SF until the patch above. If the SF version I compiled yesterday continues to work, and it looks like it is, then I will try to compile one of the patches prior to the pthreads patch to see it was a SF bug. BrainLearn 4.1a doesn't have the pthreads patch so it's pointing to the pthreads patch as the fix.

hero2017 commented 5 years ago

I'm hoping this help. I've enabled SF engine logging and started a Cutefish tournament with 6 simultaneous games (6 cores each @ 2GB hash). No books. The 3rd game hung.

I'm not sure if the log is useful because I see a lot of NUL values there which could be because 6 copies of Stockfish.exe are trying to write to it.

Here's the pgn and log file: https://mir.cr/15CL4DFP

Also, if I play only one game at a time SF doesn't seem to hang but I don't know what that means. Maybe it just didn't happen after only 3 games.

hero2017 commented 5 years ago

Btw, I compiled with gcc 7.4 and 9.1 (after reinstalling MSYS2 & gcc 9.1.0 completely) and it doesn't hang with either one. I then installed a fresh copy of MSYS2 with gcc 9.2.0) and SF started hanging again. So it's something with gcc 9.2.0 only.

There's only one problem. When I compile with gcc 9.1 or mingw then tablebases don't work. Very strange.

What should I do?

CoffeeOne commented 5 years ago

@hero2017 I would have liked to help, because I was interested, if your problem is similar to the problem that happened in TCEC (2 times). Also you have similar hardware: You have 2 times Xeon 18 cores, TCEC has 2 times Xeon 22 cores. Unfortunately I still don't know. You wrote "SF was beaten badly in a few tournaments due to time forfeits" and "my very first game hung", so that are strong hints, that your self-compiled binaries have the known problem with the libwinpthread.

The problem happens only when you play 6 games simultaneously (with giving 6 threads to each engine? That could mean that it's hard to reproduce, but it still could point to fact that you could have the "libwinpthread problem".

Strongest point would be when the problem exists only with your self compiled and you have no problems at all with the binaries from abrok.eu, then it points also to the winpthread library problem.

So just stick to the abrok builds, most likely there will be a new library in msys2 very soon, then you can switch back to self compilation (after updating everything to latest in MSYS2)

One more comment: You wrote: "Btw, I compiled with gcc 7.4 and 9.1 (after reinstalling MSYS2 & gcc 9.1.0 completely) and it doesn't hang with either one. I then installed a fresh copy of MSYS2 with gcc 9.2.0) and SF started hanging again. So it's something with gcc 9.2.0 only." That's not clear at all to me. with gcc 7.4 you probably mean that you didn't use the gcc/g++ inside MSYS2, you used the downloaded toolchain from sourceforge and used it from you MSYS2 shell. But again, the libwinpthread problem has absolutely NOTHING to do with gcc 9.1 or gcc 9.2. You need to check your version of libwinpthread. You can easily find out which pthread lib you use with pacman -Qs pthread See my first comment which version is the last good one.

Very last comment about: "There's only one problem. When I compile with gcc 9.1 or mingw then tablebases don't work. Very strange." That's completely off-topic. Open an extra issue for it and describe the problem.

CoffeeOne commented 5 years ago

My very last comment :D The whole thread is off-topic. We are discussing about compilation with MSYS2, but the heading is still "fstream error when compiling SF in Windows", please change the heading.

CoffeeOne commented 5 years ago

@hero2017 I forgot the very very last comment: Take care about the dlls, when you make builds with COMP=gcc, you know I am talking of

libgcc_s_seh-1.dll
libstdc++-6.dll
libwinpthread-1.dll

That's important when you use different gcc versions on the PC: When you compile with gcc 7.4 you need to copy those files from the folder where gcc7.4 is located, of course the same for gcc 9.1 and gcc 9.2, for example you should never combine an exe created with gcc7.4 with the dlls of gcc 9.1!!! Imagine that you want to make a small tournament where stockfish compiled with your gcc 7.4 plays against stockfish compiled with your gcc 9.2, that is only possible when you copy the right dll files to the folder where you put the exe. So do NOT copy the dlls to a windows system folder (which is in the PATH), when you have to copy the dlls anyway. Why do you want to copy the dlls to a windows folder, too? From which gcc version you have the dlls in your PATH (you can have only one)? Please also note that those dlls can change after every pacman -Syuu command in MSYS2 for the latest gcc (which is now 9.2).

So please search for those dll files on all your harddisks and delete it everywhere, except the mingw folders. If you forget to copy it, stockfish does not start, that is much better than to use wrong dlls, that could lead to weird problems. Maybe you have such problems already.

Sorry for the lengthy comment.

hero2017 commented 5 years ago

Thank you for the response. Sorry I've been away for so long.

I'm well aware of those dll files. I've already made that adjustment (putting the dll's with the engine and deleting them from the system path) but it made no difference. As I mentioned it doesn't really matter in this case because I have the same problem when compiling with COMP=mingw

Last night I spent some more time trying gcc 8.1 this time and still no issues when playing tourneys in Cutefish. Not a single SF loss due to "connection stalls." So it works fine with gcc 7.3, 7.4, and 8.1.

One thing that's weird is that I thought that when SF is compiled with COMP=gcc then you need to have those 3 dll files with the engine (of course from the same version of gcc) but for 8.1 I don't need them. The engine still runs fine. I made sure the files are not in my system path and not with the engine. Maybe it's like that with older versions of gcc on windows? I don't remember if it's like that with gcc 7.x. With gcc 9.x as soon as I forget to copy the dll files to the engine folder and try to run it I get an error.

Yes I believe I have almost exactly the same hardware as TCEC. Even the same SuperMicro motherboard. The only diff is as you said the extra 4 cores because they use the v4 Xeons and I use the older v3 and the ONLY difference is the 4 cores.

EDIT: Btw I believe the only reason why the abrok builds work fine on my system is because they're compiled with gcc 7.3 which also works fine for me if I compile it myself using this version.

hero2017 commented 5 years ago

You need to check your version of libwinpthread. You can easily find out which pthread lib you use with pacman -Qs pthread See my first comment which version is the last good one.

@CoffeeOne Also, I tried your command for checking libwinpthread but I don't know what this is telling me:

# pacman -Qs pthread local/mingw-w64-x86_64-libwinpthread-git 7.0.0.5522.977a9720-1 (mingw-w64-x86_64-toolchain) MinGW-w64 winpthreads library local/mingw-w64-x86_64-winpthreads-git 7.0.0.5522.977a9720-1 (mingw-w64-x86_64-toolchain) MinGW-w64 winpthreads library

This is the version it installed after running pacman -Syuu so if it's not the right one for gcc 9.2 then how do I install the right one?

CoffeeOne commented 5 years ago

One thing that's weird is that I thought that when SF is compiled with COMP=gcc then you need to have those 3 dll files with the engine (of course from the same version of gcc) but for 8.1 I don't need them. The engine still runs fine. I made sure the files are not in my system path and not with the engine. Maybe it's like that with older versions of gcc on windows? I don't remember if it's like that with gcc 7.x. With gcc 9.x as soon as I forget to copy the dll files to the engine folder and try to run it I get an error.

You can check that by yourself, too. After compiling stockfish, run the ldd command on it: ldd stockfish.exe. You have to do it in MSYS2 environment, ldd is not available in the windows cmd. I see for a build with COMP=gcc:

$ ldd stockfish.exe
        ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffd9cfa0000)
        KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ffd9c5d0000)
        KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll (0x7ffd99fc0000)
        msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll (0x7ffd9be10000)
        libwinpthread-1.dll => /mingw64/bin/libwinpthread-1.dll (0x64940000)
        libstdc++-6.dll => /mingw64/bin/libstdc++-6.dll (0x6fc40000)
        libgcc_s_seh-1.dll => /mingw64/bin/libgcc_s_seh-1.dll (0x61440000)

When it's like this, you need to have those dlls somewhere, otherwise the exe won't run. For a build with COMP=mingw I get:

$ ldd stockfish.exe
        ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffd9cfa0000)
        KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ffd9c5d0000)
        KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll (0x7ffd99fc0000)
        msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll (0x7ffd9be10000)

CoffeeOne commented 5 years ago

# pacman -Qs pthread local/mingw-w64-x86_64-libwinpthread-git 7.0.0.5522.977a9720-1 (mingw-w64-x86_64-toolchain) MinGW-w64 winpthreads library local/mingw-w64-x86_64-winpthreads-git 7.0.0.5522.977a9720-1 (mingw-w64-x86_64-toolchain) MinGW-w64 winpthreads library

This is the version it installed after running pacman -Syuu so if it's not the right one for gcc 9.2 then how do I install the right one?

=> You have the latest libwinpthread from MSYS2 (yes, the one you get with pacman -Syuu) This version has the "hanging issue". So downgrade it, I described it above. You already did it, as far as I remember.

CoffeeOne commented 5 years ago

One more remark about: " ..... but for 8.1 I don't need them. The engine still runs fine. I made sure the files are not in my system path and not with the engine. Maybe it's like that with older versions of gcc on windows? "

I wanted to give you the chance to find out by yourself..... But I spoil it now, the answer is: You need the dlls ALWAYS for builds with COMP=gcc, also with older toolchain versions.

Building with g++.exe (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0 (please note: This is not a gcc / g++ version only, it's a complete tool-chain, with tools and libraries, ....) results in a exe:

$ ldd stockfish.exe
        ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffd9cfa0000)
        KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ffd9c5d0000)
        KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll (0x7ffd99fc0000)
        msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll (0x7ffd9be10000)
        libstdc++-6.dll => /mingw64-810/bin/libstdc++-6.dll (0x6fc40000)
        libwinpthread-1.dll => /mingw64-810/bin/libwinpthread-1.dll (0x64940000)
        USER32.dll => /c/WINDOWS/System32/USER32.dll (0x7ffd9c320000)
        win32u.dll => /c/WINDOWS/System32/win32u.dll (0x7ffd9aeb0000)
        GDI32.dll => /c/WINDOWS/System32/GDI32.dll (0x7ffd9c0c0000)
        libgcc_s_seh-1.dll => /mingw64-810/bin/libgcc_s_seh-1.dll (0x61440000)
        gdi32full.dll => /c/WINDOWS/System32/gdi32full.dll (0x7ffd9ad10000)
        msvcp_win.dll => /c/WINDOWS/System32/msvcp_win.dll (0x7ffd9a270000)
        ucrtbase.dll => /c/WINDOWS/System32/ucrtbase.dll (0x7ffd9af00000)

hero2017 commented 5 years ago

@CoffeeOne Thank you for confirming what I believed to be true. Which is why I always copied the 3 dll files when compiling with COMP=gcc but at that time I thought \windows\system32 was ok. I've created a script for my compilations now depending on the version of gcc I plan to use so these 3 files always get copied to where stockfish.exe is going to live. I'm now using gcc 9.1 (I believe I tried it already but since I barely remember I wanted to make sure) and that still hangs.

Thanks for reminding me about the ldd command. Very useful. As you said, I had those dll files just not where I expected them. They were in \windows. I must've copied them there by accident a long time ago because I always copied them to windows\system32 or the engine location before and never anywhere else. I feel like an idiot now, thank you :-/

Yes I did downgrade libwinpthread before but it didn't solve the hanging issue. Maybe now with the 3 dll's removed from \windows and then libwinpthread downgraded it'll work. But if I'm going to use gcc 9.2 and I downgrade libwinpthread (I assume your instructions will downgrade it to 9.1) then isn't that a no-no?

hero2017 commented 5 years ago

Btw, thank you @CoffeeOne for all your help. I learned some valuable tools like ldd, 'pacman -Qs pthrea'd and being able to downgrade libwinpthread. Now my 3 library files match the 3 id's that you have in your screenshot.

After removing the 3 dll's found in my \windows path and downgrading libwinpthread my SF compiles using gcc v9.2.0 are finally working (16 games played so far and no a single hang in cutechess, not to mention SF is finallying winning again even if it is still very early).

The only thing that still doesn't make sense is why was it also hanging whenever I compiled with COMP=mingw? And I know it was mingw because those stockfish.exe binaries are always much bigger.

CoffeeOne commented 5 years ago

Fine, so problem solved. Since the fix of the libwinpthread is not available via MSYS2 packet manager, we have to downgrade the lib in the MSYS2 environment (now with we have gcc 9.2.0 Rev.2.

The hang is - of course - independant from compiling with COMP=mingw or COMP=gcc, when we create one big exe with static linking, we use still the libwinpthread, it's just part of the exe.

By the way, I also gave you instructions how to build with -march=native without modifying the Makefile, you forgot to thank me for that. :D :D :D @ppigazzini Would it make sense to put in into the compiling instructions to build windows and linux? With that trick, we can also increase effiency in the fishtest framework (when people put it into custom_make.txt)

ppigazzini commented 5 years ago

@CoffeeOne :

Wiki updated, commit message: "Use the COMPCXX variable to build SF with -march=native. Suggested by CoffeeOne (thanks!)"
the Wiki needs a refactoring, and I need a clone :)
in Makefile we could keep COMPILER and drop COMPCXX, double check with @snicolet @vondele
we should have all the pieces to write the "custom_make.txt" according to the CPU architecture and to use -march=native flag. Please open an Issue on fishtest.

vondele commented 5 years ago

@ppigazzini thanks for documenting!

Some comments however:

the more canonical way to do this (rather than defining COMPCXX) is to set the CXXFLAGS as in:
```
CXXFLAGS='-march=native' make -j ARCH=x86-64-modern build
```
this might in that case also work if the underlying compiler binary is not called g++.

note that this is not yet the optimal architecture build, e.g. even though -march=native will provide -mbmi2, we still need to set -DUSE_PEXT (pext = yes) manually (using ARCH=x86-64-bmi2 currently).

I'm traveling and away from computers in the next week or so, feel free to adopt these suggestions.

apart from some CI legacy, I don't know why we have both COMPILER and COMPCXX (in addition to COMP). At least one of the two can be removed, AFAICT. In principle, they should both be superfluous as defining CXX like above should work, but it doesn't quite with our current Makefile.

hero2017 commented 5 years ago

Fine, so problem solved. Since the fix of the libwinpthread is not available via MSYS2 packet manager, we have to downgrade the lib in the MSYS2 environment (now with we have gcc 9.2.0 Rev.2. The hang is - of course - independant from compiling with COMP=mingw or COMP=gcc, when we create one big exe with static linking, we use still the libwinpthread, it's just part of the exe.

Makes total sense.

By the way, I also gave you instructions how to build with -march=native without modifying the Makefile, you forgot to thank me for that. :D :D :D

I'm so glad you mentioned that or I'd have missed those instructions. Somehow I didn't see that and you mentioned it so long ago now. I've been manually adding this to the Makefile all this time :-( Thanks so much for this. I can finally compile SF without any manual intervention. I have my script that adds larges pages to SF and now -march=native will be part of all my SF binaries.

Now I'm going to get a little greedy and say that it would really be nice to have SF include Large Pages. It doesn't 'have to be enabled by default. I always see people asking how to add LP and I'm tired of answering the same questions for so many years. I'm also getting tired of manually updating that part of the code whenever SF makes changes to tt.h, tt.cpp and ucioption.h. My script relies on copying these 3 files which is fine as long as SF hasn't updated them which would break LP otherwise.

@ppigazzini Would it make sense to put in into the compiling instructions to build windows and linux? With that trick, we can also increase effiency in the fishtest framework (when people put it into custom_make.txt)

ppigazzini commented 5 years ago

@vondele ~~CXXFLAGS='-march=native' breaks PGO~~ Ignore, I viewed only now that you wrote build and not profile-build.

ppigazzini commented 5 years ago

@vondele thank you, I learned a new thing, Wiki page updated. We should document in the Makefile help target how to use a temporary shell variable CXXFLAGS to pass some optional compiler flags to the make command line, something like this:

### ==========================================================================
### Section 4. Public targets
### ==========================================================================

help:
        @echo ""
        @echo "To compile stockfish, type: "
        @echo ""
        @echo "[CXXFLAGS=additional flags] make target ARCH=arch [COMP=compiler] [COMPCXX=cxx]"

...

        @echo "Advanced examples, for experienced users: "
        @echo ""
        @echo "make build ARCH=x86-64 COMP=clang"
        @echo "CXXFLAGS='-march=native' make profile-build ARCH=x86-64-bmi2 COMP=gcc COMPCXX=g++-4.8"
        @echo ""

Anyway prepending the CXXFLAGS shell variable appends 3 times all the flags crowding the output, e.g. CXXFLAGS='' make profile-build ARCH=x86-64-modern COMP=gcc

ppigazzini commented 5 years ago

@CoffeeOne @snicolet @vondele @noobpwnftw the msys2 update seems to have fixed the libwinpthead bug

$ pacman -Syuu
...
Packages (5) mingw-w64-x86_64-binutils-2.33.1-1
             mingw-w64-x86_64-crt-git-7.0.0.5546.d200317d-1
             mingw-w64-x86_64-headers-git-7.0.0.5546.d200317d-1
             mingw-w64-x86_64-libwinpthread-git-7.0.0.5544.15da3ce2-1
             mingw-w64-x86_64-winpthreads-git-7.0.0.5544.15da3ce2-1

The SF binary built with this libwinpthread version completed successfully a 1000 games match concurrency 8 @ 1 thread:

.\cutechess-cli.exe -repeat -rounds 1000 -tournament gauntlet -resign movecount=3 score=400 -draw movenumber=34 movecount=8 score=20 -concurrency 8 -engine cmd=stockfish.exe option.Hash=8 option.Threads=1 -engine cmd=stockfish.exe option.Hash=8 option.Threads=1 -each proto=uci tc=10+0.1 -openings file=2moves_v1.pgn format=pgn order=random plies=16

a 100 games match concurrency 2 @ 4 threads:

.\cutechess-cli.exe -repeat -rounds 100 -tournament gauntlet -resign movecount=3 score=400 -draw movenumber=34 movecount=8 score=20 -concurrency 2 -engine cmd=stockfish.exe option.Hash=8 option.Threads=4 -engine cmd=stockfish.exe option.Hash=8 option.Threads=4 -each proto=uci tc=10+0.1 -openings file=2moves_v1.pgn format=pgn order=random plies=16

and completed successfully a cycle of 500 fishtest bench processes:

1 process @ 7 threads go infinite and,
1 process @ 1 thread fishtest bench

CoffeeOne commented 5 years ago

Yes, works for me, too. I was in the "lucky" position, that the hang came very quickly on my machine, so I did not need very much time to test. I closed the mingw issue.

CoffeeOne commented 5 years ago

@hero2017 You can safely upgrade all packages in MSYS2 now, no more need to downgrade the libwinpthread. About large pages: There was already a pull-request (as far as I remember from the author of asmfish), but it was rejected. So I don't think this time it will be different. You can close this issue now.

snicolet commented 5 years ago

I am closing the issue now, as the original problem has been understood and solved.

Concerning large pages, maybe a paragraph in one of our wiki pages?

hero2017 commented 4 years ago

@hero2017 You can safely upgrade all packages in MSYS2 now, no more need to downgrade the libwinpthread. About large pages: There was already a pull-request (as far as I remember from the author of asmfish), but it was rejected. So I don't think this time it will be different. You can close this issue now.

@CoffeeOne - I'm so glad to hear that the issue has been resolved and we can safely upgrade all packages in MSYS2.

However I don't understand what you meant by "there was already a pull-request..." pull meaning rejected? Why reject LP? Most engines, especially clones have it and it's not a problem so I don't know why it would be with SF. Not a big deal as I have it scripted but I don't have to still keep updating if any of the files that have LP code change when SF releases an update.

official-stockfish / Stockfish

stockfish.exe hangs in Cutechess and loses random games due to 'connection stalls' #2305