Unused variable

[vdbergh]

clang++ 6.0.0 (Google colab) gives the following error

"evaluate.cpp:52:30: warning: unused variable 'gEmbeddedNNUEEnd' [-Wunused-const-variable]\n", '  const unsigned char *const 

[vondele]

that's a variable declared by the INCBIN macro. I think we should ignore that warning, it is also not present in more recent clang++.

[vdbergh]

I know it is not an error but it prevents Google colab workers from using clang. I thought I'd mention that here.

[ppigazzini]

Executing make -j 2 profile-build ARCH=x86-64-bmi2 COMP=clang failed. Error: ["evaluate.cpp:52:30: warning: unused variable 'gEmbeddedNNUEEnd' [-Wunused-const-variable]\n", '  const unsigned char *const gEmbeddedNNUEEnd = &gEmbeddedNNUEData[1];\n', '                             ^\n', '1 warning generated.\n', '/bin/sh: 1: llvm-profdata: not found\n', 'make[1]: *** [clang-profile-use] Error 127\n', 'make: *** [profile-build] Error 2\n']

Same exception using clang++ 12.0.1 on colab

!wget https://apt.llvm.org/llvm.sh && bash llvm.sh 12 && update-alternatives --install /usr/bin/clang clang /usr/bin/clang-12 100 --slave /usr/bin/clang++ clang++ /usr/bin/clang++-12

Worker Linux with the same clang works fine, though.

[vondele]

diff --git a/src/evaluate.cpp b/src/evaluate.cpp
index a5c049a88..002450e13 100644
--- a/src/evaluate.cpp
+++ b/src/evaluate.cpp
@@ -53,7 +53,6 @@
   const unsigned int         gEmbeddedNNUESize = 1;
 #endif

-
 using namespace std;

 namespace Stockfish {
@@ -108,6 +107,8 @@ namespace Eval {

                 MemoryBuffer buffer(const_cast<char*>(reinterpret_cast<const char*>(gEmbeddedNNUEData)),
                                     size_t(gEmbeddedNNUESize));
+                // silence warning on unused gEmbeddedNNUEEnd
+                (void) gEmbeddedNNUEEnd;

                 istream stream(&buffer);
                 if (load_eval(eval_file, stream))

should silence the warning.

[vondele]

btw, I think the reason the worker fails is a different one: https://tests.stockfishchess.org/actions?count=1&before=1642859486.122 error loading plugin: /usr/lib/llvm-6.0/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory\n'

[vondele]

on my ubuntu setup, the first clang version that works for profile build is version 10.

[ppigazzini]

The error on colab is due to the missing profiled data, here the SF build with clang-12:

!wget https://apt.llvm.org/llvm.sh && bash llvm.sh 12 && update-alternatives --install /usr/bin/clang clang /usr/bin/clang-12 100 --slave /usr/bin/clang++ clang++ /usr/bin/clang++-12
!git clone --single-branch --branch master https://github.com/official-stockfish/Stockfish.git
!cd Stockfish/src && make -j 2 profile-build ARCH=x86-64-bmi2 COMP=clang && make strip && stockfish compiler && stockfish bench

clang++ -Wall -Wcast-qual -fno-exceptions -std=c++17 -fprofile-instr-generate  -pedantic -Wextra -Wshadow -m64 -DUSE_PTHREADS -DNDEBUG -O3 -fexperimental-new-pass-manager -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -DUSE_AVX2 -mavx2 -DUSE_SSE41 -msse4.1 -DUSE_SSSE3 -mssse3 -DUSE_SSE2 -msse2 -DUSE_PEXT -mbmi2 -flto   -c -o evaluate_nnue.o nnue/evaluate_nnue.cpp
clang++ -Wall -Wcast-qual -fno-exceptions -std=c++17 -fprofile-instr-generate  -pedantic -Wextra -Wshadow -m64 -DUSE_PTHREADS -DNDEBUG -O3 -fexperimental-new-pass-manager -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -DUSE_AVX2 -mavx2 -DUSE_SSE41 -msse4.1 -DUSE_SSSE3 -mssse3 -DUSE_SSE2 -msse2 -DUSE_PEXT -mbmi2 -flto   -c -o half_ka_v2_hm.o nnue/features/half_ka_v2_hm.cpp
clang++ -o stockfish benchmark.o bitbase.o bitboard.o endgame.o evaluate.o main.o material.o misc.o movegen.o movepick.o pawns.o position.o psqt.o search.o thread.o timeman.o tt.o uci.o ucioption.o tune.o tbprobe.o evaluate_nnue.o half_ka_v2_hm.o -fprofile-instr-generate -latomic -m64 -lpthread -Wall -Wcast-qual -fno-exceptions -std=c++17 -fprofile-instr-generate  -pedantic -Wextra -Wshadow -m64 -DUSE_PTHREADS -DNDEBUG -O3 -fexperimental-new-pass-manager -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -DUSE_AVX2 -mavx2 -DUSE_SSE41 -msse4.1 -DUSE_SSSE3 -mssse3 -DUSE_SSE2 -msse2 -DUSE_PEXT -mbmi2 -flto
make[2]: Leaving directory '/content/Stockfish/src'
make[1]: Leaving directory '/content/Stockfish/src'

Step 2/4. Running benchmark for pgo-build ...
./stockfish bench 2>&1 | tail -n 4
===========================
Total time (ms) : 6550
Nodes searched  : 4550528
Nodes/second    : 694737

Step 3/4. Building optimized executable ...
make ARCH=x86-64-bmi2 COMP=clang objclean
make[1]: Entering directory '/content/Stockfish/src'
make[1]: Leaving directory '/content/Stockfish/src'
make ARCH=x86-64-bmi2 COMP=clang clang-profile-use
make[1]: Entering directory '/content/Stockfish/src'
llvm-profdata merge -output=stockfish.profdata *.profraw
/bin/sh: 1: llvm-profdata: not found
Makefile:923: recipe for target 'clang-profile-use' failed
make[1]: *** [clang-profile-use] Error 127
make[1]: Leaving directory '/content/Stockfish/src'
Makefile:784: recipe for target 'profile-build' failed
make: *** [profile-build] Error 2

[vdbergh]

llvm-profdata: not found

This seems to be the culprit. https://llvm.org/docs/CommandGuide/llvm-profdata.html

[ppigazzini]

on my ubuntu setup, the first clang version that works for profile build is version 10.

PGO seems to be still very broken on clang v13.0.1, I got 2 different error messages in a couple of day. PGO works fin with msys2 clang v13.0.0

[ppigazzini]

@vdbergh I sort out the issues:

• a clang version higher than the distro default installs /usr/bin/llvm-profdata-<version> (either with llvm.org script or ubuntu official package)
• clang-13 seems to use a profile data format incompatible with previous versions of llvm-profdata

colab script to run the worker with clang-13:

!wget https://apt.llvm.org/llvm.sh && bash llvm.sh 13
!update-alternatives --install /usr/bin/clang clang /usr/bin/clang-13 100 --slave /usr/bin/clang++ clang++ /usr/bin/clang++-13 --slave /usr/bin/llvm-profdata llvm-profdata /usr/bin/llvm-profdata-13
!git clone --single-branch --branch master https://github.com/glinscott/fishtest.git
!cd fishtest && git config user.email "you@example.com" && git config user.name "your_name"
#!cd fishtest && git pull --no-edit origin pull/1260/head
!python3 fishtest/worker/worker.py user01 user01 --host dfts-0.pigazzini.it --concurrency max --compiler clang++

colab script to build SF with clang-13:

!wget https://apt.llvm.org/llvm.sh && bash llvm.sh 13
!update-alternatives --install /usr/bin/clang clang /usr/bin/clang-13 100 --slave /usr/bin/clang++ clang++ /usr/bin/clang++-13 --slave /usr/bin/llvm-profdata llvm-profdata /usr/bin/llvm-profdata-13
!git clone --single-branch --branch master https://github.com/official-stockfish/Stockfish.git
!cd Stockfish/src && make -j 2 profile-build ARCH=x86-64-bmi2 COMP=clang && make strip && ./stockfish compiler && ./stockfish bench