search

ninja-build / ninja

a small build system with a focus on speed
https://ninja-build.org/
Apache License 2.0
11.32k stars 1.61k forks source link

Ninja randomly fails without logging anything #2223

Open Kaaserne opened 1 year ago

Kaaserne commented 1 year ago

For my work, I'm using ninja to build the project with vscode. Using Unix Makefiles, the build succeeds. With Ninja, it fails with the following output:

[build] [127/2406 4% :: 19.355] Building CXX object vptlib/techsupport/CMakeFiles/vptlib_tech.dir/motif_wrap.cxx.o [build] [128/2406 4% :: 19.408] Building C object vptlib/techsupport/CMakeFiles/vptlib_tech.dir/aggregate_messages.c.o [build] [129/2406 4% :: 19.424] Building CXX object vptlib/util/CMakeFiles/vptlib_util.dir/otabp_check.cxx.o [build] FAILED: vptlib/util/CMakeFiles/vptlib_util.dir/otabp_check.cxx.o [build] /usr/bin/ccache /opt/rh/devtoolset-7/root/usr/bin/g++ -I../vptlib/diag -I../vptlib/frm -I../vptlib/helper -I../vptlib/main -I../vptlib/xo -I../vptlib/sbs -I../vptlib/json -I../vptlib/sb -I../vptlib/sem -I../vptlib/lib -I../vptlib/util -I../vptlib/postbus -I../vptlib/prot -I../vptlib/ufw_lib -I../vptlib/log/include -I../vptprllib/alg -I../vptprllib/vptprl_basis -I../vptprllib/vptprl_alg -I../vptprllib/vptprl_alg_c -I../vptprllib/database -I../vptprllib/unittest -I../prl/vpt24a_alg -I../prl/vpt24a_tk_i5 -I../prl/vpt24x_alg -I../prl/vpt25g_alg -I../prl/vpt25d_alg -I../vptlib/util/../helper -isystem /opt/openvms_tools/include -isystem /opt/tibco/ems/8.4/include/tibems -isystem /usr/include/oracle/12.2/client64 -isystem /opt/motif-itr/include -isystem ../vptlib/techsupport_include -isystem /usr/include/X11 -m64 -g -D_LARGE_FILES -D_FILE_OFFSET_BITS=64 -DLINUX -D_REENTRANT -DSPINLOCK -DD_DLLEXP= -DD_DLLIMP= -DLIBR8_AVAILABLE -Dlinux -std=c++17 -g --coverage -O0 -fno-omit-frame-pointer -fprofile-dir="/var/prl/profile" -fpermissive -pipe -fuse-ld=gold -Wall -Wextra -Werror -Wnon-virtual-dtor -pedantic-errors -Wno-unused-parameter -Wno-deprecated -fPIC -fPIC -fdiagnostics-color=always -MD -MT vptlib/util/CMakeFiles/vptlib_util.dir/otabp_check.cxx.o -MF vptlib/util/CMakeFiles/vptlib_util.dir/otabp_check.cxx.o.d -o vptlib/util/CMakeFiles/vptlib_util.dir/otabp_check.cxx.o -c ../vptlib/util/otabp_check.cxx [build] [129/2406 4% :: 19.583] Building CXX object vptlib/sbs/CMakeFiles/vptlib_bewaak.dir/bewaak_logger.cxx.o [build] [129/2406 4% :: 19.597] Building CXX object vptlib/sbs/CMakeFiles/vptlib_bewaak.dir/bewaak_data_updater.cxx.o

If I build again, the error is elsewhere:

[build] [76/2278 2% :: 13.124] Building CXX object vptlib/frm/CMakeFiles/vptlib_frmtest.dir/smt_frm_edit_test.cxx.o [build] [77/2278 2% :: 13.126] Building C object vptlib/xo/CMakeFiles/vptlib_xo.dir/bbox.c.o [build] FAILED: vptlib/xo/CMakeFiles/vptlib_xo.dir/bbox.c.o [build] /usr/bin/ccache /opt/rh/devtoolset-7/root/usr/bin/gcc -I../vptlib/diag -I../vptlib/frm -I../vptlib/helper -I../vptlib/main -I../vptlib/xo -I../vptlib/sbs -I../vptlib/json -I../vptlib/sb -I../vptlib/sem -I../vptlib/lib -I../vptlib/util -I../vptlib/postbus -I../vptlib/prot -I../vptlib/ufw_lib -I../vptlib/log/include -I../vptprllib/alg -I../vptprllib/vptprl_basis -I../vptprllib/vptprl_alg -I../vptprllib/vptprl_alg_c -I../vptprllib/database -I../vptprllib/unittest -I../prl/vpt24a_alg -I../prl/vpt24a_tk_i5 -I../prl/vpt24x_alg -I../prl/vpt25g_alg -I../prl/vpt25d_alg -isystem /opt/openvms_tools/include -isystem /opt/tibco/ems/8.4/include/tibems -isystem /usr/include/oracle/12.2/client64 -isystem /opt/motif-itr/include -isystem ../vptlib/techsupport_include -m64 -g -D_LARGE_FILES -D_FILE_OFFSET_BITS=64 -DLINUX -D_REENTRANT -DSPINLOCK -DD_DLLEXP= -DD_DLLIMP= -DLIBR8_AVAILABLE -D_GNU_SOURCE -std=c11 -g --coverage -O0 -fno-omit-frame-pointer -pipe -fuse-ld=gold -fprofile-dir="/var/prl/profile" -Wall -Wextra -Werror -Wno-unused-parameter -fPIC -fdiagnostics-color=always -MD -MT vptlib/xo/CMakeFiles/vptlib_xo.dir/bbox.c.o -MF vptlib/xo/CMakeFiles/vptlib_xo.dir/bbox.c.o.d -o vptlib/xo/CMakeFiles/vptlib_xo.dir/bbox.c.o -c ../vptlib/xo/bbox.c [build] [77/2278 2% :: 13.253] Building C object vptlib/xo/CMakeFiles/vptlib_xotest.dir/xo_geometry_test_c.c.o [build] [77/2278 2% :: 13.373] Building C object vptlib/xo/CMakeFiles/vptlib_xotest.dir/xo_test_c.c.o [build] [77/2278 2% :: 13.387] Building C object vptlib/xo/CMakeFiles/vptlib_xo.dir/arc.c.o

I am running Ninja 1.10.2 on a RH Red Hat Enterprise Linux Server 7.9 (Maipo) (VM), using Windows.

FlykeSpice commented 5 months ago

I don't know if it's the same issue but when I try to compile mame huge codebase with ninja, it compiles fine the very first source files until it reaches this point:

[2/236] cxx ../../../../../android/obj/arm64/Release/src/emu/emumem_mview.o FAILED: ../../../../../android/obj/arm64/Release/src/emu/emumem_mview.o /home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -DPTR64=1 -DNDEBUG -DCRLF=2 -DLSB_FIRST -DFLACNO_DLL -DPUGIXML_HEADER_ONLY -DMAME_NOASM -DLUA_COMPAT_ALL -DLUA_COMPAT_5_1 -DLUA_COMPAT_5_2 -I"../../../../../../src/osd" -I"../../../../../../src/emu" -I"../../../../../../src/lib" -I"../../../../../../src/lib/util" -I"../../../../../../3rdparty" -I"../../../../../../3rdparty/sol2" -I"../../../../../generated/emu" -I"../../../../../generated/emu/layout" -I"../../../../../../3rdparty/asio/include" -I"../../../../../../3rdparty/expat/lib" -I"../../../../../../3rdparty/zlib" -I"../../../../../../3rdparty/flac/include" -I"../../../../../../3rdparty/libjpeg" -I"../../../../../../3rdparty/rapidjson/include" -I"../../../../../../3rdparty/pugixml/src" -std=c++17 -Werror -O3 -fno-strict-aliasing -Wno-unknown-pragmas -Wall -Wcast-align -Wformat-security -Wundef -Wwrite-strings -Wno-conversion -Wno-sign-compare -Wno-error=deprecated-declarations -fdiagnostics-show-note-include-stack -Wno-cast-align -Wno-constant-logical-operand -Wno-extern-c-compat -Wno-ignored-qualifiers -Wno-pragma-pack -Wno-tautological-compare -Wno-unknown-attributes -Wno-unknown-warning-option -Wno-unused-value -Wno-unused-const-variable -Wno-xor-used-as-pow -Wno-bitwise-instead-of-logical --gcc-toolchain=/home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64 --sysroot=/home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/sysroot -fpic -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -target aarch64-linux-android21 -Wno-error=undef -Wno-error=macro-redefined -std=c++17 -Woverloaded-virtual -Wvla -include ../../../../../android/obj/arm64/Release/src/emu/emu.h -MMD -MF ../../../../../android/obj/arm64/Release/src/emu/emumem_mview.o.d -c -o ../../../../../android/obj/arm64/Release/src/emu/emumem_mview.o ../../../../../../src/emu/emumem_mview.cpp Killed [3/236] cxx ../../../../../android/obj/arm64/Release/src/emu/devcb.o FAILED: ../../../../../android/obj/arm64/Release/src/emu/devcb.o /home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -DPTR64=1 -DNDEBUG -DCRLF=2 -DLSB_FIRST -DFLACNO_DLL -DPUGIXML_HEADER_ONLY -DMAME_NOASM -DLUA_COMPAT_ALL -DLUA_COMPAT_5_1 -DLUA_COMPAT_5_2 -I"../../../../../../src/osd" -I"../../../../../../src/emu" -I"../../../../../../src/lib" -I"../../../../../../src/lib/util" -I"../../../../../../3rdparty" -I"../../../../../../3rdparty/sol2" -I"../../../../../generated/emu" -I"../../../../../generated/emu/layout" -I"../../../../../../3rdparty/asio/include" -I"../../../../../../3rdparty/expat/lib" -I"../../../../../../3rdparty/zlib" -I"../../../../../../3rdparty/flac/include" -I"../../../../../../3rdparty/libjpeg" -I"../../../../../../3rdparty/rapidjson/include" -I"../../../../../../3rdparty/pugixml/src" -std=c++17 -Werror -O3 -fno-strict-aliasing -Wno-unknown-pragmas -Wall -Wcast-align -Wformat-security -Wundef -Wwrite-strings -Wno-conversion -Wno-sign-compare -Wno-error=deprecated-declarations -fdiagnostics-show-note-include-stack -Wno-cast-align -Wno-constant-logical-operand -Wno-extern-c-compat -Wno-ignored-qualifiers -Wno-pragma-pack -Wno-tautological-compare -Wno-unknown-attributes -Wno-unknown-warning-option -Wno-unused-value -Wno-unused-const-variable -Wno-xor-used-as-pow -Wno-bitwise-instead-of-logical --gcc-toolchain=/home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64 --sysroot=/home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/sysroot -fpic -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -target aarch64-linux-android21 -Wno-error=undef -Wno-error=macro-redefined -std=c++17 -Woverloaded-virtual -Wvla -include ../../../../../android/obj/arm64/Release/src/emu/emu.h -MMD -MF ../../../../../android/obj/arm64/Release/src/emu/devcb.o.d -c -o ../../../../../android/obj/arm64/Release/src/emu/devcb.o ../../../../../../src/emu/devcb.cpp Killed [4/236] cxx ../../../../../android/obj/arm64/Release/src/emu/emumem_hedr1.o FAILED: ../../../../../android/obj/arm64/Release/src/emu/emumem_hedr1.o /home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -DPTR64=1 -DNDEBUG -DCRLF=2 -DLSB_FIRST -DFLACNO_DLL -DPUGIXML_HEADER_ONLY -DMAME_NOASM -DLUA_COMPAT_ALL -DLUA_COMPAT_5_1 -DLUA_COMPAT_5_2 -I"../../../../../../src/osd" -I"../../../../../../src/emu" -I"../../../../../../src/lib" -I"../../../../../../src/lib/util" -I"../../../../../../3rdparty" -I"../../../../../../3rdparty/sol2" -I"../../../../../generated/emu" -I"../../../../../generated/emu/layout" -I"../../../../../../3rdparty/asio/include" -I"../../../../../../3rdparty/expat/lib" -I"../../../../../../3rdparty/zlib" -I"../../../../../../3rdparty/flac/include" -I"../../../../../../3rdparty/libjpeg" -I"../../../../../../3rdparty/rapidjson/include" -I"../../../../../../3rdparty/pugixml/src" -std=c++17 -Werror -O3 -fno-strict-aliasing -Wno-unknown-pragmas -Wall -Wcast-align -Wformat-security -Wundef -Wwrite-strings -Wno-conversion -Wno-sign-compare -Wno-error=deprecated-declarations -fdiagnostics-show-note-include-stack -Wno-cast-align -Wno-constant-logical-operand -Wno-extern-c-compat -Wno-ignored-qualifiers -Wno-pragma-pack -Wno-tautological-compare -Wno-unknown-attributes -Wno-unknown-warning-option -Wno-unused-value -Wno-unused-const-variable -Wno-xor-used-as-pow -Wno-bitwise-instead-of-logical --gcc-toolchain=/home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64 --sysroot=/home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/sysroot -fpic -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -target aarch64-linux-android21 -Wno-error=undef -Wno-error=macro-redefined -std=c++17 -Woverloaded-virtual -Wvla -include ../../../../../android/obj/arm64/Release/src/emu/emu.h -MMD -MF ../../../../../android/obj/arm64/Release/src/emu/emumem_hedr1.o.d -c -o ../../../../../android/obj/arm64/Release/src/emu/emumem_hedr1.o ../../../../../../src/emu/emumem_hedr1.cpp Killed [5/236] cxx ../../../../../android/obj/arm64/Release/src/emu/emumem_hedr2.o FAILED: ../../../../../android/obj/arm64/Release/src/emu/emumem_hedr2.o /home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -DPTR64=1 -DNDEBUG -DCRLF=2 -DLSB_FIRST -DFLACNO_DLL -DPUGIXML_HEADER_ONLY -DMAME_NOASM -DLUA_COMPAT_ALL -DLUA_COMPAT_5_1 -DLUA_COMPAT_5_2 -I"../../../../../../src/osd" -I"../../../../../../src/emu" -I"../../../../../../src/lib" -I"../../../../../../src/lib/util" -I"../../../../../../3rdparty" -I"../../../../../../3rdparty/sol2" -I"../../../../../generated/emu" -I"../../../../../generated/emu/layout" -I"../../../../../../3rdparty/asio/include" -I"../../../../../../3rdparty/expat/lib" -I"../../../../../../3rdparty/zlib" -I"../../../../../../3rdparty/flac/include" -I"../../../../../../3rdparty/libjpeg" -I"../../../../../../3rdparty/rapidjson/include" -I"../../../../../../3rdparty/pugixml/src" -std=c++17 -Werror -O3 -fno-strict-aliasing -Wno-unknown-pragmas -Wall -Wcast-align -Wformat-security -Wundef -Wwrite-strings -Wno-conversion -Wno-sign-compare -Wno-error=deprecated-declarations -fdiagnostics-show-note-include-stack -Wno-cast-align -Wno-constant-logical-operand -Wno-extern-c-compat -Wno-ignored-qualifiers -Wno-pragma-pack -Wno-tautological-compare -Wno-unknown-attributes -Wno-unknown-warning-option -Wno-unused-value -Wno-unused-const-variable -Wno-xor-used-as-pow -Wno-bitwise-instead-of-logical --gcc-toolchain=/home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64 --sysroot=/home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/sysroot -fpic -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -target aarch64-linux-android21 -Wno-error=undef -Wno-error=macro-redefined -std=c++17 -Woverloaded-virtual -Wvla -include ../../../../../android/obj/arm64/Release/src/emu/emu.h -MMD -MF ../../../../../android/obj/arm64/Release/src/emu/emumem_hedr2.o.d -c -o ../../../../../android/obj/arm64/Release/src/emu/emumem_hedr2.o ../../../../../../src/emu/emumem_hedr2.cpp Killed [6/236] cxx ../../../../../android/obj/arm64/Release/src/emu/emumem_aspace.o FAILED: ../../../../../android/obj/arm64/Release/src/emu/emumem_aspace.o /home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -DPTR64=1 -DNDEBUG -DCRLF=2 -DLSB_FIRST -DFLAC__NO_DLL -DPUGIXML_HEADER_ONLY -DMAME_NOASM -DLUA_COMPAT_ALL -DLUA_COMPAT_5_1 -DLUA_COMPAT_5_2 -I"../../../../../../src/osd" -I"../../../../../../src/emu" -I"../../../../../../src/lib" -I"../../../../../../src/lib/util" -I"../../../../../../3rdparty" -I"../../../../../../3rdparty/sol2" -I"../../../../../generated/emu" -I"../../../../../generated/emu/layout" -I"../../../../../../3rdparty/asio/include" -I"../../../../../../3rdparty/expat/lib" -I"../../../../../../3rdparty/zlib" -I"../../../../../../3rdparty/flac/include" -I"../../../../../../3rdparty/libjpeg" -I"../../../../../../3rdparty/rapidjson/include" -I"../../../../../../3rdparty/pugixml/src" -std=c++17 -Werror -O3 -fno-strict-aliasing -Wno-unknown-pragmas -Wall -Wcast-align -Wformat-security -Wundef -Wwrite-strings -Wno-conversion -Wno-sign-compare -Wno-error=deprecated-declarations -fdiagnostics-show-note-include-stack -Wno-cast-align -Wno-constant-logical-operand -Wno-extern-c-compat -Wno-ignored-qualifiers -Wno-pragma-pack -Wno-tautological-compare -Wno-unknown-attributes -Wno-unknown-warning-option -Wno-unused-value -Wno-unused-const-variable -Wno-xor-used-as-pow -Wno-bitwise-instead-of-logical --gcc-toolchain=/home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64 --sysroot=/home/flyke/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/sysroot -fpic -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -target aarch64-linux-android21 -Wno-error=undef -Wno-error=macro-redefined -std=c++17 -Woverloaded-virtual -Wvla -include ../../../../../android/obj/arm64/Release/src/emu/emu.h -MMD -MF ../../../../../android/obj/arm64/Release/src/emu/emumem_aspace.o.d -c -o ../../../../../android/obj/arm64/Release/src/emu/emumem_aspace.o ../../../../../../src/emu/emumem_aspace.cpp Killed [11/236] cxx ../../../../../android/obj/arm64/Release/src/emu/emumem_hedr3.o ninja: build stopped: subcommand failed. make[1]: [Makefile:13: all] Error 1 make: [makefile:1236: android-arm64] Error 2

When I manually invoke the gcc compilation command from the command line, it compiles fine.

ArtistArthur commented 2 months ago

#

When I manually invoke the gcc compilation command from the command line, it compiles fine.

I encountered the same problem

digit-google commented 2 months ago

A simple Killed failure message is generally the symptom of the system killing processes when running out of memory. What happens if you try to run Ninja with the -j1 flag? If the build succeeds, it is likely that your commands are using too much RAM for the machine you have.

By default Ninja tries to run as many parallel jobs as cores on your machine (roughly speaking). If you have many cores, but not a large amount of RAM, this can easily produce this situation. A work-around is to enforce a lower parallel count (e.g. using -j4).

Apart from that, there is not enough details in your descriptions to know what's happening. A simple way to reproduce what you see whould be helpful.