Snapshot build (2022.01.15 ~ 2022.01.17) failed

[Atry]

What is interesting is that the first failure log is build-failure-2022.01.14-debian-11-bullseye-.cloud-init-output.log, while the snapshot build 2022.01.14 was not failed according to https://hhvm.com/api/build-status/2022.01.14

Due to #264 ?

[Atry]

The error in build-failure-2022.01.14-debian-11-bullseye-.cloud-init-output.log is different from build-failure-2022.01.17-debian-11-bullseye-.cloud-init-output.log.

---

build-failure-2022.01.14-debian-11-bullseye-.cloud-init-output.log

cd /tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu/hphp/parser && /usr/bin/c++ -DDEFAULT_CONFIG_DIR=\"/etc/hhvm/\" -DENABLE_FASTCGI=1 -DHAVE_CURL_MULTI_WAIT -DHAVE_ELF_GETSHDRSTRNDX -DHAVE_FT2BUILD -DHAVE_INTTYPES_H -DHAVE_LIBDL -DHAVE_LIBSODIUM=1 -DHAVE_SYSTEM_TZDATA -DHHVM -DHHVM_DYNAMIC_EXTENSION_DIR=\"/opt/hhvm/2022.01.14/lib/x86_64-linux-gnu/hhvm/extensions\" -DHPHP_OSS=1 -DLIBDWARF_CONST_NAME -DNDEBUG -DNO_TCMALLOC=1 -DOPENSSL_NO_RAND_EGD -DPACKAGE=hhvm -DPACKAGE_VERSION=Release -DTHRIFT_MUTEX_EMULATE_PTHREAD_TIMEDLOCK -DUSE_CMAKE -DUSE_EDITLINE -DUSE_JEMALLOC=1 -DU_USING_ICU_NAMESPACE=1 -D_GNU_SOURCE -D_PTHREADS=1 -D_REENTRANT=1 -D__STDC_FORMAT_MACROS -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu -I/usr/include/libxml2 -I/usr/include/libdwarf -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/hphp -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/forks/libmbfl -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/forks/libmbfl/mbfl -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/forks/libmbfl/filters -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/fatal -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/libafdt/src/src -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/fastlz/src -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14 -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu/third-party/brotli/bundled_brotli-prefix/include -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu/third-party/folly/bundled_folly-prefix/include -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/folly/include -I/usr/include/libiberty -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu/third-party/double-conversion/bundled_double-conversion-prefix/include -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu/third-party/fmt/bundled_fmt-prefix/include -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu/third-party/zstd/bundled_zstd-prefix/include -g -O2 -ffile-prefix-map=/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wno-error=array-bounds -Wno-error=switch -Wno-attributes -Wno-deprecated -Wno-invalid-offsetof -Wno-register -Wno-sign-compare -Wno-strict-aliasing -Wno-unused-function -Wno-unused-local-typedefs -Wno-unused-result -Wno-write-strings -Wno-deprecated-declarations -Wno-maybe-uninitialized -Wno-bool-compare -ffunction-sections -fno-delete-null-pointer-checks -DFOLLY_HAVE_MALLOC_H -std=gnu++1z -fno-omit-frame-pointer -fno-operator-names -Wall -Woverloaded-virtual -Werror=format-security -fdata-sections -fno-gcse -fno-canonical-system-headers -Wvla -Wno-misleading-indentation -mcrc32 -Wno-error=array-bounds -Wno-error=switch -Wno-attributes -Wno-deprecated -Wno-invalid-offsetof -Wno-register -Wno-sign-compare -Wno-strict-aliasing -Wno-unused-function -Wno-unused-local-typedefs -Wno-unused-result -Wno-write-strings -Wno-deprecated-declarations -Wno-maybe-uninitialized -Wno-bool-compare -ffunction-sections -fno-delete-null-pointer-checks -DFOLLY_HAVE_MALLOC_H -std=gnu++1z -fno-omit-frame-pointer -fno-operator-names -Wall -Woverloaded-virtual -Werror=format-security -fdata-sections -fno-gcse -fno-canonical-system-headers -Wvla -Wno-misleading-indentation -mcrc32 -O2 -g -DNDEBUG -o CMakeFiles/hphp_parser.dir/parser.cpp.o -c /tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/hphp/parser/parser.cpp
make[5]: *** [Makefile.dune:58: build-hack] Error 1

---

build-failure-2022.01.17-debian-11-bullseye-.cloud-init-output.log

cd /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/cpp2 && /usr/bin/c++ -DFMT_LOCALE -DTHRIFT_HAVE_LIBSNAPPY=0 -I/tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift -I/tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/zstd/bundled_zstd-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/fmt/bundled_fmt-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/double-conversion/bundled_double-conversion-prefix/include -isystem /usr/include/libdwarf -isystem /usr/include/libiberty -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/folly/bundled_folly-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/wangle/bundled_wangle-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/fizz/bundled_fizz-prefix/include -I/usr/include -O2 -g -DNDEBUG -std=c++17 -o CMakeFiles/thriftprotocol.dir/__/thrift/gen-cpp2/reflection_types.cpp.o -c /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/thrift/gen-cpp2/reflection_types.cpp
The bug is not reproducible, so it is likely a hardware or OS problem.
make[7]: *** [thrift/lib/cpp2/CMakeFiles/thriftprotocol.dir/build.make:173: thrift/lib/cpp2/CMakeFiles/thriftprotocol.dir/protocol/TableBasedSerializer.cpp.o] Error 1

[Atry]

I successfully executed the failure command after loading the failure image.

cd /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/cpp2 && /usr/bin/c++ -DFMT_LOCALE -DTHRIFT_HAVE_LIBSNAPPY=0 -I/tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift -I/tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/zstd/bundled_zstd-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/fmt/bundled_fmt-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/double-conversion/bundled_double-conversion-prefix/include -isystem /usr/include/libdwarf -isystem /usr/include/libiberty -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/folly/bundled_folly-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/wangle/bundled_wangle-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/fizz/bundled_fizz-prefix/include -I/usr/include -O2 -g -DNDEBUG -std=c++17 -o CMakeFiles/thriftprotocol.dir/__/thrift/gen-cpp2/reflection_types.cpp.o -c /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/thrift/gen-cpp2/reflection_types.cpp

So this is not a reproducible issue.

[fredemmott]

What is interesting is that the first failure log is build-failure-2022.01.14-debian-11-bullseye-.cloud-init-output.log, while the snapshot build 2022.01.14 was not failed according to https://hhvm.com/api/build-status/2022.01.14

We automatically retry on failure; the failure log shows an attempt failed, but we then tried again on another machine and it worked

It's worth checking the dmesg logs etc to see if there's anything correlating to the non-reproducible error (it does happen occassionally, especially on 11, but without a reliable repro)

But a single attempt failing isn't necessarily an issue worth investigating

[Atry]

I killed the hanging EC2 instance and terminated all running step functions. However, the 2022.01.18 build still failed due to gcc crashing.

        "debian-11-bullseye": {
          "MakeBinaryPackage": {
            "failure": {
              "Error": null,
              "Cause": "{\"ec2\":\"i-072ac8413970a007d\",\"time_sec\":\"1824\"}"
            }
          }
        },

cd /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/cpp2 && /usr/bin/c++ -DFMT_LOCALE -DTHRIFT_HAVE_LIBSNAPPY=0 -I/tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift -I/tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/zstd/bundled_zstd-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/fmt/bundled_fmt-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/double-conversion/bundled_double-conversion-prefix/include -isystem /usr/include/libdwarf -isystem /usr/include/libiberty -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/folly/bundled_folly-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/wangle/bundled_wangle-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/fizz/bundled_fizz-prefix/include -I/usr/include -O2 -g -DNDEBUG -std=c++17 -o CMakeFiles/thriftprotocol.dir/__/thrift/gen-cpp2/reflection_types.cpp.o -c /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/thrift/gen-cpp2/reflection_types.cpp
Preprocessed source stored into /tmp/cc8x2P1d.out file, please attach this to your bugreport.
=== BEGIN GCC DUMP ===
139073: // Target: x86_64-linux-gnu
139073: // Configured with: ../src/configure -v --with-pkgversion='Debian 10.2.1-6' --with-bugurl=file:///usr/share/doc/gcc-10/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-10 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-gcn/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-mutex
139073: // Thread model: posix
139073: // Supported LTO compression algorithms: zlib zstd
139073: // gcc version 10.2.1 20210110 (Debian 10.2.1-6) 
139073: // 
139073: // /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp: In substitution of 'template<class T> constexpr T apache::thrift::detail::identity(T) [with T = <missing>]':
139073: // /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp:37:1:   required from here
139073: // /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp:32:67: internal compiler error: Segmentation fault
139073: //    32 |       reinterpret_cast<VoidFuncPtr>(identity(set<Type, ThriftType>)),  \
139073: //       |                                                                   ^
139073: // /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp:37:1: note: in expansion of macro 'THRIFT_DEFINE_PRIMITIVE_TYPE_TO_INFO'
139073: //    37 | THRIFT_DEFINE_PRIMITIVE_TYPE_TO_INFO(
139073: //       | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
139073: // Please submit a full bug report,
139073: // with preprocessed source if appropriate.
139073: // See <file:///usr/share/doc/gcc-10/README.Bugs> for instructions.
139073: 
139073: // /usr/lib/gcc/x86_64-linux-gnu/10/cc1plus -quiet -I /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift -I /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build -I /usr/include -imultiarch x86_64-linux-gnu -D_GNU_SOURCE -D FMT_LOCALE -D THRIFT_HAVE_LIBSNAPPY=0 -D NDEBUG -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/zstd/bundled_zstd-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/fmt/bundled_fmt-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/double-conversion/bundled_double-conversion-prefix/include -isystem /usr/include/libdwarf -isystem /usr/include/libiberty -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/folly/bundled_folly-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/wangle/bundled_wangle-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/fizz/bundled_fizz-prefix/include /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp -quiet -dumpbase TableBasedSerializer.cpp -mtune=generic -march=x86-64 -auxbase-strip CMakeFiles/thriftprotocol.dir/protocol/TableBasedSerializer.cpp.o -g -O2 -std=c++17 -fasynchronous-unwind-tables -o - -frandom-seed=0 -fdump-noaddr
139073: 
139073: # 1 "/tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp"
139073: # 1 "/tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/cpp2//"
139073: # 1 "<built-in>"
139073: # 1 "<command-line>"
139073: # 1 "/usr/include/stdc-predef.h" 1 3 4
139073: # 1 "<command-line>" 2
139073: # 1 "/tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp"

[fredemmott]

let's kick off another build just for 11, in case it's intermittent (to hopefully unblock this weeks' release), but also try and pull up the logs for what happened there. >2h is ok, > 3h is not.

[Atry]

Sorry, it's not a timeout. It's due to gcc crash.

[Atry]

Why didn't the step function retry?

[fredemmott]

Is it possible that the step function did retry, but GCC crashed every time?

A GCC crash is not neccessarily a GCC bug - e.g. it could be the kernel killing it due to running out of memory. Needs correlating with syslog/dmesg

[Atry]

Is it possible that the step function did retry, but GCC crashed every time?

I think so. The previous failure log file build-failure-2022.01.19-debian-11-bullseye-.cloud-init-output.log in S3 was overridden by the last retry, because there is a bug in file name generation, missing the EC2 instance name part.

[Atry]

I triggered bin/build-on-aws 2022.01.19 debian-11-bullseye MakeBinaryPackage and the step functions succeeded. I am not sure why the previous retry did not succeed.

[Atry]

Now triggering bin/build-on-aws 2022.01.19 debian-11-bullseye PublishBinaryPackages PublishDockerImages