clang-10 crash on ubuntu 20.10

[llvmbot]

Bugzilla Link	48789
Version	10.0
OS	Linux
Reporter	LLVM Bugzilla Contributor
CC	@DimitryAndric,@zygoloid

Extended Description

See details (with reproducer) here

https://bugs.launchpad.net/ubuntu/+source/llvm-toolchain-10/+bug/1910932

clang-11 crashes similarly:

https://bugs.launchpad.net/ubuntu/+source/llvm-toolchain-11/+bug/1910863

[llvmbot]

actually we use clang to compile and it requires 8 to 12 Go of RAM to compile some of our codes. boost + template + expression templates + ...

basically when we compile the codes we compile the library that come with it

However we never got these issues with clang up to 11 included, only on 20.10 and recently in docker/focal

[DimitryAndric]

There's something in this source that causes huge RAM usage during compilation. Even when I tune down -template-depth from 1024 to e.g. 256, it still easily gets to >16GiB of RAM while compiling, and at some point it eats all my swap and dies.

Must be Boost. :-)

[Endilll]

The second reproducer from Clang 11 is still crashing Clang 16 with assertions, so this is worth reducing. But it takes about 8 GB of RAM and 90 seconds, so for 72 MB of sources we're talking about not even hours, but days of C-Reduce working.

[Endilll]

I forgot to add that it's crashing in preprocessor, and Boost.Preprocessor is indeed involved. That's why it's so painful.

[llvmbot]

@llvm/issue-subscribers-clang-frontend

[Endilll]

I started reducing this. Let's see how much it would take.

[Endilll]

Clang 16 crash:

In file included from /home/prudhomm/Devel/feelpp.groovy/feelpp/quickstart/qs_adr.cpp:25:
In file included from /home/prudhomm/Devel/feelpp.groovy/feelpp/feel/../feel/feel.hpp:33:
In file included from /home/prudhomm/Devel/feelpp.groovy/feelpp/feel/../feel/feelcore/core.hpp:27:
In file included from /home/prudhomm/Devel/feelpp.groovy/feelpp/feel/../feel/feelcore/environment.hpp:41:
In file included from /home/prudhomm/Devel/feelpp.groovy/feelpp/feel/../feel/feelcore/feel.hpp:138:
In file included from /home/prudhomm/Devel/feelpp.groovy/feelpp/feel/../feel/feelcore/serialization.hpp:34:
In file included from /home/prudhomm/Devel/feelpp.groovy/feelpp/contrib/eigen/Eigen/Core:22:
In file included from /home/prudhomm/Devel/feelpp.groovy/feelpp/contrib/eigen/Eigen/src/Core/util/ConfigureVectorization.h:342:
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2224:19: error: use of undeclared identifier '__builtin_ia32_paddsb128'; did you mean '__builtin_ia32_phaddsw128'?
  return (__m128i)__builtin_ia32_paddsb128((__v16qi)__a, (__v16qi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2224:19: note: '__builtin_ia32_phaddsw128' declared here
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2246:19: error: use of undeclared identifier '__builtin_ia32_paddsw128'; did you mean '__builtin_ia32_phaddsw128'?
  return (__m128i)__builtin_ia32_paddsw128((__v8hi)__a, (__v8hi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2224:19: note: '__builtin_ia32_phaddsw128' declared here
  return (__m128i)__builtin_ia32_paddsb128((__v16qi)__a, (__v16qi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2267:19: error: use of undeclared identifier '__builtin_ia32_paddusb128'
  return (__m128i)__builtin_ia32_paddusb128((__v16qi)__a, (__v16qi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2288:19: error: use of undeclared identifier '__builtin_ia32_paddusw128'
  return (__m128i)__builtin_ia32_paddusw128((__v8hi)__a, (__v8hi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2374:19: error: use of undeclared identifier '__builtin_ia32_pmaxsw128'
  return (__m128i)__builtin_ia32_pmaxsw128((__v8hi)__a, (__v8hi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2394:19: error: use of undeclared identifier '__builtin_ia32_pmaxub128'
  return (__m128i)__builtin_ia32_pmaxub128((__v16qi)__a, (__v16qi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2414:19: error: use of undeclared identifier '__builtin_ia32_pminsw128'
  return (__m128i)__builtin_ia32_pminsw128((__v8hi)__a, (__v8hi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2434:19: error: use of undeclared identifier '__builtin_ia32_pminub128'
  return (__m128i)__builtin_ia32_pminub128((__v16qi)__a, (__v16qi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2666:19: error: use of undeclared identifier '__builtin_ia32_psubsb128'; did you mean '__builtin_ia32_phsubsw128'?
  return (__m128i)__builtin_ia32_psubsb128((__v16qi)__a, (__v16qi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2666:19: note: '__builtin_ia32_phsubsw128' declared here
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2687:19: error: use of undeclared identifier '__builtin_ia32_psubsw128'; did you mean '__builtin_ia32_phsubsw128'?
  return (__m128i)__builtin_ia32_psubsw128((__v8hi)__a, (__v8hi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2666:19: note: '__builtin_ia32_phsubsw128' declared here
  return (__m128i)__builtin_ia32_psubsb128((__v16qi)__a, (__v16qi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2707:19: error: use of undeclared identifier '__builtin_ia32_psubusb128'
  return (__m128i)__builtin_ia32_psubusb128((__v16qi)__a, (__v16qi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2727:19: error: use of undeclared identifier '__builtin_ia32_psubusw128'; did you mean '__builtin_ia32_phsubsw128'?
  return (__m128i)__builtin_ia32_psubusw128((__v8hi)__a, (__v8hi)__b);
                  ^
/usr/lib/llvm-11/lib/clang/11.0.0/include/emmintrin.h:2666:19: note: '__builtin_ia32_phsubsw128' declared here
  return (__m128i)__builtin_ia32_psubsb128((__v16qi)__a, (__v16qi)__b);
                  ^

clang: /root/llvm-project/clang/lib/Basic/SourceManager.cpp:676:
clang::SourceLocation clang::SourceManager::createExpansionLocImpl(const clang::SrcMgr::ExpansionInfo&, unsigned int, int, clang::SourceLocation::UIntTy):
Assertion `NextLocalOffset + Length + 1 > NextLocalOffset && NextLocalOffset + Length + 1 <= CurrentLoadedOffset && "Ran out of source locations!"' failed.

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /opt/compiler-explorer/clang-assertions-16.0.0/bin/clang -cc1 -triple x86_64-pc-linux-gnu -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name qs_adr.cpp -mrelocation-model static -mframe-pointer=none -fmath-errno -fno-rounding-math -mconstructor-aliases -target-cpu x86-64 -debugger-tuning=gdb -D BOOST_ALL_NO_LIB -D BOOST_ATOMIC_DYN_LINK -D BOOST_DATE_TIME_DYN_LINK -D BOOST_FILESYSTEM_DYN_LINK -D BOOST_FILESYSTEM_VERSION=3 -D BOOST_IOSTREAMS_DYN_LINK -D BOOST_MPI_DYN_LINK -D BOOST_OPTIONAL_USE_OLD_DEFINITION_OF_NONE=1 -D BOOST_PARAMETER_MAX_ARITY=25 -D BOOST_PROGRAM_OPTIONS_DYN_LINK -D BOOST_REGEX_DYN_LINK -D BOOST_RESULT_OF_USE_TR1=1 -D BOOST_SERIALIZATION_DYN_LINK -D BOOST_SYSTEM_DYN_LINK -D BOOST_TEST_DYN_LINK -D BOOST_THREAD_DYN_LINK -D BOOST_UBLAS_SHALLOW_ARRAY_ADAPTOR=1 -D BOOST_UNIT_TEST_FRAMEWORK_DYN_LINK -D FEELPP_DIM=2 -D FEELPP_HAS_ANN -D FEELPP_HAS_ANN_H -D FEELPP_HAS_BOOST -D FEELPP_HAS_BOOST_HANA -D FEELPP_HAS_BSONCXX -D FEELPP_HAS_CPR -D FEELPP_HAS_DLFCN_H -D FEELPP_HAS_DLOPEN -D FEELPP_HAS_EIGEN3 -D FEELPP_HAS_FFTW -D FEELPP_HAS_FMILIB -D FEELPP_HAS_GFLAGS -D FEELPP_HAS_GINAC -D FEELPP_HAS_GLOG -D FEELPP_HAS_GLPK -D FEELPP_HAS_GLPK_H -D FEELPP_HAS_GMSH -D FEELPP_HAS_GMSH_API -D FEELPP_HAS_GMSH_H -D FEELPP_HAS_GMSH_LIBRARY -D FEELPP_HAS_HDF5 -D FEELPP_HAS_INDICATORS -D FEELPP_HAS_KWSYS -D FEELPP_HAS_LIBCURL -D FEELPP_HAS_LIBXML2 -D FEELPP_HAS_METIS -D FEELPP_HAS_MMG -D FEELPP_HAS_MONGOCXX -D FEELPP_HAS_MPI -D FEELPP_HAS_MPIIO -D FEELPP_HAS_MPI_H -D FEELPP_HAS_NLOPT -D FEELPP_HAS_PARMMG -D FEELPP_HAS_PETSC -D FEELPP_HAS_PETSC_H -D FEELPP_HAS_PYBIND11 -D FEELPP_HAS_PYTHON -D FEELPP_HAS_QUICKSTART -D FEELPP_HAS_RANGE_V3 -D FEELPP_HAS_SIMPLE_WEB_SERVER -D FEELPP_HAS_SLEPC -D FEELPP_HAS_TABULATE -D GFLAGS_IS_A_DLL=0 -D GMSH_API_VERSION_MAJOR=4 -D GMSH_API_VERSION_MINOR=6 -D GMSH_EXECUTABLE=/usr/bin/gmsh -D GOOGLE_GLOG_DLL_DECL= -D GOOGLE_GLOG_DLL_DECL_FOR_UNITTESTS= -D HAVE_LIBDL -D HAVE_OPENSSL -D IN_GINAC -D NLOPT_DLL -D NDEBUG -O3 -Wall -Wno-unused -Wno-sign-compare -Wno-deprecated-register -std=gnu++17 -fdeprecated-macro -ftemplate-depth 1024 -ferror-limit 19 -ftemplate-backtrace-limit 0 -fvisibility-inlines-hidden -pthread -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -vectorize-loops -vectorize-slp -faddrsig -x c++ qs_adr-d57687.cpp
1.      /home/prudhomm/Devel/feelpp.groovy/feelpp/feel/../feel/feelfilters/loadmesh.hpp:56:1 <Spelling=/usr/include/boost/parameter/aux_/preprocessor/impl/function_dispatch_layer.hpp:19:5>: current parser token 'typename'
2.      /home/prudhomm/Devel/feelpp.groovy/feelpp/feel/../feel/feelfilters/loadmesh.hpp:44:1: parsing namespace 'Feel'
 #0 0x0000000003669fe8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x3669fe8)
 #1 0x0000000003667aec SignalHandler(int) Signals.cpp:0:0
 #2 0x00007f4be2f7b510 (/lib/x86_64-linux-gnu/libc.so.6+0x3c510)
 #3 0x00007f4be2fc90fc __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
 #4 0x00007f4be2f7b472 raise ./signal/../sysdeps/posix/raise.c:27:6
 #5 0x00007f4be2f654b2 abort ./stdlib/abort.c:81:7
 #6 0x00007f4be2f653d5 _nl_load_domain ./intl/loadmsgcat.c:1177:9
 #7 0x00007f4be2f743a2 (/lib/x86_64-linux-gnu/libc.so.6+0x353a2)
 #8 0x00000000039f2aa6 clang::SourceManager::createExpansionLocImpl(clang::SrcMgr::ExpansionInfo const&, unsigned int, int, unsigned int) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x39f2aa6)
 #9 0x00000000039f2bfb clang::SourceManager::createMacroArgExpansionLoc(clang::SourceLocation, clang::SourceLocation, unsigned int) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x39f2bfb)
#10 0x000000000746d489 clang::TokenLexer::updateLocForMacroArgTokens(clang::SourceLocation, clang::Token*, clang::Token*) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x746d489)
#11 0x000000000746ef15 clang::TokenLexer::ExpandFunctionArguments() (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x746ef15)
#12 0x000000000746f69f clang::TokenLexer::Init(clang::Token&, clang::SourceLocation, clang::MacroInfo*, clang::MacroArgs*) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x746f69f)
#13 0x0000000007435d21 clang::Preprocessor::EnterMacro(clang::Token&, clang::SourceLocation, clang::MacroInfo*, clang::MacroArgs*) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x7435d21)
#14 0x0000000007447c16 clang::Preprocessor::HandleMacroExpandedIdentifier(clang::Token&, clang::MacroDefinition const&) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x7447c16)
#15 0x000000000746466a clang::Preprocessor::HandleIdentifier(clang::Token&) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x746466a)
#16 0x0000000007466fb7 clang::Preprocessor::Lex(clang::Token&) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x7466fb7)
#17 0x00000000074181f6 clang::Preprocessor::PeekAhead(unsigned int) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x74181f6)
#18 0x0000000005ce01ba clang::Parser::isStartOfTemplateTypeParameter() (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x5ce01ba)
#19 0x0000000005ce103f clang::Parser::ParseTemplateParameter(unsigned int, unsigned int) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x5ce103f)
#20 0x0000000005ce1ede clang::Parser::ParseTemplateParameterList(unsigned int, llvm::SmallVectorImpl<clang::NamedDecl*>&) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x5ce1ede)
#21 0x0000000005ce266d clang::Parser::ParseTemplateParameters(clang::Parser::MultiParseScope&, unsigned int, llvm::SmallVectorImpl<clang::NamedDecl*>&, clang::SourceLocation&, clang::SourceLocation&) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x5ce266d)
#22 0x0000000005ce4f9e clang::Parser::ParseTemplateDeclarationOrSpecialization(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::AccessSpecifier) (.part.0) ParseTemplate.cpp:0:0
#23 0x0000000005ce5383 clang::Parser::ParseDeclarationStartingWithTemplate(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::AccessSpecifier) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x5ce5383)
#24 0x0000000005c26459 clang::Parser::ParseDeclaration(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::ParsedAttributes&, clang::SourceLocation*) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x5c26459)
#25 0x0000000005bf3d83 clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x5bf3d83)
#26 0x0000000005c3a2ab clang::Parser::ParseInnerNamespace(llvm::SmallVector<clang::Parser::InnerNamespaceInfo, 4u> const&, unsigned int, clang::SourceLocation&, clang::ParsedAttributes&, clang::BalancedDelimiterTracker&) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x5c3a2ab)
#27 0x0000000005c3e69f clang::Parser::ParseNamespace(clang::DeclaratorContext, clang::SourceLocation&, clang::SourceLocation) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x5c3e69f)
#28 0x0000000005c264ab clang::Parser::ParseDeclaration(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::ParsedAttributes&, clang::SourceLocation*) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x5c264ab)
#29 0x0000000005bf3d83 clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x5bf3d83)
#30 0x0000000005bf5274 clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, clang::Sema::ModuleImportState&) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x5bf5274)
#31 0x0000000005be996a clang::ParseAST(clang::Sema&, bool, bool) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x5be996a)
#32 0x0000000004976458 clang::CodeGenAction::ExecuteAction() (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x4976458)
#33 0x00000000042123c9 clang::FrontendAction::Execute() (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x42123c9)
#34 0x000000000419a896 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x419a896)
#35 0x00000000042f38c3 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0x42f38c3)
#36 0x0000000000b8cdc5 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0xb8cdc5)
#37 0x0000000000b88a07 ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&) driver.cpp:0:0
#38 0x0000000000b8a0cc clang_main(int, char**) (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0xb8a0cc)
#39 0x00007f4be2f666ca __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:74:3
#40 0x00007f4be2f66785 call_init ./csu/../csu/libc-start.c:128:20
#41 0x00007f4be2f66785 __libc_start_main ./csu/../csu/libc-start.c:347:5
#42 0x0000000000b83e95 _start (/opt/compiler-explorer/clang-assertions-16.0.0/bin/clang+0xb83e95)

[Endilll]

I managed to get this from 72 MB down to 60 KB, but my server continuously crashing trying to reduce this. gh48133 partial reduction.zip

[Endilll]

It's of a size that can be put into CE, but it doesn't reproduce there: https://godbolt.org/z/G78G5d6qb