ROOT 6.26/04 fails to build with LLVM/Clang 14.0.6 with C++17 / `libc++` on SLF7

greenc-FNAL commented 2 years ago

[x] Checked for duplicates

Describe the bug

ROOT 6.26/04 (https://github.com/root-project/root/tree/02b2af7af1be3430a3124236a7cdc6bb13cf90d2) fails to build with LLVM/Clang 14.0.6 with the following error from rootcling:

[7221/9777] Generating G__Core.cxx, ../lib/Core.pcm
FAILED: core/G__Core.cxx lib/Core.pcm ...
<snip/>
rootcling_stage1: /scratch/greenc/test-products/root/v6_26_04/source/root-6.26.04/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp:2799: unsigned int clang::ASTWriter::getSubmoduleID(clang::Module *): Assertion `(ID || !Mod) && "asked for module ID for non-local, non-imported module"' failed.
 #0 0x0000000003ee5514 PrintStackTraceSignalHandler(void*) Signals.cpp:0:0
 #1 0x0000000003ee32cc llvm::sys::RunSignalHandlers() Signals.cpp:0:0
 #2 0x0000000003ee5935 SignalHandler(int) Signals.cpp:0:0
 #3 0x00007f1552225630 __restore_rt sigaction.c:0:0
 #4 0x00007f1551316387 raise (/lib64/libc.so.6+0x36387)
 #5 0x00007f1551317a78 abort (/lib64/libc.so.6+0x37a78)
 #6 0x00007f155130f1a6 __assert_fail_base (/lib64/libc.so.6+0x2f1a6)
 #7 0x00007f155130f252 (/lib64/libc.so.6+0x2f252)
 #8 0x0000000001d23bf3 clang::ASTWriter::WriteSubmodules(clang::Module*) ASTWriter.cpp:0:0
 #9 0x0000000001d34a01 clang::ASTWriter::WriteASTCore(clang::Sema&, llvm::StringRef, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&, clang::Module*) ASTWriter.cpp:0:0
#10 0x0000000001d31700 clang::ASTWriter::WriteAST(clang::Sema&, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&, clang::Module*, llvm::StringRef, bool, bool) ASTWriter.cpp:0:0
#11 0x0000000001d7f347 clang::PCHGenerator::HandleTranslationUnit(clang::ASTContext&) GeneratePCH.cpp:0:0
#12 0x0000000001a4269c clang::MultiplexConsumer::HandleTranslationUnit(clang::ASTContext&) MultiplexConsumer.cpp:0:0
#13 0x0000000001ba6284 clang::ParseAST(clang::Sema&, bool, bool) ParseAST.cpp:0:0
#14 0x00000000019f9690 clang::FrontendAction::Execute() FrontendAction.cpp:0:0
#15 0x00000000019a4939 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) CompilerInstance.cpp:0:0
#16 0x00000000019afbe2 void llvm::function_ref<void ()>::callback_fn<compileModuleImpl(clang::CompilerInstance&, clang::SourceLocation, llvm::StringRef, clang::FrontendInputFile, llvm::StringRef, llvm::StringRef, llvm::function_ref<void (clang::CompilerInstance&)>, llvm:::function_ref<void (clang::CompilerInstance&)>)::$_3>(long) CompilerInstance.cpp:0:0
#17 0x0000000003e4378f llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) CrashRecoveryContext.cpp:0:0
#18 0x0000000003e438b4 RunSafelyOnThread_Dispatch(void*) CrashRecoveryContext.cpp:0:0
#19 0x0000000003e4388d llvm::CrashRecoveryContext::RunSafelyOnThread(llvm::function_ref<void ()>, unsigned int) CrashRecoveryContext.cpp:0:0
#20 0x00000000019ac171 compileModuleImpl(clang::CompilerInstance&, clang::SourceLocation, llvm::StringRef, clang::FrontendInputFile, llvm::StringRef, llvm::StringRef, llvm::function_ref<void (clang::CompilerInstance&)>, llvm::function_ref<void (clang::CompilerInstance&)>) CompilerInstance.cpp:0:0
#21 0x00000000019aa3b1 compileAndLoadModule(clang::CompilerInstance&, clang::SourceLocation, clang::SourceLocation, clang::Module*, llvm::StringRef) CompilerInstance.cpp:0:0
#22 0x00000000019a964a clang::CompilerInstance::loadModule(clang::SourceLocation, llvm::ArrayRef<std::__2::pair<clang::IdentifierInfo*, clang::SourceLocation> >, clang::Module::NameVisibilityKind, bool) CompilerInstance.cpp:0:0
#23 0x0000000003b2a5be clang::Preprocessor::HandleHeaderIncludeOrImport(clang::SourceLocation, clang::Token&, clang::Token&, clang::SourceLocation, clang::DirectoryLookup const*, clang::FileEntry const*) PPDirectives.cpp:0:0
#24 0x0000000003b221d5 clang::Preprocessor::HandleIncludeDirective(clang::SourceLocation, clang::Token&, clang::DirectoryLookup const*, clang::FileEntry const*) PPDirectives.cpp:0:0
#25 0x0000000003b22b03 clang::Preprocessor::HandleDirective(clang::Token&) PPDirectives.cpp:0:0
#26 0x0000000003afa515 clang::Lexer::LexTokenInternal(clang::Token&, bool) Lexer.cpp:0:0
#27 0x0000000003af7b42 clang::Lexer::Lex(clang::Token&) Lexer.cpp:0:0
#28 0x0000000003b5a264 clang::Preprocessor::Lex(clang::Token&) Preprocessor.cpp:0:0
#29 0x0000000001c40f00 clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, bool) Parser.cpp:0:0
#30 0x000000000152a89e cling::IncrementalParser::ParseInternal(llvm::StringRef) IncrementalParser.cpp:0:0
#31 0x0000000001529c34 cling::IncrementalParser::Initialize(llvm::SmallVectorImpl<llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > > >&, bool) IncrementalParser.cpp:0:0
#32 0x000000000151d992 cling::Interpreter::Interpreter(int, char const* const*, char const*, std::__2::vector<std::__2::shared_ptr<clang::ModuleFileExtension>, std::__2::allocator<std::__2::shared_ptr<clang::ModuleFileExtension> > > const&, bool, cling::Interpreter const*) Interpreter.cpp:0:0
#33 0x00000000014a5117 std::__2::vector<std::__2::shared_ptr<clang::ModuleFileExtension>, std::__2::allocator<std::__2::shared_ptr<clang::ModuleFileExtension> > >::~vector() /scratch/greenc/test-products/clang/v14_0_6b/Linux64bit+3.10-2.17/bin/../include/c++/v1/vector:399:17
#34 0x00000000014a5117 RootClingMain(int, char**, bool) /scratch/greenc/test-products/root/v6_26_04/source/root-6.26.04/core/dictgen/src/rootcling_impl.cxx:4315:7
#35 0x00000000014b2135 ROOT_rootcling_Driver /scratch/greenc/test-products/root/v6_26_04/source/root-6.26.04/core/dictgen/src/rootcling_impl.cxx:0:16
#36 0x00000000014f6b7e main /scratch/greenc/test-products/root/v6_26_04/source/root-6.26.04/core/rootcling_stage1/src/rootcling_stage1.cxx:46:4
#37 0x00007f1551302555 __libc_start_main (/lib64/libc.so.6+0x22555)
#38 0x000000000143eb89 _start (/scratch/greenc/test-products/root/v6_26_04/build/Linux64bit+3.10-2.17-c13-p3912-prof/core/rootcling_stage1/src/rootcling_stage1+0x143eb89)
<snip/>

Expected behavior

Successful build.

To Reproduce

Attempt to build ROOT with LLVM/Clang 14.0.6 and C++17 / libc++ (precise configuration should be irrelevant provided runtime_cxxmodules is ON).

Additional context

This was a second, simplified attempt after rootcling failed to find PSTL headers (e.g. __pstl_memory) in <clang-lib>/clang/14.0.6/include during the ROOT build when LLVM/Clang was compiled with parallel algorithms enabled in libc++.

~Currently attempting to reproduce on master branch.~

Fail with identical error on master (https://github.com/root-project/root/tree/7b7983fa8ab9e1a4af98776f7f80bea1f9b2b0b0).

Axel-Naumann commented 2 years ago

Is there any chance to get the backtrace with some more debug info from the llvm side, by adding -DLLVM_BUILD_TYPE=Debug to ROOT's configuration?

greenc-FNAL commented 2 years ago

Should be relatively straightforward. Working on it now.

Axel-Naumann commented 2 years ago

Sorry for the slow response from me! To make things worse, it's highly asymmetrical: thanks for your super fast reaction :-)

greenc-FNAL commented 2 years ago

I'm on vacation for the next week, but I'll try to provide whatever extra information you can tell me you need today ASAP. Log file with LLVM_DEBUG attached as build-root-6.26.04b-sl7-x86_64-c14-p3912-debug-runtime_cxxmodules-llvm_debug.log

Axel-Naumann commented 2 years ago

@vgvassilev FYI:

 cd /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/core && /scratch/greenc/test-products/cmake/v3_23_2/Linux64bit+3.10-2.17/bin/cmake -E env LD_LIBRARY_PATH=/scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/lib:/products/xrootd/v5_4_3a/Linux64bit+3.10-2.17-c14-p3912-debug/lib:/scratch/greenc/test-products/tbb/v2021_5_0b/Linux64bit+3.10-2.17-c14/lib:/scratch/greenc/test-products/gcc/v12_1_0/Linux64bit+3.10-2.17/lib64:/scratch/greenc/test-products/gcc/v12_1_0/Linux64bit+3.10-2.17/lib:/products/pythia/v6_4_28u/Linux64bit+3.10-2.17-c14-debug/lib:/scratch/greenc/test-products/postgresql/v14_4/Linux64bit+3.10-2.17-p3912/lib:/products/openblas/v0_3_20c/Linux64bit+3.10-2.17-c14/lib:/scratch/greenc/test-products/python/v3_9_12/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/sqlite/v3_38_05_00/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/libxml2/v2_9_12/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/gsl/v2_7/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib/x86_64-unknown-linux-gnu:/scratch/greenc/test-products/fftw/v3_3_10/Linux64bit+3.10-2.17/lib: /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/core/rootcling_stage1/src/rootcling_stage1 -v2 -f G__Core.cxx -cxxmodule -s /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/lib/libCore.so -excludePath /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04 -excludePath /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/ginclude -excludePath /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/externals -excludePath /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/builtins -writeEmptyRootPCM -m _Builtin_intrinsics -mByproduct _Builtin_intrinsics -mByproduct ROOT_Foundation_Stage1_NoRTTI -mByproduct ROOT_Foundation_C -mByproduct ROOT_Rtypes -D__CORRECT_ISO_CPP_WCHAR_H_PROTO -D__STDC_NO_COMPLEX__ -D__COMPLEX_H__ -D_COMPLEX_H -D__CLANG_STDATOMIC_H -compilerI/scratch/greenc/test-products/tbb/v2021_5_0b/Linux64bit+3.10-2.17-c14/include -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/x86_64-unknown-linux-gnu/c++/v1 -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/c++/v1 -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib/clang/14.0.6/include -compilerI/usr/local/include -compilerI/usr/include -compilerI/scratch/greenc/test-products/tbb/v2021_5_0b/Linux64bit+3.10-2.17-c14/include -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib/clang/14.0.6/include -compilerI/usr/local/include -compilerI/usr/include -I/scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/include -I/scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/ginclude -I/scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/ginclude -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/base/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/foundation/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/cont/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/gui/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/meta/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/clib/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/rint/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/zip/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/thread/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/textinput/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/clingutils/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/base/v7/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/foundation/v7/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/unix/inc -I/usr/include ROOT/TErrorDefaultHandler.hxx ROOT/TSequentialExecutor.hxx ROOT/StringConv.hxx Buttons.h Bytes.h Byteswap.h KeySymbols.h MessageTypes.h Riostream.h Rtypes.h TApplication.h TAtt3D.h TAttAxis.h TAttBBox2D.h TAttBBox.h TAttFill.h TAttLine.h TAttMarker.h TAttPad.h TAttText.h TBase64.h TBenchmark.h TBuffer3D.h TBuffer3DTypes.h TBuffer.h TColor.h TColorGradient.h TDatime.h TDirectory.h TEnv.h TException.h TExec.h TFileCollection.h TFileInfo.h TFolder.h TInetAddress.h TMacro.h TMathBase.h TMD5.h TMemberInspector.h TMessageHandler.h TNamed.h TNotifyLink.h TObject.h TObjString.h TParameter.h TPluginManager.h TPoint.h TPRegexp.h TProcessID.h TProcessUUID.h TQClass.h TQCommand.h TQConnection.h TQObject.h TRedirectOutputGuard.h TRefCnt.h TRef.h TRegexp.h TRemoteObject.h TROOT.h TRootIOCtor.h TStopwatch.h TStorage.h TString.h TStringLong.h TStyle.h TSysEvtHandler.h TSystemDirectory.h TSystemFile.h TSystem.h TTask.h TThreadSlots.h TTime.h TTimer.h TTimeStamp.h TUri.h TUrl.h TUUID.h TVersionCheck.h TVirtualAuth.h TVirtualFFT.h TVirtualGL.h TVirtualMonitoring.h TVirtualMutex.h TVirtualPadEditor.h TVirtualPad.h TVirtualPadPainter.h TVirtualPerfStats.h TVirtualPS.h TVirtualQConnection.h TVirtualRWMutex.h TVirtualTableInterface.h TVirtualViewer3D.h TVirtualX.h ROOT/RDirectoryEntry.hxx ROOT/RIndexIter.hxx strlcpy.h snprintf.h strtok.h ROOT/TSeq.hxx TArrayC.h TArrayD.h TArrayF.h TArray.h TArrayI.h TArrayL64.h TArrayL.h TArrayS.h TBits.h TBtree.h TClassTable.h TClonesArray.h TCollection.h TCollectionProxyInfo.h TExMap.h THashList.h THashTable.h TIterator.h TList.h TMap.h TObjArray.h TObjectTable.h TOrdCollection.h TRefArray.h TRefTable.h TSeqCollection.h TSortedList.h TVirtualCollectionProxy.h ESTLType.h RStringView.h TClassEdit.h TError.h ROOT/RLogger.hxx ROOT/RNotFn.hxx ROOT/RRangeCast.hxx ROOT/RSpan.hxx ROOT/RStringView.hxx ROOT/StringUtils.hxx ROOT/span.hxx ROOT/TypeTraits.hxx TUnixSystem.h root_std_complex.h GuiTypes.h TApplicationImp.h TBrowser.h TBrowserImp.h TCanvasImp.h TClassMenuItem.h TContextMenu.h TContextMenuImp.h TControlBarImp.h TGuiFactory.h TInspectorImp.h TObjectSpy.h TToggleGroup.h TToggle.h TBaseClass.h TClassGenerator.h TClass.h TClassRef.h TClassStreamer.h TDataMember.h TDataType.h TDictAttributeMap.h TDictionary.h TEnumConstant.h TEnum.h TFileMergeInfo.h TFunction.h TFunctionTemplate.h TGenericClassInfo.h TGlobal.h TInterpreter.h TInterpreterValue.h TIsAProxy.h TListOfDataMembers.h TListOfEnums.h TListOfEnumsWithLock.h TListOfFunctions.h TListOfFunctionTemplates.h TMemberStreamer.h TMethodArg.h TMethodCall.h TMethod.h TProtoClass.h TRealData.h TSchemaHelper.h TSchemaRule.h TSchemaRuleSet.h TStatusBitsChecker.h TStreamerElement.h TStreamer.h TVirtualIsAProxy.h TVirtualRefProxy.h TVirtualStreamerInfo.h TVirtualArray.h TVirtualObject.h Getline.h /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/base/inc/LinkDef.h
FAILED: core/G__Core.cxx lib/Core.pcm /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/core/G__Core.cxx /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/lib/Core.pcm 
cd /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/core && /scratch/greenc/test-products/cmake/v3_23_2/Linux64bit+3.10-2.17/bin/cmake -E env LD_LIBRARY_PATH=/scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/lib:/products/xrootd/v5_4_3a/Linux64bit+3.10-2.17-c14-p3912-debug/lib:/scratch/greenc/test-products/tbb/v2021_5_0b/Linux64bit+3.10-2.17-c14/lib:/scratch/greenc/test-products/gcc/v12_1_0/Linux64bit+3.10-2.17/lib64:/scratch/greenc/test-products/gcc/v12_1_0/Linux64bit+3.10-2.17/lib:/products/pythia/v6_4_28u/Linux64bit+3.10-2.17-c14-debug/lib:/scratch/greenc/test-products/postgresql/v14_4/Linux64bit+3.10-2.17-p3912/lib:/products/openblas/v0_3_20c/Linux64bit+3.10-2.17-c14/lib:/scratch/greenc/test-products/python/v3_9_12/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/sqlite/v3_38_05_00/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/libxml2/v2_9_12/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/gsl/v2_7/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib/x86_64-unknown-linux-gnu:/scratch/greenc/test-products/fftw/v3_3_10/Linux64bit+3.10-2.17/lib: /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/core/rootcling_stage1/src/rootcling_stage1 -v2 -f G__Core.cxx -cxxmodule -s /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/lib/libCore.so -excludePath /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04 -excludePath /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/ginclude -excludePath /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/externals -excludePath /scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/builtins -writeEmptyRootPCM -m _Builtin_intrinsics -mByproduct _Builtin_intrinsics -mByproduct ROOT_Foundation_Stage1_NoRTTI -mByproduct ROOT_Foundation_C -mByproduct ROOT_Rtypes -D__CORRECT_ISO_CPP_WCHAR_H_PROTO -D__STDC_NO_COMPLEX__ -D__COMPLEX_H__ -D_COMPLEX_H -D__CLANG_STDATOMIC_H -compilerI/scratch/greenc/test-products/tbb/v2021_5_0b/Linux64bit+3.10-2.17-c14/include -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/x86_64-unknown-linux-gnu/c++/v1 -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/c++/v1 -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib/clang/14.0.6/include -compilerI/usr/local/include -compilerI/usr/include -compilerI/scratch/greenc/test-products/tbb/v2021_5_0b/Linux64bit+3.10-2.17-c14/include -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib/clang/14.0.6/include -compilerI/usr/local/include -compilerI/usr/include -I/scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/include -I/scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/ginclude -I/scratch/greenc/test-products/root/v6_26_04b/build/Linux64bit+3.10-2.17-c14-p3912-debug/ginclude -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/base/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/foundation/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/cont/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/gui/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/meta/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/clib/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/rint/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/zip/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/thread/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/textinput/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/clingutils/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/base/v7/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/foundation/v7/inc -I/scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/unix/inc -I/usr/include ROOT/TErrorDefaultHandler.hxx ROOT/TSequentialExecutor.hxx ROOT/StringConv.hxx Buttons.h Bytes.h Byteswap.h KeySymbols.h MessageTypes.h Riostream.h Rtypes.h TApplication.h TAtt3D.h TAttAxis.h TAttBBox2D.h TAttBBox.h TAttFill.h TAttLine.h TAttMarker.h TAttPad.h TAttText.h TBase64.h TBenchmark.h TBuffer3D.h TBuffer3DTypes.h TBuffer.h TColor.h TColorGradient.h TDatime.h TDirectory.h TEnv.h TException.h TExec.h TFileCollection.h TFileInfo.h TFolder.h TInetAddress.h TMacro.h TMathBase.h TMD5.h TMemberInspector.h TMessageHandler.h TNamed.h TNotifyLink.h TObject.h TObjString.h TParameter.h TPluginManager.h TPoint.h TPRegexp.h TProcessID.h TProcessUUID.h TQClass.h TQCommand.h TQConnection.h TQObject.h TRedirectOutputGuard.h TRefCnt.h TRef.h TRegexp.h TRemoteObject.h TROOT.h TRootIOCtor.h TStopwatch.h TStorage.h TString.h TStringLong.h TStyle.h TSysEvtHandler.h TSystemDirectory.h TSystemFile.h TSystem.h TTask.h TThreadSlots.h TTime.h TTimer.h TTimeStamp.h TUri.h TUrl.h TUUID.h TVersionCheck.h TVirtualAuth.h TVirtualFFT.h TVirtualGL.h TVirtualMonitoring.h TVirtualMutex.h TVirtualPadEditor.h TVirtualPad.h TVirtualPadPainter.h TVirtualPerfStats.h TVirtualPS.h TVirtualQConnection.h TVirtualRWMutex.h TVirtualTableInterface.h TVirtualViewer3D.h TVirtualX.h ROOT/RDirectoryEntry.hxx ROOT/RIndexIter.hxx strlcpy.h snprintf.h strtok.h ROOT/TSeq.hxx TArrayC.h TArrayD.h TArrayF.h TArray.h TArrayI.h TArrayL64.h TArrayL.h TArrayS.h TBits.h TBtree.h TClassTable.h TClonesArray.h TCollection.h TCollectionProxyInfo.h TExMap.h THashList.h THashTable.h TIterator.h TList.h TMap.h TObjArray.h TObjectTable.h TOrdCollection.h TRefArray.h TRefTable.h TSeqCollection.h TSortedList.h TVirtualCollectionProxy.h ESTLType.h RStringView.h TClassEdit.h TError.h ROOT/RLogger.hxx ROOT/RNotFn.hxx ROOT/RRangeCast.hxx ROOT/RSpan.hxx ROOT/RStringView.hxx ROOT/StringUtils.hxx ROOT/span.hxx ROOT/TypeTraits.hxx TUnixSystem.h root_std_complex.h GuiTypes.h TApplicationImp.h TBrowser.h TBrowserImp.h TCanvasImp.h TClassMenuItem.h TContextMenu.h TContextMenuImp.h TControlBarImp.h TGuiFactory.h TInspectorImp.h TObjectSpy.h TToggleGroup.h TToggle.h TBaseClass.h TClassGenerator.h TClass.h TClassRef.h TClassStreamer.h TDataMember.h TDataType.h TDictAttributeMap.h TDictionary.h TEnumConstant.h TEnum.h TFileMergeInfo.h TFunction.h TFunctionTemplate.h TGenericClassInfo.h TGlobal.h TInterpreter.h TInterpreterValue.h TIsAProxy.h TListOfDataMembers.h TListOfEnums.h TListOfEnumsWithLock.h TListOfFunctions.h TListOfFunctionTemplates.h TMemberStreamer.h TMethodArg.h TMethodCall.h TMethod.h TProtoClass.h TRealData.h TSchemaHelper.h TSchemaRule.h TSchemaRuleSet.h TStatusBitsChecker.h TStreamerElement.h TStreamer.h TVirtualIsAProxy.h TVirtualRefProxy.h TVirtualStreamerInfo.h TVirtualArray.h TVirtualObject.h Getline.h /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/core/base/inc/LinkDef.h

rootcling_stage1: /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp:2799: unsigned int clang::ASTWriter::getSubmoduleID(clang::Module *): Assertion `(ID || !Mod) && "asked for module ID for non-local, non-imported module"' failed.

 #7 0x00007fe099a431a6 __assert_fail_base (/lib64/libc.so.6+0x2f1a6)
 #8 0x00007fe099a43252 (/lib64/libc.so.6+0x2f252)
 #9 0x0000000003784bb9 clang::ASTWriter::getSubmoduleID(clang::Module*) /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp:2800:10
#10 0x00000000037869ad clang::ASTWriter::WriteSubmodules(clang::Module*) /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp:3002:26
#11 0x000000000379c09f clang::ASTWriter::WriteASTCore(clang::Sema&, llvm::StringRef, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&, clang::Module*) /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp:0:5
#12 0x0000000003799fec clang::ASTWriter::WriteAST(clang::Sema&, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&, clang::Module*, llvm::StringRef, bool, bool) /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp:0:7
#13 0x000000000382e2c2 clang::PCHGenerator::HandleTranslationUnit(clang::ASTContext&) /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/interpreter/llvm/src/tools/clang/lib/Serialization/GeneratePCH.cpp:63:14
#14 0x00000000033cf873 clang::MultiplexConsumer::HandleTranslationUnit(clang::ASTContext&) /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/interpreter/llvm/src/tools/clang/lib/Frontend/MultiplexConsumer.cpp:291:23
#15 0x0000000003542e96 clang::ParseAST(clang::Sema&, bool, bool) /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/interpreter/llvm/src/tools/clang/lib/Parse/ParseAST.cpp:178:12
#16 0x000000000336cc4c clang::ASTFrontendAction::ExecuteAction() /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/interpreter/llvm/src/tools/clang/lib/Frontend/FrontendAction.cpp:1043:1
#17 0x000000000336c68c clang::FrontendAction::Execute() /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/interpreter/llvm/src/tools/clang/lib/Frontend/FrontendAction.cpp:938:7
#18 0x0000000003310825 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:944:23
#19 0x000000000331779d compileModuleImpl(clang::CompilerInstance&, clang::SourceLocation, llvm::StringRef, clang::FrontendInputFile, llvm::StringRef, llvm::StringRef, llvm::function_ref<void (clang::CompilerInstance&)>, llvm::function_ref<void (clang::CompilerInstance&)>)::$_3::operator()() const /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1129:7

Please lmk if you have any indication what could cause this.

@chissg thanks a lot, that's super helpful (and very unexpected).

Axel-Naumann commented 2 years ago

Does LLVM/Clang 14.0.6 / libc++ introduce additional stdlib modules?

greenc-FNAL commented 2 years ago

Does LLVM/Clang 14.0.6 / libc++ introduce additional stdlib modules?

How would I find out?

Axel-Naumann commented 2 years ago

I am hoping that @vgvassilev will either know of be able to tell us what to look for (likely something *modulemap)

vgvassilev commented 2 years ago

Which is the failing submodule?

greenc-FNAL commented 2 years ago

@vgvassilev how would I ascertain that?

Attached: result of find . -name '*modulemap' in LLVM/Clang install tree. LLVM-Clang-14.0.6-modulemaps.tar.gz

Axel-Naumann commented 2 years ago

@vgvassilev not sure what you expect here - gdb doesn't print StringRef etc, so maybe a 2 line patch that prints this before asserting would be more constructive?

I've lowered this to "default priority" because it likely affects only current libc++ - and that's a very rare environment. We cannot block 6.26/06 because of this issue.

greenc-FNAL commented 2 years ago

Bump?

vgvassilev commented 2 years ago

Apologies for the delay.

Can you apply that patch and see which is the output it currently fails upon:

diff --git a/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp b/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp
index 3829014415..e4a2fab675 100644
--- a/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp
+++ b/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp
@@ -2789,11 +2789,12 @@ unsigned ASTWriter::getLocalOrImportedSubmoduleID(Module *Mod) {

   return SubmoduleIDs[Mod] = NextSubmoduleID++;
 }
-
+extern "C" int printf(const char*,...);
 unsigned ASTWriter::getSubmoduleID(Module *Mod) {
   // FIXME: This can easily happen, if we have a reference to a submodule that
   // did not result in us loading a module file for that submodule. For
   // instance, a cross-top-level-module 'conflict' declaration will hit this.
+  printf("Submodule=%s\n", Mod->Name.c_str());
   unsigned ID = getLocalOrImportedSubmoduleID(Mod);
   assert((ID || !Mod) &&
          "asked for module ID for non-local, non-imported module");

I need to remember how to reproduce your setup and I could be more helpful...

greenc-FNAL commented 1 year ago

Sorry for the radio silence on this: other priorities took over and this wasn't a showstopper at the time.

I have just verified the same failure with 66df8b8ee6b336197678950c4f9c83f2c479662a, and will apply the suggested patch to that and see where we get to.

greenc-FNAL commented 1 year ago

(Patch applied with offset)

greenc-FNAL commented 1 year ago

@vgvassilev @Axel-Naumann Report: no-one called ASTWriter::getSubmoduleID(), but I did get the following output:

FAILED: core/G__Core.cxx lib/Core.pcm /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/core/G__Core.cxx /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/lib/Core.pcm 
cd /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/core && /scratch/greenc/test-products/cmake/v3_24_1/Linux64bit+3.10-2.17/bin/cmake -E env LD_LIBRARY_PATH=/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/lib:/scratch/greenc/test-products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-c14-p3913-debug/lib:/scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/lib:/scratch/greenc/test-products/gcc/v12_1_0/Linux64bit+3.10-2.17/lib64:/scratch/greenc/test-products/gcc/v12_1_0/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/pythia/v6_4_28u/Linux64bit+3.10-2.17-c14-debug/lib:/scratch/greenc/test-products/postgresql/v14_5/Linux64bit+3.10-2.17-p3913/lib:/scratch/greenc/test-products/openblas/v0_3_21/Linux64bit+3.10-2.17-c14/lib:/scratch/greenc/test-products/python/v3_9_13/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/sqlite/v3_39_02_00/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/libxml2/v2_9_12/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/gsl/v2_7/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib:/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib/x86_64-unknown-linux-gnu:/scratch/greenc/test-products/fftw/v3_3_10/Linux64bit+3.10-2.17/lib: /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/core/rootcling_stage1/src/rootcling_stage1 -v4 -f G__Core.cxx -cxxmodule -s /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/lib/libCore.so -excludePath /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99 -excludePath /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude -excludePath /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/externals -excludePath /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/builtins -writeEmptyRootPCM -m _Builtin_intrinsics -mByproduct _Builtin_intrinsics -mByproduct ROOT_Foundation_Stage1_NoRTTI -mByproduct ROOT_Foundation_C -mByproduct ROOT_Rtypes -D__CORRECT_ISO_CPP_WCHAR_H_PROTO -D__STDC_NO_COMPLEX__ -D__COMPLEX_H__ -D_COMPLEX_H -D__CLANG_STDATOMIC_H -compilerI/scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/x86_64-unknown-linux-gnu/c++/v1 -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/c++/v1 -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib/clang/14.0.6/include -compilerI/usr/local/include -compilerI/usr/include -compilerI/scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib/clang/14.0.6/include -compilerI/usr/local/include -compilerI/usr/include -I/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/include -I/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude -I/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/cont/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/gui/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/meta/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clib/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/rint/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/zip/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/thread/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/textinput/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clingutils/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/v7/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/v7/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/unix/inc -I/usr/include ROOT/TErrorDefaultHandler.hxx ROOT/TSequentialExecutor.hxx ROOT/StringConv.hxx Buttons.h Bytes.h Byteswap.h KeySymbols.h MessageTypes.h Riostream.h Rtypes.h TApplication.h TAtt3D.h TAttAxis.h TAttBBox2D.h TAttBBox.h TAttFill.h TAttLine.h TAttMarker.h TAttPad.h TAttText.h TBase64.h TBenchmark.h TBuffer3D.h TBuffer3DTypes.h TBuffer.h TColor.h TColorGradient.h TDatime.h TDirectory.h TEnv.h TException.h TExec.h TFileCollection.h TFileInfo.h TFolder.h TInetAddress.h TMacro.h TMathBase.h TMD5.h TMemberInspector.h TMessageHandler.h TNamed.h TNotifyLink.h TObject.h TObjString.h TParameter.h TPluginManager.h TPoint.h TPRegexp.h TProcessID.h TProcessUUID.h TQClass.h TQCommand.h TQConnection.h TQObject.h TRedirectOutputGuard.h TRefCnt.h TRef.h TRegexp.h TRemoteObject.h TROOT.h TRootIOCtor.h TStopwatch.h TStorage.h TString.h TStringLong.h TStyle.h TSysEvtHandler.h TSystemDirectory.h TSystemFile.h TSystem.h TTask.h TThreadSlots.h TTime.h TTimer.h TTimeStamp.h TUri.h TUrl.h TUUID.h TVersionCheck.h TVirtualAuth.h TVirtualFFT.h TVirtualGL.h TVirtualMonitoring.h TVirtualMutex.h TVirtualPadEditor.h TVirtualPad.h TVirtualPadPainter.h TVirtualPerfStats.h TVirtualPS.h TVirtualQConnection.h TVirtualRWMutex.h TVirtualTableInterface.h TVirtualViewer3D.h TVirtualX.h ROOT/RDirectoryEntry.hxx ROOT/RIndexIter.hxx strlcpy.h snprintf.h strtok.h ROOT/TSeq.hxx TArrayC.h TArrayD.h TArrayF.h TArray.h TArrayI.h TArrayL64.h TArrayL.h TArrayS.h TBits.h TBtree.h TClassTable.h TClonesArray.h TCollection.h TCollectionProxyInfo.h TExMap.h THashList.h THashTable.h TIterator.h TList.h TMap.h TObjArray.h TObjectTable.h TOrdCollection.h TRefArray.h TRefTable.h TSeqCollection.h TSortedList.h TVirtualCollectionProxy.h ESTLType.h RStringView.h TClassEdit.h TError.h ThreadLocalStorage.h ROOT/RLogger.hxx ROOT/RNotFn.hxx ROOT/RRangeCast.hxx ROOT/RSpan.hxx ROOT/RStringView.hxx ROOT/StringUtils.hxx ROOT/span.hxx ROOT/TypeTraits.hxx TUnixSystem.h root_std_complex.h GuiTypes.h TApplicationImp.h TBrowser.h TBrowserImp.h TCanvasImp.h TClassMenuItem.h TContextMenu.h TContextMenuImp.h TControlBarImp.h TGuiFactory.h TInspectorImp.h TObjectSpy.h TToggleGroup.h TToggle.h TBaseClass.h TClassGenerator.h TClass.h TClassRef.h TClassStreamer.h TDataMember.h TDataType.h TDictAttributeMap.h TDictionary.h TEnumConstant.h TEnum.h TFileMergeInfo.h TFunction.h TFunctionTemplate.h TGenericClassInfo.h TGlobal.h TInterpreter.h TInterpreterValue.h TIsAProxy.h TListOfDataMembers.h TListOfEnums.h TListOfEnumsWithLock.h TListOfFunctions.h TListOfFunctionTemplates.h TMemberStreamer.h TMethodArg.h TMethodCall.h TMethod.h TProtoClass.h TRealData.h TSchemaHelper.h TSchemaRule.h TSchemaRuleSet.h TStatusBitsChecker.h TStreamerElement.h TStreamer.h TVirtualIsAProxy.h TVirtualRefProxy.h TVirtualStreamerInfo.h TVirtualArray.h TVirtualObject.h Getline.h /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/inc/LinkDef.h
cling version ROOT_1.0~dev
Looking for C++ headers with:
  LC_ALL=C clang-14   -xc++ -E -v /dev/null 2>&1 | sed -n -e '/^.include/,${' -e '/^ \/.*++/p' -e '}'
Found:
  /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/x86_64-unknown-linux-gnu/c++/v1
  /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1
clang version 13.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/core/rootcling_stage1/src
Found candidate GCC installation: /usr/lib/gcc/i686-redhat-linux/4.8.2
Found candidate GCC installation: /usr/lib/gcc/i686-redhat-linux/4.8.5
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.2
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.5
Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.5
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
ignoring nonexistent directory "/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../x86_64-redhat-linux/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude"
ignoring duplicate directory "/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/include"
ignoring duplicate directory "/scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include"
ignoring duplicate directory "/usr/include"
  as it is a non-system directory that duplicates a system directory
#include "..." search starts here:
 .
#include <...> search starts here:
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/include
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/cont/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/gui/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/meta/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clib/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/rint/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/zip/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/thread/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/textinput/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clingutils/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/v7/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/v7/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/unix/inc
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc
 /scratch/greenc/test-products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-c14-p3913-debug/include
 /scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include
 /scratch/greenc/test-products/pythia/v6_4_28u/Linux64bit+3.10-2.17-c14-debug/include
 /scratch/greenc/test-products/postgresql/v14_5/Linux64bit+3.10-2.17-p3913/include
 /scratch/greenc/test-products/openblas/v0_3_21/Linux64bit+3.10-2.17-c14/include
 /scratch/greenc/test-products/fftw/v3_3_10/Linux64bit+3.10-2.17/include
 /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/x86_64-unknown-linux-gnu/c++/v1
 /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc/cling/lib/clang/13.0.0/include
 /usr/local/include
 /usr/include
End of search list.
'/usr/include/module.modulemap' does not exist. Mounting '/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc/cling/libc.modulemap' as '/usr/include/module.modulemap'
Loading '/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/module.modulemap'
'/usr/include/boost/module.modulemap' does not exist. Mounting '/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc/cling/boost.modulemap' as '/usr/include/boost/module.modulemap'
modulemap.overlay.yaml
{
 'version': 0,
 'roots': [
{ 'name': '/usr/include', 'type': 'directory',
'contents': [
   { 'name': 'libc.modulemap', 'type': 'file',
  'external-contents': '/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc/cling/libc.modulemap'
}
 ]
 },
{ 'name': '/usr/include/boost', 'type': 'directory',
'contents': [
   { 'name': 'boost.modulemap', 'type': 'file',
  'external-contents': '/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc/cling/boost.modulemap'
}
 ]
 }]
 }
 ]
 }
Adding runtime include paths:
  ""
Added include paths:
Setting up system headers with clang:
ignoring nonexistent directory "/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../x86_64-redhat-linux/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude"
ignoring duplicate directory "/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/include"
ignoring duplicate directory "/scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include"
ignoring duplicate directory "/usr/include"
  as it is a non-system directory that duplicates a system directory
#include "..." search starts here:
 .
#include <...> search starts here:
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/include
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/cont/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/gui/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/meta/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clib/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/rint/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/zip/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/thread/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/textinput/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clingutils/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/v7/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/v7/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/unix/inc
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc
 /scratch/greenc/test-products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-c14-p3913-debug/include
 /scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include
 /scratch/greenc/test-products/pythia/v6_4_28u/Linux64bit+3.10-2.17-c14-debug/include
 /scratch/greenc/test-products/postgresql/v14_5/Linux64bit+3.10-2.17-p3913/include
 /scratch/greenc/test-products/openblas/v0_3_21/Linux64bit+3.10-2.17-c14/include
 /scratch/greenc/test-products/fftw/v3_3_10/Linux64bit+3.10-2.17/include
 /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/x86_64-unknown-linux-gnu/c++/v1
 /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc/cling/lib/clang/13.0.0/include
 /usr/local/include
 /usr/include
End of search list.
clang -cc1 version 13.0.0 based upon LLVM 13.0.0 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../x86_64-redhat-linux/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude"
ignoring duplicate directory "/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/include"
ignoring duplicate directory "/scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include"
ignoring duplicate directory "/usr/include"
  as it is a non-system directory that duplicates a system directory
#include "..." search starts here:
 .
#include <...> search starts here:
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/include
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/cont/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/gui/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/meta/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clib/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/rint/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/zip/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/thread/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/textinput/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clingutils/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/v7/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/v7/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/unix/inc
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc
 /scratch/greenc/test-products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-c14-p3913-debug/include
 /scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include
 /scratch/greenc/test-products/pythia/v6_4_28u/Linux64bit+3.10-2.17-c14-debug/include
 /scratch/greenc/test-products/postgresql/v14_5/Linux64bit+3.10-2.17-p3913/include
 /scratch/greenc/test-products/openblas/v0_3_21/Linux64bit+3.10-2.17-c14/include
 /scratch/greenc/test-products/fftw/v3_3_10/Linux64bit+3.10-2.17/include
 /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/x86_64-unknown-linux-gnu/c++/v1
 /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc/cling/lib/clang/13.0.0/include
 /usr/local/include
 /usr/include
End of search list.
clang -cc1 version 13.0.0 based upon LLVM 13.0.0 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../x86_64-redhat-linux/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude"
ignoring duplicate directory "/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/include"
ignoring duplicate directory "/scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include"
ignoring duplicate directory "/usr/include"
  as it is a non-system directory that duplicates a system directory
#include "..." search starts here:
 .
#include <...> search starts here:
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/include
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/cont/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/gui/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/meta/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clib/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/rint/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/zip/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/thread/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/textinput/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clingutils/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/v7/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/v7/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/unix/inc
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc
 /scratch/greenc/test-products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-c14-p3913-debug/include
 /scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include
 /scratch/greenc/test-products/pythia/v6_4_28u/Linux64bit+3.10-2.17-c14-debug/include
 /scratch/greenc/test-products/postgresql/v14_5/Linux64bit+3.10-2.17-p3913/include
 /scratch/greenc/test-products/openblas/v0_3_21/Linux64bit+3.10-2.17-c14/include
 /scratch/greenc/test-products/fftw/v3_3_10/Linux64bit+3.10-2.17/include
 /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/x86_64-unknown-linux-gnu/c++/v1
 /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc/cling/lib/clang/13.0.0/include
 /usr/local/include
 /usr/include
End of search list.
clang -cc1 version 13.0.0 based upon LLVM 13.0.0 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../x86_64-redhat-linux/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude"
ignoring duplicate directory "/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/include"
ignoring duplicate directory "/scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include"
ignoring duplicate directory "/usr/include"
  as it is a non-system directory that duplicates a system directory
#include "..." search starts here:
 .
#include <...> search starts here:
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/include
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/cont/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/gui/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/meta/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clib/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/rint/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/zip/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/thread/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/textinput/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clingutils/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/v7/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/v7/inc
 /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/unix/inc
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc
 /scratch/greenc/test-products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-c14-p3913-debug/include
 /scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include
 /scratch/greenc/test-products/pythia/v6_4_28u/Linux64bit+3.10-2.17-c14-debug/include
 /scratch/greenc/test-products/postgresql/v14_5/Linux64bit+3.10-2.17-p3913/include
 /scratch/greenc/test-products/openblas/v0_3_21/Linux64bit+3.10-2.17-c14/include
 /scratch/greenc/test-products/fftw/v3_3_10/Linux64bit+3.10-2.17/include
 /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/x86_64-unknown-linux-gnu/c++/v1
 /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1
 /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc/cling/lib/clang/13.0.0/include
 /usr/local/include
 /usr/include
End of search list.
  #0 0x0000000009107d3a llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Unix/Signals.inc:565:11
  #1 0x0000000009107eeb PrintStackTraceSignalHandler(void*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Unix/Signals.inc:632:1
  #2 0x0000000009106713 llvm::sys::RunSignalHandlers() /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Signals.cpp:97:5
  #3 0x0000000009108505 SignalHandler(int) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Unix/Signals.inc:407:1
  #4 0x00007ff5834a3630 __restore_rt sigaction.c:0:0
  #5 0x00000000031e3975 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::__is_long() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1445:22
  #6 0x00000000031e3929 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::__get_pointer() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1544:17
  #7 0x00000000031e38e5 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::data() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1251:75
  #8 0x00000000031e1925 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::c_str() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1249:48
  #9 0x0000000004638fb9 clang::ASTWriter::getSubmoduleID(clang::Module*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp:2501:38
 #10 0x00000000046bdc47 clang::ASTDeclWriter::VisitDecl(clang::Decl*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriterDecl.cpp:364:27
 #11 0x00000000046c3071 clang::ASTDeclWriter::VisitLinkageSpecDecl(clang::LinkageSpecDecl*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriterDecl.cpp:1265:3
 #12 0x00000000046caf39 clang::declvisitor::Base<std::__2::add_pointer, clang::ASTDeclWriter, void>::Visit(clang::Decl*) /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/interpreter/llvm/src/tools/clang/include/clang/AST/DeclNodes.inc:95:1
 #13 0x00000000046bd5b1 clang::ASTDeclWriter::Visit(clang::Decl*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriterDecl.cpp:327:53
 #14 0x00000000046cab4a clang::ASTWriter::WriteDecl(clang::ASTContext&, clang::Decl*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriterDecl.cpp:2481:28
 #15 0x0000000004652de7 clang::ASTWriter::WriteASTCore(clang::Sema&, llvm::StringRef, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&, clang::Module*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp:4687:5
 #16 0x0000000004650dae clang::ASTWriter::WriteAST(clang::Sema&, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&, clang::Module*, llvm::StringRef, bool, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp:0:7
 #17 0x00000000046efca1 clang::PCHGenerator::HandleTranslationUnit(clang::ASTContext&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Serialization/GeneratePCH.cpp:68:14
 #18 0x00000000042038e3 clang::MultiplexConsumer::HandleTranslationUnit(clang::ASTContext&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/MultiplexConsumer.cpp:291:23
 #19 0x00000000043aecc6 clang::ParseAST(clang::Sema&, bool, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/ParseAST.cpp:178:12
 #20 0x000000000419c5ec clang::ASTFrontendAction::ExecuteAction() /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/FrontendAction.cpp:1060:1
 #21 0x000000000419c01c clang::FrontendAction::Execute() /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/FrontendAction.cpp:955:7
 #22 0x00000000040de3f6 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:974:23
 #23 0x00000000040e52dd compileModuleImpl(clang::CompilerInstance&, clang::SourceLocation, llvm::StringRef, clang::FrontendInputFile, llvm::StringRef, llvm::StringRef, llvm::function_ref<void (clang::CompilerInstance&)>, llvm::function_ref<void (clang::CompilerInstance&)>)::$_4::operator()() const /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1168:7
 #24 0x00000000040e52a5 void llvm::function_ref<void ()>::callback_fn<compileModuleImpl(clang::CompilerInstance&, clang::SourceLocation, llvm::StringRef, clang::FrontendInputFile, llvm::StringRef, llvm::StringRef, llvm::function_ref<void (clang::CompilerInstance&)>, llvm::function_ref<void (clang::CompilerInstance&)>)::$_4>(long) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/include/llvm/ADT/STLExtras.h:177:5
 #25 0x00000000043d3c19 llvm::function_ref<void ()>::operator()() const /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/include/llvm/ADT/STLExtras.h:200:5
 #26 0x0000000008fddacb llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/CrashRecoveryContext.cpp:426:3
 #27 0x0000000008fdde5f RunSafelyOnThread_Dispatch(void*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/CrashRecoveryContext.cpp:498:29
 #28 0x0000000008fddeb5 llvm::thread::thread<void (&)(void*), (anonymous namespace)::RunSafelyOnThreadInfo*>(llvm::Optional<unsigned int>, void (&)(void*), (anonymous namespace)::RunSafelyOnThreadInfo*&&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/include/llvm/Support/thread.h:232:3
 #29 0x0000000008fdddca llvm::CrashRecoveryContext::RunSafelyOnThread(llvm::function_ref<void ()>, unsigned int) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/CrashRecoveryContext.cpp:508:10
 #30 0x00000000040e39d6 compileModuleImpl(clang::CompilerInstance&, clang::SourceLocation, llvm::StringRef, clang::FrontendInputFile, llvm::StringRef, llvm::StringRef, llvm::function_ref<void (clang::CompilerInstance&)>, llvm::function_ref<void (clang::CompilerInstance&)>) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1164:7
 #31 0x00000000040e46c6 compileModule(clang::CompilerInstance&, clang::SourceLocation, clang::Module*, llvm::StringRef) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1225:12
 #32 0x00000000040e1108 compileModuleAndReadAST(clang::CompilerInstance&, clang::SourceLocation, clang::SourceLocation, clang::Module*, llvm::StringRef) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1305:11
 #33 0x00000000040e0bb4 clang::CompilerInstance::findOrCompileModuleAndReadAST(llvm::StringRef, clang::SourceLocation, clang::SourceLocation, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1834:7
 #34 0x00000000040e16ad clang::CompilerInstance::loadModule(clang::SourceLocation, llvm::ArrayRef<std::__2::pair<clang::IdentifierInfo*, clang::SourceLocation> >, clang::Module::NameVisibilityKind, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1888:31
 #35 0x0000000008afe5da clang::Preprocessor::HandleHeaderIncludeOrImport(clang::SourceLocation, clang::Token&, clang::Token&, clang::SourceLocation, clang::DirectoryLookup const*, clang::FileEntry const*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/PPDirectives.cpp:2088:49
 #36 0x0000000008af9c5a clang::Preprocessor::HandleIncludeDirective(clang::SourceLocation, clang::Token&, clang::DirectoryLookup const*, clang::FileEntry const*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/PPDirectives.cpp:1792:17
 #37 0x0000000008afa308 clang::Preprocessor::HandleDirective(clang::Token&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/PPDirectives.cpp:1094:14
 #38 0x0000000008abe5f3 clang::Lexer::LexTokenInternal(clang::Token&, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/Lexer.cpp:4013:7
 #39 0x0000000008abac44 clang::Lexer::Lex(clang::Token&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/Lexer.cpp:3228:8
 #40 0x0000000008b4686f clang::Preprocessor::Lex(clang::Token&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/Preprocessor.cpp:931:21
 #41 0x00000000043ce55a clang::Parser::TryConsumeToken(clang::tok::TokenKind) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/include/clang/Parse/Parser.h:532:5
 #42 0x00000000044ab797 clang::Parser::ExpectAndConsumeSemi(unsigned int) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:162:7
 #43 0x00000000043be923 clang::Parser::ParseDeclGroup(clang::ParsingDeclSpec&, clang::DeclaratorContext, clang::SourceLocation*, clang::Parser::ForRangeInit*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/ParseDecl.cpp:2151:7
 #44 0x00000000044b103f clang::Parser::ParseDeclOrFunctionDefInternal(clang::ParsedAttributesWithRange&, clang::ParsingDeclSpec&, clang::AccessSpecifier) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:1146:10
 #45 0x00000000044b0710 clang::Parser::ParseDeclarationOrFunctionDefinition(clang::ParsedAttributesWithRange&, clang::ParsingDeclSpec*, clang::AccessSpecifier) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:1162:12
 #46 0x00000000044b004c clang::Parser::ParseExternalDeclaration(clang::ParsedAttributesWithRange&, clang::ParsingDeclSpec*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:984:12
 #47 0x00000000043e8d99 clang::Parser::ParseLinkage(clang::ParsingDeclSpec&, clang::DeclaratorContext) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/ParseDeclCXX.cpp:399:7
 #48 0x00000000044b0ff7 clang::Parser::ParseDeclOrFunctionDefInternal(clang::ParsedAttributesWithRange&, clang::ParsingDeclSpec&, clang::AccessSpecifier) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:1142:21
 #49 0x00000000044b0710 clang::Parser::ParseDeclarationOrFunctionDefinition(clang::ParsedAttributesWithRange&, clang::ParsingDeclSpec*, clang::AccessSpecifier) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:1162:12
 #50 0x00000000044b004c clang::Parser::ParseExternalDeclaration(clang::ParsedAttributesWithRange&, clang::ParsingDeclSpec*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:984:12
 #51 0x00000000043e8d99 clang::Parser::ParseLinkage(clang::ParsingDeclSpec&, clang::DeclaratorContext) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/ParseDeclCXX.cpp:399:7
 #52 0x00000000044b0ff7 clang::Parser::ParseDeclOrFunctionDefInternal(clang::ParsedAttributesWithRange&, clang::ParsingDeclSpec&, clang::AccessSpecifier) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:1142:21
 #53 0x00000000044b0710 clang::Parser::ParseDeclarationOrFunctionDefinition(clang::ParsedAttributesWithRange&, clang::ParsingDeclSpec*, clang::AccessSpecifier) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:1162:12
 #54 0x00000000044b004c clang::Parser::ParseExternalDeclaration(clang::ParsedAttributesWithRange&, clang::ParsingDeclSpec*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:984:12
 #55 0x00000000044ae43b clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:729:12
 #56 0x00000000044adce3 clang::Parser::ParseFirstTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:601:8
 #57 0x00000000043aeb42 clang::ParseAST(clang::Sema&, bool, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/ParseAST.cpp:157:15
 #58 0x000000000419c5ec clang::ASTFrontendAction::ExecuteAction() /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/FrontendAction.cpp:1060:1
 #59 0x000000000419c01c clang::FrontendAction::Execute() /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/FrontendAction.cpp:955:7
 #60 0x00000000040de3f6 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:974:23
 #61 0x00000000040e52dd compileModuleImpl(clang::CompilerInstance&, clang::SourceLocation, llvm::StringRef, clang::FrontendInputFile, llvm::StringRef, llvm::StringRef, llvm::function_ref<void (clang::CompilerInstance&)>, llvm::function_ref<void (clang::CompilerInstance&)>)::$_4::operator()() const /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1168:7
 #62 0x00000000040e52a5 void llvm::function_ref<void ()>::callback_fn<compileModuleImpl(clang::CompilerInstance&, clang::SourceLocation, llvm::StringRef, clang::FrontendInputFile, llvm::StringRef, llvm::StringRef, llvm::function_ref<void (clang::CompilerInstance&)>, llvm::function_ref<void (clang::CompilerInstance&)>)::$_4>(long) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/include/llvm/ADT/STLExtras.h:177:5
 #63 0x00000000043d3c19 llvm::function_ref<void ()>::operator()() const /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/include/llvm/ADT/STLExtras.h:200:5
 #64 0x0000000008fddacb llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/CrashRecoveryContext.cpp:426:3
 #65 0x0000000008fdde5f RunSafelyOnThread_Dispatch(void*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/CrashRecoveryContext.cpp:498:29
 #66 0x0000000008fddeb5 llvm::thread::thread<void (&)(void*), (anonymous namespace)::RunSafelyOnThreadInfo*>(llvm::Optional<unsigned int>, void (&)(void*), (anonymous namespace)::RunSafelyOnThreadInfo*&&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/include/llvm/Support/thread.h:232:3
 #67 0x0000000008fdddca llvm::CrashRecoveryContext::RunSafelyOnThread(llvm::function_ref<void ()>, unsigned int) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/CrashRecoveryContext.cpp:508:10
 #68 0x00000000040e39d6 compileModuleImpl(clang::CompilerInstance&, clang::SourceLocation, llvm::StringRef, clang::FrontendInputFile, llvm::StringRef, llvm::StringRef, llvm::function_ref<void (clang::CompilerInstance&)>, llvm::function_ref<void (clang::CompilerInstance&)>) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1164:7
 #69 0x00000000040e46c6 compileModule(clang::CompilerInstance&, clang::SourceLocation, clang::Module*, llvm::StringRef) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1225:12
 #70 0x00000000040e1108 compileModuleAndReadAST(clang::CompilerInstance&, clang::SourceLocation, clang::SourceLocation, clang::Module*, llvm::StringRef) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1305:11
 #71 0x00000000040e0bb4 clang::CompilerInstance::findOrCompileModuleAndReadAST(llvm::StringRef, clang::SourceLocation, clang::SourceLocation, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1834:7
 #72 0x00000000040e16ad clang::CompilerInstance::loadModule(clang::SourceLocation, llvm::ArrayRef<std::__2::pair<clang::IdentifierInfo*, clang::SourceLocation> >, clang::Module::NameVisibilityKind, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1888:31
 #73 0x0000000008afe5da clang::Preprocessor::HandleHeaderIncludeOrImport(clang::SourceLocation, clang::Token&, clang::Token&, clang::SourceLocation, clang::DirectoryLookup const*, clang::FileEntry const*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/PPDirectives.cpp:2088:49
 #74 0x0000000008af9c5a clang::Preprocessor::HandleIncludeDirective(clang::SourceLocation, clang::Token&, clang::DirectoryLookup const*, clang::FileEntry const*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/PPDirectives.cpp:1792:17
 #75 0x0000000008afc7e9 clang::Preprocessor::HandleIncludeNextDirective(clang::SourceLocation, clang::Token&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/PPDirectives.cpp:2414:3
 #76 0x0000000008afa45d clang::Preprocessor::HandleDirective(clang::Token&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/PPDirectives.cpp:1121:14
 #77 0x0000000008abe5f3 clang::Lexer::LexTokenInternal(clang::Token&, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/Lexer.cpp:4013:7
 #78 0x0000000008abac44 clang::Lexer::Lex(clang::Token&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/Lexer.cpp:3228:8
 #79 0x0000000008b4686f clang::Preprocessor::Lex(clang::Token&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/Preprocessor.cpp:931:21
 #80 0x000000000375c7f6 clang::Parser::ConsumeBrace() /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/include/clang/Parse/Parser.h:664:8
 #81 0x00000000043ced8d clang::BalancedDelimiterTracker::consumeClose() /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/include/clang/Parse/RAIIObjectsForParser.h:448:18
 #82 0x00000000043e872e clang::Parser::ParseInnerNamespace(llvm::SmallVector<clang::Parser::InnerNamespaceInfo, 4u> const&, unsigned int, clang::SourceLocation&, clang::ParsedAttributes&, clang::BalancedDelimiterTracker&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/ParseDeclCXX.cpp:253:5
 #83 0x00000000043e812a clang::Parser::ParseNamespace(clang::DeclaratorContext, clang::SourceLocation&, clang::SourceLocation) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/ParseDeclCXX.cpp:227:18
 #84 0x00000000043b8d85 clang::Parser::ParseDeclaration(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributesWithRange&, clang::SourceLocation*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/ParseDecl.cpp:1731:12
 #85 0x00000000044afbca clang::Parser::ParseExternalDeclaration(clang::ParsedAttributesWithRange&, clang::ParsingDeclSpec*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:918:14
 #86 0x00000000043e8d99 clang::Parser::ParseLinkage(clang::ParsingDeclSpec&, clang::DeclaratorContext) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/ParseDeclCXX.cpp:399:7
 #87 0x00000000044b0ff7 clang::Parser::ParseDeclOrFunctionDefInternal(clang::ParsedAttributesWithRange&, clang::ParsingDeclSpec&, clang::AccessSpecifier) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:1142:21
 #88 0x00000000044b0710 clang::Parser::ParseDeclarationOrFunctionDefinition(clang::ParsedAttributesWithRange&, clang::ParsingDeclSpec*, clang::AccessSpecifier) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:1162:12
 #89 0x00000000044b004c clang::Parser::ParseExternalDeclaration(clang::ParsedAttributesWithRange&, clang::ParsingDeclSpec*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:984:12
 #90 0x00000000044ae43b clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:729:12
 #91 0x00000000044adce3 clang::Parser::ParseFirstTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:601:8
 #92 0x00000000043aeb42 clang::ParseAST(clang::Sema&, bool, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/ParseAST.cpp:157:15
 #93 0x000000000419c5ec clang::ASTFrontendAction::ExecuteAction() /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/FrontendAction.cpp:1060:1
 #94 0x000000000419c01c clang::FrontendAction::Execute() /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/FrontendAction.cpp:955:7
 #95 0x00000000040de3f6 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:974:23
 #96 0x00000000040e52dd compileModuleImpl(clang::CompilerInstance&, clang::SourceLocation, llvm::StringRef, clang::FrontendInputFile, llvm::StringRef, llvm::StringRef, llvm::function_ref<void (clang::CompilerInstance&)>, llvm::function_ref<void (clang::CompilerInstance&)>)::$_4::operator()() const /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1168:7
 #97 0x00000000040e52a5 void llvm::function_ref<void ()>::callback_fn<compileModuleImpl(clang::CompilerInstance&, clang::SourceLocation, llvm::StringRef, clang::FrontendInputFile, llvm::StringRef, llvm::StringRef, llvm::function_ref<void (clang::CompilerInstance&)>, llvm::function_ref<void (clang::CompilerInstance&)>)::$_4>(long) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/include/llvm/ADT/STLExtras.h:177:5
 #98 0x00000000043d3c19 llvm::function_ref<void ()>::operator()() const /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/include/llvm/ADT/STLExtras.h:200:5
 #99 0x0000000008fddacb llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/CrashRecoveryContext.cpp:426:3
#100 0x0000000008fdde5f RunSafelyOnThread_Dispatch(void*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/CrashRecoveryContext.cpp:498:29
#101 0x0000000008fddeb5 llvm::thread::thread<void (&)(void*), (anonymous namespace)::RunSafelyOnThreadInfo*>(llvm::Optional<unsigned int>, void (&)(void*), (anonymous namespace)::RunSafelyOnThreadInfo*&&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/include/llvm/Support/thread.h:232:3
#102 0x0000000008fdddca llvm::CrashRecoveryContext::RunSafelyOnThread(llvm::function_ref<void ()>, unsigned int) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/CrashRecoveryContext.cpp:508:10
#103 0x00000000040e39d6 compileModuleImpl(clang::CompilerInstance&, clang::SourceLocation, llvm::StringRef, clang::FrontendInputFile, llvm::StringRef, llvm::StringRef, llvm::function_ref<void (clang::CompilerInstance&)>, llvm::function_ref<void (clang::CompilerInstance&)>) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1164:7
#104 0x00000000040e46c6 compileModule(clang::CompilerInstance&, clang::SourceLocation, clang::Module*, llvm::StringRef) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1225:12
#105 0x00000000040e1108 compileModuleAndReadAST(clang::CompilerInstance&, clang::SourceLocation, clang::SourceLocation, clang::Module*, llvm::StringRef) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1305:11
#106 0x00000000040e0bb4 clang::CompilerInstance::findOrCompileModuleAndReadAST(llvm::StringRef, clang::SourceLocation, clang::SourceLocation, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1834:7
#107 0x00000000040e16ad clang::CompilerInstance::loadModule(clang::SourceLocation, llvm::ArrayRef<std::__2::pair<clang::IdentifierInfo*, clang::SourceLocation> >, clang::Module::NameVisibilityKind, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Frontend/CompilerInstance.cpp:1888:31
#108 0x0000000008afe5da clang::Preprocessor::HandleHeaderIncludeOrImport(clang::SourceLocation, clang::Token&, clang::Token&, clang::SourceLocation, clang::DirectoryLookup const*, clang::FileEntry const*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/PPDirectives.cpp:2088:49
#109 0x0000000008af9c5a clang::Preprocessor::HandleIncludeDirective(clang::SourceLocation, clang::Token&, clang::DirectoryLookup const*, clang::FileEntry const*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/PPDirectives.cpp:1792:17
#110 0x0000000008afa308 clang::Preprocessor::HandleDirective(clang::Token&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/PPDirectives.cpp:1094:14
#111 0x0000000008abe5f3 clang::Lexer::LexTokenInternal(clang::Token&, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/Lexer.cpp:4013:7
#112 0x0000000008abac44 clang::Lexer::Lex(clang::Token&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/Lexer.cpp:3228:8
#113 0x0000000008b4686f clang::Preprocessor::Lex(clang::Token&) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Lex/Preprocessor.cpp:931:21
#114 0x000000000375b20c clang::Parser::ConsumeToken() /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/include/clang/Parse/Parser.h:521:8
#115 0x00000000044adde9 clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Parse/Parser.cpp:621:5
#116 0x0000000003454eef cling::IncrementalParser::ParseInternal(llvm::StringRef) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/cling/lib/Interpreter/IncrementalParser.cpp:929:23
#117 0x000000000345433e cling::IncrementalParser::Initialize(llvm::SmallVectorImpl<llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > > >&, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/cling/lib/Interpreter/IncrementalParser.cpp:399:7
#118 0x000000000342fc92 cling::Interpreter::Interpreter(int, char const* const*, char const*, std::__2::vector<std::__2::shared_ptr<clang::ModuleFileExtension>, std::__2::allocator<std::__2::shared_ptr<clang::ModuleFileExtension> > > const&, void*, bool, cling::Interpreter const*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/cling/lib/Interpreter/Interpreter.cpp:295:24
#119 0x000000000334d2fa cling::Interpreter::Interpreter(int, char const* const*, char const*, std::__2::vector<std::__2::shared_ptr<clang::ModuleFileExtension>, std::__2::allocator<std::__2::shared_ptr<clang::ModuleFileExtension> > > const&, void*, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/cling/include/cling/Interpreter/Interpreter.h:364:44
#120 0x000000000333797d RootClingMain(int, char**, bool) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/dictgen/src/rootcling_impl.cxx:4330:33
#121 0x0000000003344a4a ROOT_rootcling_Driver /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/dictgen/src/rootcling_impl.cxx:6165:16
#122 0x00000000034038ca main /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/rootcling_stage1/src/rootcling_stage1.cxx:46:4
#123 0x00007ff582580555 __libc_start_main (/lib64/libc.so.6+0x22555)
#124 0x00000000031d8aa9 _start (/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/core/rootcling_stage1/src/rootcling_stage1+0x31d8aa9)
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.  Program arguments: /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/core/rootcling_stage1/src/rootcling_stage1 -v4 -f G__Core.cxx -cxxmodule -s /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/lib/libCore.so -excludePath /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99 -excludePath /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude -excludePath /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/externals -excludePath /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/builtins -writeEmptyRootPCM -m _Builtin_intrinsics -mByproduct _Builtin_intrinsics -mByproduct ROOT_Foundation_Stage1_NoRTTI -mByproduct ROOT_Foundation_C -mByproduct ROOT_Rtypes -D__CORRECT_ISO_CPP_WCHAR_H_PROTO -D__STDC_NO_COMPLEX__ -D__COMPLEX_H__ -D_COMPLEX_H -D__CLANG_STDATOMIC_H -compilerI/scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/x86_64-unknown-linux-gnu/c++/v1 -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/c++/v1 -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib/clang/14.0.6/include -compilerI/usr/local/include -compilerI/usr/include -compilerI/scratch/greenc/test-products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include -compilerI/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib/clang/14.0.6/include -compilerI/usr/local/include -compilerI/usr/include -I/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/include -I/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude -I/scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/ginclude -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/cont/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/gui/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/meta/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clib/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/rint/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/zip/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/thread/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/textinput/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/clingutils/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/v7/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/foundation/v7/inc -I/scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/unix/inc -I/usr/include ROOT/TErrorDefaultHandler.hxx ROOT/TSequentialExecutor.hxx ROOT/StringConv.hxx Buttons.h Bytes.h Byteswap.h KeySymbols.h MessageTypes.h Riostream.h Rtypes.h TApplication.h TAtt3D.h TAttAxis.h TAttBBox2D.h TAttBBox.h TAttFill.h TAttLine.h TAttMarker.h TAttPad.h TAttText.h TBase64.h TBenchmark.h TBuffer3D.h TBuffer3DTypes.h TBuffer.h TColor.h TColorGradient.h TDatime.h TDirectory.h TEnv.h TException.h TExec.h TFileCollection.h TFileInfo.h TFolder.h TInetAddress.h TMacro.h TMathBase.h TMD5.h TMemberInspector.h TMessageHandler.h TNamed.h TNotifyLink.h TObject.h TObjString.h TParameter.h TPluginManager.h TPoint.h TPRegexp.h TProcessID.h TProcessUUID.h TQClass.h TQCommand.h TQConnection.h TQObject.h TRedirectOutputGuard.h TRefCnt.h TRef.h TRegexp.h TRemoteObject.h TROOT.h TRootIOCtor.h TStopwatch.h TStorage.h TString.h TStringLong.h TStyle.h TSysEvtHandler.h TSystemDirectory.h TSystemFile.h TSystem.h TTask.h TThreadSlots.h TTime.h TTimer.h TTimeStamp.h TUri.h TUrl.h TUUID.h TVersionCheck.h TVirtualAuth.h TVirtualFFT.h TVirtualGL.h TVirtualMonitoring.h TVirtualMutex.h TVirtualPadEditor.h TVirtualPad.h TVirtualPadPainter.h TVirtualPerfStats.h TVirtualPS.h TVirtualQConnection.h TVirtualRWMutex.h TVirtualTableInterface.h TVirtualViewer3D.h TVirtualX.h ROOT/RDirectoryEntry.hxx ROOT/RIndexIter.hxx strlcpy.h snprintf.h strtok.h ROOT/TSeq.hxx TArrayC.h TArrayD.h TArrayF.h TArray.h TArrayI.h TArrayL64.h TArrayL.h TArrayS.h TBits.h TBtree.h TClassTable.h TClonesArray.h TCollection.h TCollectionProxyInfo.h TExMap.h THashList.h THashTable.h TIterator.h TList.h TMap.h TObjArray.h TObjectTable.h TOrdCollection.h TRefArray.h TRefTable.h TSeqCollection.h TSortedList.h TVirtualCollectionProxy.h ESTLType.h RStringView.h TClassEdit.h TError.h ThreadLocalStorage.h ROOT/RLogger.hxx ROOT/RNotFn.hxx ROOT/RRangeCast.hxx ROOT/RSpan.hxx ROOT/RStringView.hxx ROOT/StringUtils.hxx ROOT/span.hxx ROOT/TypeTraits.hxx TUnixSystem.h root_std_complex.h GuiTypes.h TApplicationImp.h TBrowser.h TBrowserImp.h TCanvasImp.h TClassMenuItem.h TContextMenu.h TContextMenuImp.h TControlBarImp.h TGuiFactory.h TInspectorImp.h TObjectSpy.h TToggleGroup.h TToggle.h TBaseClass.h TClassGenerator.h TClass.h TClassRef.h TClassStreamer.h TDataMember.h TDataType.h TDictAttributeMap.h TDictionary.h TEnumConstant.h TEnum.h TFileMergeInfo.h TFunction.h TFunctionTemplate.h TGenericClassInfo.h TGlobal.h TInterpreter.h TInterpreterValue.h TIsAProxy.h TListOfDataMembers.h TListOfEnums.h TListOfEnumsWithLock.h TListOfFunctions.h TListOfFunctionTemplates.h TMemberStreamer.h TMethodArg.h TMethodCall.h TMethod.h TProtoClass.h TRealData.h TSchemaHelper.h TSchemaRule.h TSchemaRuleSet.h TStatusBitsChecker.h TStreamerElement.h TStreamer.h TVirtualIsAProxy.h TVirtualRefProxy.h TVirtualStreamerInfo.h TVirtualArray.h TVirtualObject.h Getline.h /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/core/base/inc/LinkDef.h
1.  /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/c++/v1/ctype.h:38:2: current parser token 'include_next'
2.  /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/c++/v1/__config:827:1 <Spelling=/scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/c++/v1/__config:824:37>: parsing namespace 'std'
3.  /usr/include/string.h:159:3: current parser token 'include'
4.  <eof> parser at end of file
5.  <module-includes>:1:8: serializing
Segmentation fault

greenc-FNAL commented 1 year ago

Correction: getSubmoduleID() was called, not sure where the output went. Checking now...

greenc-FNAL commented 1 year ago

Output from printf()is apparently being suppressed, from where is unclear. ~Switching to fprintf(stderr,...) also not helpful. Suggestions for next steps, please?~

greenc-FNAL commented 1 year ago

Possible inconsistent partial compilation: retrying with fprintf(stderr,) and greenfield build now.

greenc-FNAL commented 1 year ago

OK: Mod is not valid, so with the diagnostic the error occurs in Mod->Name rather than the assert() in the following line: hence, no diagnostic indicating the submodule.

greenc-FNAL commented 1 year ago

Indeed, applying the following patch:

--- interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp~   2023-01-10 11:09:06.666075151 -0600
+++ interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp    2023-01-10 11:34:45.644464781 -0600
@@ -105,6 +105,7 @@
 #include <algorithm>
 #include <cassert>
 #include <cstdint>
+#include <cstdio>
 #include <cstdlib>
 #include <cstring>
 #include <ctime>
@@ -2498,6 +2499,8 @@
   // did not result in us loading a module file for that submodule. For
   // instance, a cross-top-level-module 'conflict' declaration will hit this.
   unsigned ID = getLocalOrImportedSubmoduleID(Mod);
+  fprintf(stderr, "Mod=%X\n", Mod);
+  fprintf(stderr, "Submodule=%s\n", Mod->Name.c_str());
   assert((ID || !Mod) &&
          "asked for module ID for non-local, non-imported module");
   return ID;

gives:

<snip/>
End of search list.
Mod=0
  #0 0x0000000009107d6a llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Unix/Signals.inc:565:11
  #1 0x0000000009107f1b PrintStackTraceSignalHandler(void*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Unix/Signals.inc:632:1
  #2 0x0000000009106743 llvm::sys::RunSignalHandlers() /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Signals.cpp:97:5
  #3 0x0000000009108535 SignalHandler(int) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Unix/Signals.inc:407:1
  #4 0x00007f1a911a0630 __restore_rt sigaction.c:0:0
  #5 0x00000000031e3975 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::__is_long() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1445:22
  #6 0x00000000031e3929 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::__get_pointer() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1544:17
  #7 0x00000000031e38e5 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::data() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1251:75
  #8 0x00000000031e1925 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::c_str() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1249:48
  #9 0x0000000004638fe3 clang::ASTWriter::getSubmoduleID(clang::Module*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp:2503:47
<snip/>

Suggestions for next steps, please?

vgvassilev commented 1 year ago

Can we do D->dump() in #10 0x00000000046bdc47 clang::ASTDeclWriter::VisitDecl That would likely produce a lot of output. Also maybe print D->getOwningModule()->Name?

greenc-FNAL commented 1 year ago

Can do.

greenc-FNAL commented 1 year ago

Applied patch:

--- interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriterDecl.cpp~   2023-01-06 05:04:43.000000000 -0600
+++ interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriterDecl.cpp    2023-01-10 13:21:47.554190854 -0600
@@ -26,6 +26,8 @@
 using namespace clang;
 using namespace serialization;

+#include <cstdio>
+
 //===----------------------------------------------------------------------===//
 // Declaration serialization
 //===----------------------------------------------------------------------===//
@@ -361,6 +363,8 @@
   Record.push_back(D->isTopLevelDeclInObjCContainer());
   Record.push_back(D->getAccess());
   Record.push_back(D->isModulePrivate());
+  D->dump();
+  if (D->getOwningModule()) fprintf(stderr, "D->getOwningModule()->Name=%s", D->getOwningModule()->Name.c_str());
   Record.push_back(Writer.getSubmoduleID(D->getOwningModule()));

   // If this declaration injected a name into a context different from its

with result:

<snip/>
LinkageSpecDecl 0xbf7b930 <<module-includes>:1:1, line:3:1> line:1:8 C
|-CXXRecordDecl 0xbf7b988 </usr/include/xlocale.h:27:9, line:39:1> line:27:16 in xlocale.h hidden struct __locale_struct definition
| |-DefinitionData pass_in_registers aggregate standard_layout trivially_copyable pod trivial literal
| | |-DefaultConstructor exists trivial needs_implicit
| | |-CopyConstructor simple trivial has_const_param needs_implicit implicit_has_const_param
| | |-MoveConstructor exists simple trivial needs_implicit
| | |-CopyAssignment simple trivial has_const_param needs_implicit implicit_has_const_param
| | |-MoveAssignment exists simple trivial needs_implicit
| | `-Destructor simple irrelevant trivial needs_implicit
| |-CXXRecordDecl 0xbf7bad0 <col:9, col:16> col:16 in xlocale.h hidden implicit struct __locale_struct
| |-CXXRecordDecl 0xbf7bb88 parent 0xbf3f498 <line:30:3, col:10> col:10 in xlocale.h hidden struct __locale_data
| |-FieldDecl 0xbf7be08 <col:3, col:37> col:25 in xlocale.h hidden __locales 'struct __locale_data *[13]'
| |-FieldDecl 0xbf7beb8 <line:33:3, col:29> col:29 in xlocale.h hidden __ctype_b 'const unsigned short *'
| |-FieldDecl 0xbf7bf28 <line:34:3, col:14> col:14 in xlocale.h hidden __ctype_tolower 'const int *'
| |-FieldDecl 0xbf7bf98 <line:35:3, col:14> col:14 in xlocale.h hidden __ctype_toupper 'const int *'
| `-FieldDecl 0xbf7c098 <line:38:3, col:25> col:15 in xlocale.h hidden __names 'const char *[13]'
|-TypedefDecl 0xbf7c1c8 <line:27:1, line:39:4> col:4 in xlocale.h hidden referenced __locale_t 'struct __locale_struct *'
| `-PointerType 0xbf7c170 'struct __locale_struct *'
|   `-ElaboratedType 0xbf7c100 'struct __locale_struct' sugar
|     `-RecordType 0xbf7ba40 '__locale_struct'
|       `-CXXRecord 0xbf7b988 '__locale_struct'
`-TypedefDecl 0xbf7c278 <line:42:1, col:20> col:20 in xlocale.h hidden locale_t '__locale_t':'struct __locale_struct *'
  `-TypedefType 0xbf7c240 '__locale_t' sugar
    |-Typedef 0xbf7c1c8 '__locale_t'
    `-PointerType 0xbf7c170 'struct __locale_struct *'
      `-ElaboratedType 0xbf7c100 'struct __locale_struct' sugar
        `-RecordType 0xbf7ba40 '__locale_struct'
          `-CXXRecord 0xbf7b988 '__locale_struct'
Mod=0
  #0 0x0000000009107dda llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Unix/Signals.inc:565:11
  #1 0x0000000009107f8b PrintStackTraceSignalHandler(void*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Unix/Signals.inc:632:1
  #2 0x00000000091067b3 llvm::sys::RunSignalHandlers() /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Signals.cpp:97:5
  #3 0x00000000091085a5 SignalHandler(int) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Unix/Signals.inc:407:1
  #4 0x00007f7e8feef630 __restore_rt sigaction.c:0:0
  #5 0x00000000031e3995 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::__is_long() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1445:22
  #6 0x00000000031e3949 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::__get_pointer() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1544:17
  #7 0x00000000031e3905 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::data() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1251:75
  #8 0x00000000031e1945 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::c_str() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1249:48
  #9 0x0000000004639003 clang::ASTWriter::getSubmoduleID(clang::Module*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp:2503:47
 #10 0x00000000046bdcea clang::ASTDeclWriter::VisitDecl(clang::Decl*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriterDecl.cpp:368:27
<snip/>

vgvassilev commented 1 year ago

If you apply this patch would it make any difference?

git diff interpreter/cling/include/cling/libc.modulemap | more
diff --git a/interpreter/cling/include/cling/libc.modulemap b/interpreter/cling/include/cling/libc.modulemap
index 87654b1ecf..40d990f3f5 100644
--- a/interpreter/cling/include/cling/libc.modulemap
+++ b/interpreter/cling/include/cling/libc.modulemap
@@ -63,12 +63,9 @@ module "libc" [system] [extern_c] [no_undeclared_includes] {
     header "wchar.h"
   }

-  use "xlocale.h"
+  module "xlocale.h" {
+    export *
+    header "xlocale.h"
+  }
 }

-// glib specific header. In it's own module because it
-// doesn't exist on some systems with unpatched glib 2.26+
-module "xlocale.h" [system] [extern_c] {
-  export *
-  header "xlocale.h"
-}

greenc-FNAL commented 1 year ago

Checking...

greenc-FNAL commented 1 year ago

Definitely gets further:

<snip/>
Mod=AEEAC90
Submodule=pthread.h
Mod=AEEDAF0
Submodule=time.h
Mod=AEEBBD0
Submodule=signal.h
Mod=AEECBB0
Submodule=stdio.h
Mod=AEED350
Submodule=stdlib.h
Mod=AEEBBD0
Submodule=signal.h
Mod=AEED350
Submodule=stdlib.h
Mod=AEEBBD0
Submodule=signal.h
Mod=AEED350
Submodule=stdlib.h
Mod=AEEBBD0
Submodule=signal.h
Mod=AEED350
Submodule=stdlib.h
Mod=AEEBBD0
Submodule=signal.h
Mod=AEED350
Submodule=stdlib.h
Mod=AEEBBD0
Submodule=signal.h
Mod=AEED350
Submodule=stdlib.h
Mod=AEEBBD0
Submodule=signal.h
Mod=AEED350
Submodule=stdlib.h
Mod=AEEBBD0
Submodule=signal.h
Mod=AEED350
Submodule=stdlib.h
Mod=AEEBBD0
Submodule=signal.h
Mod=AEED350
Submodule=stdlib.h
Mod=AEEBBD0
Submodule=signal.h
Mod=AEED350
Submodule=stdlib.h
Mod=AEEBBD0
Submodule=signal.h
Mod=AEED350
Submodule=stdlib.h
Mod=AEEBBD0
Submodule=signal.h
Mod=AEED350
Submodule=stdlib.h
LinkageSpecDecl 0xb598660 <<module-includes>:1:1, line:3:1> line:1:8 C
`-LinkageSpecDecl 0xb5986d8 </usr/include/sys/cdefs.h:99:24, line:100:22> line:99:31 in libc.string.h C
  |-TypedefDecl 0xb598748 </scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug/etc/cling/lib/clang/13.0.0/include/stddef.h:46:1, col:23> col:23 in libc.string.h hidden referenced size_t 'unsigned long'
  | `-BuiltinType 0xb55c390 'unsigned long'
  |-FunctionDecl 0xb598af0 </usr/include/string.h:42:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:42:14 in libc.string.h hidden memcpy 'void *(void *__restrict, const void *__restrict, size_t) throw()' extern
  | |-ParmVarDecl 0xb5987c0 <col:22, col:39> col:39 in libc.string.h hidden __dest 'void *__restrict'
  | |-ParmVarDecl 0xb598878 <col:47, col:70> col:70 in libc.string.h hidden __src 'const void *__restrict'
  | |-ParmVarDecl 0xb598918 <line:43:8, col:15> col:15 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-NonNullAttr 0xb598ba8 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:43:44> 1 2
  | `-BuiltinAttr 0xb598d10 <<invalid sloc>> Implicit 779
  |-FunctionDecl 0xb599000 <line:46:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:46:14 in libc.string.h hidden memmove 'void *(void *, const void *, size_t) throw()' extern
  | |-ParmVarDecl 0xb598d58 <col:23, col:29> col:29 in libc.string.h hidden __dest 'void *'
  | |-ParmVarDecl 0xb598de0 <col:37, col:49> col:49 in libc.string.h hidden __src 'const void *'
  | |-ParmVarDecl 0xb598e60 <col:56, col:63> col:63 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-NonNullAttr 0xb5990b8 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:47:30> 1 2
  | `-BuiltinAttr 0xb599190 <<invalid sloc>> Implicit 781
  |-FunctionDecl 0xb5e6cf0 <line:54:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:54:14 in libc.string.h hidden memccpy 'void *(void *__restrict, const void *__restrict, int, size_t) throw()' extern
  | |-ParmVarDecl 0xb5991d8 <col:23, col:40> col:40 in libc.string.h hidden __dest 'void *__restrict'
  | |-ParmVarDecl 0xb599260 <col:48, col:71> col:71 in libc.string.h hidden __src 'const void *__restrict'
  | |-ParmVarDecl 0xb5e6a58 <line:55:9, col:13> col:13 in libc.string.h hidden __c 'int'
  | |-ParmVarDecl 0xb5e6ad8 <col:18, col:25> col:25 in libc.string.h hidden __n 'size_t':'unsigned long'
  | `-NonNullAttr 0xb5e6db0 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:56:30> 1 2
  |-FunctionDecl 0xb5e7120 <line:62:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:62:14 in libc.string.h hidden memset 'void *(void *, int, size_t) throw()' extern
  | |-ParmVarDecl 0xb5e6e58 <col:22, col:28> col:28 in libc.string.h hidden __s 'void *'
  | |-ParmVarDecl 0xb5e6ee0 <col:33, col:37> col:37 in libc.string.h hidden __c 'int'
  | |-ParmVarDecl 0xb5e6f60 <col:42, col:49> col:49 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-NonNullAttr 0xb5e71d8 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:62:75> 1
  | `-BuiltinAttr 0xb5e72f0 <<invalid sloc>> Implicit 797
  |-FunctionDecl 0xb5e7620 <line:65:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:65:12 in libc.string.h hidden memcmp 'int (const void *, const void *, size_t) throw()' extern
  | |-ParmVarDecl 0xb5e7338 <col:20, col:32> col:32 in libc.string.h hidden __s1 'const void *'
  | |-ParmVarDecl 0xb5e73c0 <col:38, col:50> col:50 in libc.string.h hidden __s2 'const void *'
  | |-ParmVarDecl 0xb5e7440 <col:56, col:63> col:63 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-PureAttr 0xb5e76d8 </usr/include/sys/cdefs.h:229:45>
  | |-NonNullAttr 0xb5e7730 <line:285:44, /usr/include/string.h:66:49> 1 2
  | `-BuiltinAttr 0xb5e7810 <<invalid sloc>> Implicit 780
  |-FunctionDecl 0xb5e7b30 <line:92:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:92:14 in libc.string.h hidden memchr 'void *(const void *, int, size_t) throw()' extern
  | |-ParmVarDecl 0xb5e7858 <col:22, col:34> col:34 in libc.string.h hidden __s 'const void *'
  | |-ParmVarDecl 0xb5e78e0 <col:39, col:43> col:43 in libc.string.h hidden __c 'int'
  | |-ParmVarDecl 0xb5e7960 <col:48, col:55> col:55 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-PureAttr 0xb5e7be8 </usr/include/sys/cdefs.h:229:45>
  | |-NonNullAttr 0xb5e7c40 <line:285:44, /usr/include/string.h:93:47> 1
  | `-BuiltinAttr 0xb5e7d20 <<invalid sloc>> Implicit 789
  |-FunctionDecl 0xb5e7f80 <line:106:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:106:14 in libc.string.h hidden rawmemchr 'void *(const void *, int) throw()' extern
  | |-ParmVarDecl 0xb5e7d68 <col:25, col:37> col:37 in libc.string.h hidden __s 'const void *'
  | |-ParmVarDecl 0xb5e7df0 <col:42, col:46> col:46 in libc.string.h hidden __c 'int'
  | |-PureAttr 0xb5e8030 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb5e8088 <line:285:44, /usr/include/string.h:107:46> 1
  |-FunctionDecl 0xb5e8348 <line:117:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:117:14 in libc.string.h hidden memrchr 'void *(const void *, int, size_t) throw()' extern
  | |-ParmVarDecl 0xb5e8108 <col:23, col:35> col:35 in libc.string.h hidden __s 'const void *'
  | |-ParmVarDecl 0xb5e8190 <col:40, col:44> col:44 in libc.string.h hidden __c 'int'
  | |-ParmVarDecl 0xb5e8210 <col:49, col:56> col:56 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-PureAttr 0xb5e8400 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb5e8458 <line:285:44, /usr/include/string.h:118:47> 1
  |-FunctionDecl 0xb5e8710 <line:125:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:125:14 in libc.string.h hidden strcpy 'char *(char *__restrict, const char *__restrict) throw()' extern
  | |-ParmVarDecl 0xb5e84d8 <col:22, col:39> col:39 in libc.string.h hidden __dest 'char *__restrict'
  | |-ParmVarDecl 0xb5e8560 <col:47, col:70> col:70 in libc.string.h hidden __src 'const char *__restrict'
  | |-NonNullAttr 0xb5e87c0 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:126:30> 1 2
  | `-BuiltinAttr 0xb5e88c8 <<invalid sloc>> Implicit 782
  |-FunctionDecl 0xb5e8c50 <line:128:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:128:14 in libc.string.h hidden strncpy 'char *(char *__restrict, const char *__restrict, size_t) throw()' extern
  | |-ParmVarDecl 0xb5e8910 <col:23, col:40> col:40 in libc.string.h hidden __dest 'char *__restrict'
  | |-ParmVarDecl 0xb5e8998 <line:129:9, col:32> col:32 in libc.string.h hidden __src 'const char *__restrict'
  | |-ParmVarDecl 0xb5e8a78 <col:39, col:46> col:46 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-NonNullAttr 0xb5e8d08 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:130:30> 1 2
  | `-BuiltinAttr 0xb5e8e20 <<invalid sloc>> Implicit 783
  |-FunctionDecl 0xb5e9020 <line:133:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:133:14 in libc.string.h hidden strcat 'char *(char *__restrict, const char *__restrict) throw()' extern
  | |-ParmVarDecl 0xb5e8e68 <col:22, col:39> col:39 in libc.string.h hidden __dest 'char *__restrict'
  | |-ParmVarDecl 0xb5e8ef0 <col:47, col:70> col:70 in libc.string.h hidden __src 'const char *__restrict'
  | |-NonNullAttr 0xb5e90d0 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:134:30> 1 2
  | `-BuiltinAttr 0xb5e9158 <<invalid sloc>> Implicit 786
  |-FunctionDecl 0xb5e9400 <line:136:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:136:14 in libc.string.h hidden strncat 'char *(char *__restrict, const char *__restrict, size_t) throw()' extern
  | |-ParmVarDecl 0xb5e91a0 <col:23, col:40> col:40 in libc.string.h hidden __dest 'char *__restrict'
  | |-ParmVarDecl 0xb5e9228 <col:48, col:71> col:71 in libc.string.h hidden __src 'const char *__restrict'
  | |-ParmVarDecl 0xb5e92a8 <line:137:9, col:16> col:16 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-NonNullAttr 0xb5e94b8 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:137:45> 1 2
  | `-BuiltinAttr 0xb5e9548 <<invalid sloc>> Implicit 787
  |-FunctionDecl 0xb5e97c0 <line:140:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:140:12 in libc.string.h hidden strcmp 'int (const char *, const char *) throw()' extern
  | |-ParmVarDecl 0xb5e9590 <col:20, col:32> col:32 in libc.string.h hidden __s1 'const char *'
  | |-ParmVarDecl 0xb5e9618 <col:38, col:50> col:50 in libc.string.h hidden __s2 'const char *'
  | |-PureAttr 0xb5e9870 </usr/include/sys/cdefs.h:229:45>
  | |-NonNullAttr 0xb5e98c8 <line:285:44, /usr/include/string.h:141:49> 1 2
  | `-BuiltinAttr 0xb5e9968 <<invalid sloc>> Implicit 784
  |-FunctionDecl 0xb5e9ce0 <line:143:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:143:12 in libc.string.h hidden strncmp 'int (const char *, const char *, size_t) throw()' extern
  | |-ParmVarDecl 0xb5e99b0 <col:21, col:33> col:33 in libc.string.h hidden __s1 'const char *'
  | |-ParmVarDecl 0xb5e9a88 <col:39, col:51> col:51 in libc.string.h hidden __s2 'const char *'
  | |-ParmVarDecl 0xb5e9b08 <col:57, col:64> col:64 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-PureAttr 0xb5e9d98 </usr/include/sys/cdefs.h:229:45>
  | |-NonNullAttr 0xb5e9df0 <line:285:44, /usr/include/string.h:144:49> 1 2
  | `-BuiltinAttr 0xb5e9ed0 <<invalid sloc>> Implicit 785
  |-FunctionDecl 0xb5ea0d0 <line:147:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:147:12 in libc.string.h hidden strcoll 'int (const char *, const char *) throw()' extern
  | |-ParmVarDecl 0xb5e9f18 <col:21, col:33> col:33 in libc.string.h hidden __s1 'const char *'
  | |-ParmVarDecl 0xb5e9fa0 <col:39, col:51> col:51 in libc.string.h hidden __s2 'const char *'
  | |-PureAttr 0xb5ea180 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb5ea1d8 <line:285:44, /usr/include/string.h:148:49> 1 2
  |-FunctionDecl 0xb5ea518 <line:150:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:150:15 in libc.string.h hidden strxfrm 'size_t (char *__restrict, const char *__restrict, size_t) throw()' extern
  | |-ParmVarDecl 0xb5ea258 <col:24, col:41> col:41 in libc.string.h hidden __dest 'char *__restrict'
  | |-ParmVarDecl 0xb5ea2e0 <line:151:10, col:33> col:33 in libc.string.h hidden __src 'const char *__restrict'
  | |-ParmVarDecl 0xb5ea360 <col:40, col:47> col:47 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-NonNullAttr 0xb5ea5d0 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:152:27> 2
  | `-BuiltinAttr 0xb5ea6e0 <<invalid sloc>> Implicit 788
  |-CXXRecordDecl 0xb5ea710 </usr/include/xlocale.h:27:9, line:39:1> line:27:16 in libc.xlocale.h hidden struct __locale_struct definition
  | |-DefinitionData pass_in_registers aggregate standard_layout trivially_copyable pod trivial literal
  | | |-DefaultConstructor exists trivial needs_implicit
  | | |-CopyConstructor simple trivial has_const_param needs_implicit implicit_has_const_param
  | | |-MoveConstructor exists simple trivial needs_implicit
  | | |-CopyAssignment simple trivial has_const_param needs_implicit implicit_has_const_param
  | | |-MoveAssignment exists simple trivial needs_implicit
  | | `-Destructor simple irrelevant trivial needs_implicit
  | |-CXXRecordDecl 0xb5ea850 <col:9, col:16> col:16 in libc.xlocale.h hidden implicit struct __locale_struct
  | |-CXXRecordDecl 0xb5ea908 parent 0xb55c1c8 <line:30:3, col:10> col:10 in libc.xlocale.h hidden struct __locale_data
  | |-FieldDecl 0xb5ebd48 <col:3, col:37> col:25 in libc.xlocale.h hidden __locales 'struct __locale_data *[13]'
  | |-FieldDecl 0xb5ebdf8 <line:33:3, col:29> col:29 in libc.xlocale.h hidden __ctype_b 'const unsigned short *'
  | |-FieldDecl 0xb5ebe68 <line:34:3, col:14> col:14 in libc.xlocale.h hidden __ctype_tolower 'const int *'
  | |-FieldDecl 0xb5ebed8 <line:35:3, col:14> col:14 in libc.xlocale.h hidden __ctype_toupper 'const int *'
  | `-FieldDecl 0xb5ebfd8 <line:38:3, col:25> col:15 in libc.xlocale.h hidden __names 'const char *[13]'
  |-TypedefDecl 0xb5ec108 <line:27:1, line:39:4> col:4 in libc.xlocale.h hidden referenced __locale_t 'struct __locale_struct *'
  | `-PointerType 0xb5ec0b0 'struct __locale_struct *'
  |   `-ElaboratedType 0xb5ec040 'struct __locale_struct' sugar
  |     `-RecordType 0xb5ea7c0 '__locale_struct'
  |       `-CXXRecord 0xb5ea710 '__locale_struct'
  |-TypedefDecl 0xb5ec1b8 <line:42:1, col:20> col:20 in libc.xlocale.h hidden locale_t '__locale_t':'struct __locale_struct *'
  | `-TypedefType 0xb5ec180 '__locale_t' sugar
  |   |-Typedef 0xb5ec108 '__locale_t'
  |   `-PointerType 0xb5ec0b0 'struct __locale_struct *'
  |     `-ElaboratedType 0xb5ec040 'struct __locale_struct' sugar
  |       `-RecordType 0xb5ea7c0 '__locale_struct'
  |         `-CXXRecord 0xb5ea710 '__locale_struct'
  |-FunctionDecl 0xb5ec580 </usr/include/string.h:162:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:162:12 in libc.string.h hidden strcoll_l 'int (const char *, const char *, __locale_t) throw()' extern
  | |-ParmVarDecl 0xb5ec278 <col:23, col:35> col:35 in libc.string.h hidden __s1 'const char *'
  | |-ParmVarDecl 0xb5ec300 <col:41, col:53> col:53 in libc.string.h hidden __s2 'const char *'
  | |-ParmVarDecl 0xb5ec380 <col:59, col:70> col:70 in libc.string.h hidden __l '__locale_t':'struct __locale_struct *'
  | |-PureAttr 0xb5ec638 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb5ec690 <line:285:44, /usr/include/string.h:163:52> 1 2 3
  |-FunctionDecl 0xb5ecaa8 <line:165:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:165:15 in libc.string.h hidden strxfrm_l 'size_t (char *, const char *, size_t, __locale_t) throw()' extern
  | |-ParmVarDecl 0xb5ec710 <col:26, col:32> col:32 in libc.string.h hidden __dest 'char *'
  | |-ParmVarDecl 0xb5ec798 <col:40, col:52> col:52 in libc.string.h hidden __src 'const char *'
  | |-ParmVarDecl 0xb5ec818 <col:59, col:66> col:66 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-ParmVarDecl 0xb5ec898 <line:166:5, col:16> col:16 in libc.string.h hidden __l '__locale_t':'struct __locale_struct *'
  | `-NonNullAttr 0xb5ecb68 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:166:45> 2 4
  |-FunctionDecl 0xb5f6840 <line:172:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:172:14 in libc.string.h hidden strdup 'char *(const char *) throw()' extern
  | |-ParmVarDecl 0xb5f66f8 <col:22, col:34> col:34 in libc.string.h hidden __s 'const char *'
  | |-RestrictAttr 0xb5f68e8 </usr/include/sys/cdefs.h:220:47> malloc
  | `-NonNullAttr 0xb5f6940 <line:285:44, /usr/include/string.h:173:48> 1
  |-FunctionDecl 0xb5f6bc0 <line:180:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:180:14 in libc.string.h hidden strndup 'char *(const char *, size_t) throw()' extern
  | |-ParmVarDecl 0xb5f69b8 <col:23, col:35> col:35 in libc.string.h hidden __string 'const char *'
  | |-ParmVarDecl 0xb5f6a38 <col:45, col:52> col:52 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-RestrictAttr 0xb5f6c70 </usr/include/sys/cdefs.h:220:47> malloc
  | `-NonNullAttr 0xb5f6cc8 <line:285:44, /usr/include/string.h:181:48> 1
  |-FunctionDecl 0xb5f6f60 <line:232:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:232:14 in libc.string.h hidden strchr 'char *(const char *, int) throw()' extern
  | |-ParmVarDecl 0xb5f6d48 <col:22, col:34> col:34 in libc.string.h hidden __s 'const char *'
  | |-ParmVarDecl 0xb5f6dd0 <col:39, col:43> col:43 in libc.string.h hidden __c 'int'
  | |-PureAttr 0xb5f7010 </usr/include/sys/cdefs.h:229:45>
  | |-NonNullAttr 0xb5f7068 <line:285:44, /usr/include/string.h:233:46> 1
  | `-BuiltinAttr 0xb5f7108 <<invalid sloc>> Implicit 790
  |-FunctionDecl 0xb5f72e8 <line:259:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:259:14 in libc.string.h hidden strrchr 'char *(const char *, int) throw()' extern
  | |-ParmVarDecl 0xb5f7150 <col:23, col:35> col:35 in libc.string.h hidden __s 'const char *'
  | |-ParmVarDecl 0xb5f71d8 <col:40, col:44> col:44 in libc.string.h hidden __c 'int'
  | |-PureAttr 0xb5f7398 </usr/include/sys/cdefs.h:229:45>
  | |-NonNullAttr 0xb5f73f0 <line:285:44, /usr/include/string.h:260:46> 1
  | `-BuiltinAttr 0xb5f7448 <<invalid sloc>> Implicit 793
  |-FunctionDecl 0xb5f7628 <line:273:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:273:14 in libc.string.h hidden strchrnul 'char *(const char *, int) throw()' extern
  | |-ParmVarDecl 0xb5f7490 <col:25, col:37> col:37 in libc.string.h hidden __s 'const char *'
  | |-ParmVarDecl 0xb5f7518 <col:42, col:46> col:46 in libc.string.h hidden __c 'int'
  | |-PureAttr 0xb5f7db0 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb5f7e08 <line:285:44, /usr/include/string.h:274:46> 1
  |-FunctionDecl 0xb5f80b8 <line:281:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:281:15 in libc.string.h hidden strcspn 'size_t (const char *, const char *) throw()' extern
  | |-ParmVarDecl 0xb5f7e88 <col:24, col:36> col:36 in libc.string.h hidden __s 'const char *'
  | |-ParmVarDecl 0xb5f7f10 <col:41, col:53> col:53 in libc.string.h hidden __reject 'const char *'
  | |-PureAttr 0xb5f8168 </usr/include/sys/cdefs.h:229:45>
  | |-NonNullAttr 0xb5f81c0 <line:285:44, /usr/include/string.h:282:49> 1 2
  | `-BuiltinAttr 0xb5f8298 <<invalid sloc>> Implicit 791
  |-FunctionDecl 0xb5f8490 <line:285:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:285:15 in libc.string.h hidden strspn 'size_t (const char *, const char *) throw()' extern
  | |-ParmVarDecl 0xb5f82e0 <col:23, col:35> col:35 in libc.string.h hidden __s 'const char *'
  | |-ParmVarDecl 0xb5f8368 <col:40, col:52> col:52 in libc.string.h hidden __accept 'const char *'
  | |-PureAttr 0xb5f8540 </usr/include/sys/cdefs.h:229:45>
  | |-NonNullAttr 0xb5f8598 <line:285:44, /usr/include/string.h:286:49> 1 2
  | `-BuiltinAttr 0xb5f85f8 <<invalid sloc>> Implicit 794
  |-FunctionDecl 0xb5f8870 <line:311:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:311:14 in libc.string.h hidden strpbrk 'char *(const char *, const char *) throw()' extern
  | |-ParmVarDecl 0xb5f8640 <col:23, col:35> col:35 in libc.string.h hidden __s 'const char *'
  | |-ParmVarDecl 0xb5f86c8 <col:40, col:52> col:52 in libc.string.h hidden __accept 'const char *'
  | |-PureAttr 0xb5f8920 </usr/include/sys/cdefs.h:229:45>
  | |-NonNullAttr 0xb5f8978 <line:285:44, /usr/include/string.h:312:49> 1 2
  | `-BuiltinAttr 0xb5f8a18 <<invalid sloc>> Implicit 792
  |-FunctionDecl 0xb5f8c18 <line:338:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:338:14 in libc.string.h hidden strstr 'char *(const char *, const char *) throw()' extern
  | |-ParmVarDecl 0xb5f8a60 <col:22, col:34> col:34 in libc.string.h hidden __haystack 'const char *'
  | |-ParmVarDecl 0xb5f8ae8 <col:46, col:58> col:58 in libc.string.h hidden __needle 'const char *'
  | |-PureAttr 0xb5f8cc8 </usr/include/sys/cdefs.h:229:45>
  | |-NonNullAttr 0xb5f8d20 <line:285:44, /usr/include/string.h:339:49> 1 2
  | `-BuiltinAttr 0xb5f8d78 <<invalid sloc>> Implicit 795
  |-FunctionDecl 0xb5f9388 <line:344:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:344:14 in libc.string.h hidden strtok 'char *(char *__restrict, const char *__restrict) throw()' extern
  | |-ParmVarDecl 0xb5f91f0 <col:22, col:39> col:39 in libc.string.h hidden __s 'char *__restrict'
  | |-ParmVarDecl 0xb5f9278 <col:44, col:67> col:67 in libc.string.h hidden __delim 'const char *__restrict'
  | |-NonNullAttr 0xb5f9438 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:345:27> 2
  | `-BuiltinAttr 0xb5f94c8 <<invalid sloc>> Implicit 796
  |-FunctionDecl 0xb5f9820 <line:350:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:350:14 in libc.string.h hidden __strtok_r 'char *(char *__restrict, const char *__restrict, char **__restrict) throw()' extern
  | |-ParmVarDecl 0xb5f9510 <col:26, col:43> col:43 in libc.string.h hidden __s 'char *__restrict'
  | |-ParmVarDecl 0xb5f9598 <line:351:5, col:28> col:28 in libc.string.h hidden __delim 'const char *__restrict'
  | |-ParmVarDecl 0xb5f9648 <line:352:5, col:23> col:23 in libc.string.h hidden __save_ptr 'char **__restrict'
  | `-NonNullAttr 0xb5f98d8 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:353:30> 2 3
  |-FunctionDecl 0xb5f9bf0 <line:355:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:355:14 in libc.string.h hidden strtok_r 'char *(char *__restrict, const char *__restrict, char **__restrict) throw()' extern
  | |-ParmVarDecl 0xb5f9988 <col:24, col:41> col:41 in libc.string.h hidden __s 'char *__restrict'
  | |-ParmVarDecl 0xb5f9a10 <col:46, col:69> col:69 in libc.string.h hidden __delim 'const char *__restrict'
  | |-ParmVarDecl 0xb5f9a98 <line:356:10, col:28> col:28 in libc.string.h hidden __save_ptr 'char **__restrict'
  | `-NonNullAttr 0xb5f9ca8 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:357:30> 2 3
  |-FunctionDecl 0xb5f9f10 <line:369:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:369:14 in libc.string.h hidden strcasestr 'char *(const char *, const char *) throw()' extern
  | |-ParmVarDecl 0xb5f9d58 <col:26, col:38> col:38 in libc.string.h hidden __haystack 'const char *'
  | |-ParmVarDecl 0xb5f9de0 <col:50, col:62> col:62 in libc.string.h hidden __needle 'const char *'
  | |-PureAttr 0xb5f9fc0 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb5fa018 <line:285:44, /usr/include/string.h:370:49> 1 2
  |-FunctionDecl 0xb5dc230 <line:378:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:378:14 in libc.string.h hidden memmem 'void *(const void *, size_t, const void *, size_t) throw()' extern
  | |-ParmVarDecl 0xb5fa098 <col:22, col:34> col:34 in libc.string.h hidden __haystack 'const void *'
  | |-ParmVarDecl 0xb5fa118 <col:46, col:53> col:53 in libc.string.h hidden __haystacklen 'size_t':'unsigned long'
  | |-ParmVarDecl 0xb5dbf98 <line:379:8, col:20> col:20 in libc.string.h hidden __needle 'const void *'
  | |-ParmVarDecl 0xb5dc018 <col:30, col:37> col:37 in libc.string.h hidden __needlelen 'size_t':'unsigned long'
  | |-PureAttr 0xb5dc2f0 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb5dc348 <line:285:44, /usr/include/string.h:380:49> 1 3
  |-FunctionDecl 0xb5dc628 <line:384:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:384:14 in libc.string.h hidden __mempcpy 'void *(void *__restrict, const void *__restrict, size_t) throw()' extern
  | |-ParmVarDecl 0xb5dc3c8 <col:25, col:42> col:42 in libc.string.h hidden __dest 'void *__restrict'
  | |-ParmVarDecl 0xb5dc450 <line:385:4, col:27> col:27 in libc.string.h hidden __src 'const void *__restrict'
  | |-ParmVarDecl 0xb5dc4d0 <col:34, col:41> col:41 in libc.string.h hidden __n 'size_t':'unsigned long'
  | `-NonNullAttr 0xb5dc6e0 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:386:30> 1 2
  |-FunctionDecl 0xb5dc9e8 <line:387:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:387:14 in libc.string.h hidden mempcpy 'void *(void *__restrict, const void *__restrict, size_t) throw()' extern
  | |-ParmVarDecl 0xb5dc788 <col:23, col:40> col:40 in libc.string.h hidden __dest 'void *__restrict'
  | |-ParmVarDecl 0xb5dc810 <line:388:9, col:32> col:32 in libc.string.h hidden __src 'const void *__restrict'
  | |-ParmVarDecl 0xb5dc890 <col:39, col:46> col:46 in libc.string.h hidden __n 'size_t':'unsigned long'
  | `-NonNullAttr 0xb5dcaa0 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:389:30> 1 2
  |-FunctionDecl 0xb5dcc88 <line:395:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:395:15 in libc.string.h hidden strlen 'size_t (const char *) throw()' extern
  | |-ParmVarDecl 0xb5dcb48 <col:23, col:35> col:35 in libc.string.h hidden __s 'const char *'
  | |-PureAttr 0xb5dcd30 </usr/include/sys/cdefs.h:229:45>
  | |-NonNullAttr 0xb5dcd88 <line:285:44, /usr/include/string.h:396:46> 1
  | `-BuiltinAttr 0xb5dce50 <<invalid sloc>> Implicit 799
  |-FunctionDecl 0xb5fe2f8 <line:402:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:402:15 in libc.string.h hidden strnlen 'size_t (const char *, size_t) throw()' extern
  | |-ParmVarDecl 0xb5dce98 <col:24, col:36> col:36 in libc.string.h hidden __string 'const char *'
  | |-ParmVarDecl 0xb5dcf18 <col:46, col:53> col:53 in libc.string.h hidden __maxlen 'size_t':'unsigned long'
  | |-PureAttr 0xb5fe3a8 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb5fe400 <line:285:44, /usr/include/string.h:403:46> 1
  |-FunctionDecl 0xb5fe5a0 <line:409:1, /usr/include/sys/cdefs.h:61:27> /usr/include/string.h:409:14 in libc.string.h hidden strerror 'char *(int) throw()' extern
  | |-ParmVarDecl 0xb5fe478 <col:24, col:28> col:28 in libc.string.h hidden __errnum 'int'
  | `-BuiltinAttr 0xb5fe6a0 <<invalid sloc>> Implicit 798
  |-FunctionDecl 0xb5fe9e0 <line:434:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:434:14 in libc.string.h hidden strerror_r 'char *(int, char *, size_t) throw()' extern
  | |-ParmVarDecl 0xb5fe718 <col:26, col:30> col:30 in libc.string.h hidden __errnum 'int'
  | |-ParmVarDecl 0xb5fe7a0 <col:40, col:46> col:46 in libc.string.h hidden __buf 'char *'
  | |-ParmVarDecl 0xb5fe820 <col:53, col:60> col:60 in libc.string.h hidden __buflen 'size_t':'unsigned long'
  | `-NonNullAttr 0xb5fea98 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:435:27> 2
  |-FunctionDecl 0xb5fed30 <line:441:1, /usr/include/sys/cdefs.h:61:27> /usr/include/string.h:441:14 in libc.string.h hidden strerror_l 'char *(int, __locale_t) throw()' extern
  | |-ParmVarDecl 0xb5feb48 <col:26, col:30> col:30 in libc.string.h hidden __errnum 'int'
  | `-ParmVarDecl 0xb5febc8 <col:40, col:51> col:51 in libc.string.h hidden __l '__locale_t':'struct __locale_struct *'
  |-FunctionDecl 0xb5ff028 <line:447:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:447:13 in libc.string.h hidden __bzero 'void (void *, size_t) throw()' extern
  | |-ParmVarDecl 0xb5fee20 <col:22, col:28> col:28 in libc.string.h hidden __s 'void *'
  | |-ParmVarDecl 0xb5feea0 <col:33, col:40> col:40 in libc.string.h hidden __n 'size_t':'unsigned long'
  | `-NonNullAttr 0xb5ff0d8 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:447:66> 1
  |-FunctionDecl 0xb5ffce8 <line:451:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:451:13 in libc.string.h hidden bcopy 'void (const void *, void *, size_t) throw()' extern
  | |-ParmVarDecl 0xb5ffa08 <col:20, col:32> col:32 in libc.string.h hidden __src 'const void *'
  | |-ParmVarDecl 0xb5ffa90 <col:39, col:45> col:45 in libc.string.h hidden __dest 'void *'
  | |-ParmVarDecl 0xb5ffb10 <col:53, col:60> col:60 in libc.string.h hidden __n 'size_t':'unsigned long'
  | `-NonNullAttr 0xb5ffda0 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:452:30> 1 2
  |-FunctionDecl 0xb5fffd0 <line:455:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:455:13 in libc.string.h hidden bzero 'void (void *, size_t) throw()' extern
  | |-ParmVarDecl 0xb5ffe48 <col:20, col:26> col:26 in libc.string.h hidden __s 'void *'
  | |-ParmVarDecl 0xb5ffec8 <col:31, col:38> col:38 in libc.string.h hidden __n 'size_t':'unsigned long'
  | `-NonNullAttr 0xb600080 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:455:64> 1
  |-FunctionDecl 0xb600388 <line:458:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:458:12 in libc.string.h hidden bcmp 'int (const void *, const void *, size_t) throw()' extern
  | |-ParmVarDecl 0xb600128 <col:18, col:30> col:30 in libc.string.h hidden __s1 'const void *'
  | |-ParmVarDecl 0xb6001b0 <col:36, col:48> col:48 in libc.string.h hidden __s2 'const void *'
  | |-ParmVarDecl 0xb600230 <col:54, col:61> col:61 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-PureAttr 0xb600440 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb600498 <line:285:44, /usr/include/string.h:459:49> 1 2
  |-FunctionDecl 0xb6006b0 <line:485:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:485:14 in libc.string.h hidden index 'char *(const char *, int) throw()' extern
  | |-ParmVarDecl 0xb600518 <col:21, col:33> col:33 in libc.string.h hidden __s 'const char *'
  | |-ParmVarDecl 0xb6005a0 <col:38, col:42> col:42 in libc.string.h hidden __c 'int'
  | |-PureAttr 0xb600760 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb6007b8 <line:285:44, /usr/include/string.h:486:46> 1
  |-FunctionDecl 0xb600a18 <line:513:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:513:14 in libc.string.h hidden rindex 'char *(const char *, int) throw()' extern
  | |-ParmVarDecl 0xb600838 <col:22, col:34> col:34 in libc.string.h hidden __s 'const char *'
  | |-ParmVarDecl 0xb6008c0 <col:39, col:43> col:43 in libc.string.h hidden __c 'int'
  | |-PureAttr 0xb600ac8 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb600b20 <line:285:44, /usr/include/string.h:514:46> 1
  |-FunctionDecl 0xb600cc0 <line:519:1, col:60> col:12 in libc.string.h hidden ffs 'int (int) throw()' extern
  | |-ParmVarDecl 0xb600b98 <col:17, col:21> col:21 in libc.string.h hidden __i 'int'
  | `-ConstAttr 0xb600d68 <col:50>
  |-FunctionDecl 0xb600f30 <line:524:1, col:66> col:12 in libc.string.h hidden ffsl 'int (long) throw()' extern
  | |-ParmVarDecl 0xb600e00 <col:18, col:27> col:27 in libc.string.h hidden __l 'long'
  | `-ConstAttr 0xb600fd8 <col:56>
  |-FunctionDecl 0xb6011a0 <line:526:15, line:527:40> line:526:26 in libc.string.h hidden ffsll 'int (long long) throw()' extern
  | |-ParmVarDecl 0xb601070 <col:33, col:47> col:47 in libc.string.h hidden __ll 'long long'
  | `-ConstAttr 0xb601248 <line:527:30>
  |-FunctionDecl 0xb601498 <line:532:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:532:12 in libc.string.h hidden strcasecmp 'int (const char *, const char *) throw()' extern
  | |-ParmVarDecl 0xb6012e0 <col:24, col:36> col:36 in libc.string.h hidden __s1 'const char *'
  | |-ParmVarDecl 0xb601368 <col:42, col:54> col:54 in libc.string.h hidden __s2 'const char *'
  | |-PureAttr 0xb601548 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb6015a0 <line:285:44, /usr/include/string.h:533:49> 1 2
  |-FunctionDecl 0xb601878 <line:536:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:536:12 in libc.string.h hidden strncasecmp 'int (const char *, const char *, size_t) throw()' extern
  | |-ParmVarDecl 0xb601618 <col:25, col:37> col:37 in libc.string.h hidden __s1 'const char *'
  | |-ParmVarDecl 0xb6016a0 <col:43, col:55> col:55 in libc.string.h hidden __s2 'const char *'
  | |-ParmVarDecl 0xb601720 <col:61, col:68> col:68 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-PureAttr 0xb601930 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb601988 <line:285:44, /usr/include/string.h:537:49> 1 2
  |-FunctionDecl 0xb603cd8 <line:543:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:543:12 in libc.string.h hidden strcasecmp_l 'int (const char *, const char *, __locale_t) throw()' extern
  | |-ParmVarDecl 0xb603a58 <col:26, col:38> col:38 in libc.string.h hidden __s1 'const char *'
  | |-ParmVarDecl 0xb603ae0 <col:44, col:56> col:56 in libc.string.h hidden __s2 'const char *'
  | |-ParmVarDecl 0xb603b60 <line:544:5, col:16> col:16 in libc.string.h hidden __loc '__locale_t':'struct __locale_struct *'
  | |-PureAttr 0xb603d90 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb603de8 <line:285:44, /usr/include/string.h:545:52> 1 2 3
  |-FunctionDecl 0xb604230 <line:547:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:547:12 in libc.string.h hidden strncasecmp_l 'int (const char *, const char *, size_t, __locale_t) throw()' extern
  | |-ParmVarDecl 0xb603e70 <col:27, col:39> col:39 in libc.string.h hidden __s1 'const char *'
  | |-ParmVarDecl 0xb603ef8 <col:45, col:57> col:57 in libc.string.h hidden __s2 'const char *'
  | |-ParmVarDecl 0xb603f78 <line:548:6, col:13> col:13 in libc.string.h hidden __n 'size_t':'unsigned long'
  | |-ParmVarDecl 0xb603ff8 <col:18, col:29> col:29 in libc.string.h hidden __loc '__locale_t':'struct __locale_struct *'
  | |-PureAttr 0xb6042f0 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb604348 <line:285:44, /usr/include/string.h:549:52> 1 2 4
  |-FunctionDecl 0xb604600 <line:555:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:555:14 in libc.string.h hidden strsep 'char *(char **__restrict, const char *__restrict) throw()' extern
  | |-ParmVarDecl 0xb6043d0 <col:22, col:40> col:40 in libc.string.h hidden __stringp 'char **__restrict'
  | |-ParmVarDecl 0xb604458 <line:556:8, col:31> col:31 in libc.string.h hidden __delim 'const char *__restrict'
  | `-NonNullAttr 0xb6046b0 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:557:30> 1 2
  |-FunctionDecl 0xb604820 <line:562:1, /usr/include/sys/cdefs.h:61:27> /usr/include/string.h:562:14 in libc.string.h hidden strsignal 'char *(int) throw()' extern
  | `-ParmVarDecl 0xb604758 <col:25, col:29> col:29 in libc.string.h hidden __sig 'int'
  |-FunctionDecl 0xb604ae8 <line:565:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:565:14 in libc.string.h hidden __stpcpy 'char *(char *__restrict, const char *__restrict) throw()' extern
  | |-ParmVarDecl 0xb604908 <col:24, col:41> col:41 in libc.string.h hidden __dest 'char *__restrict'
  | |-ParmVarDecl 0xb604990 <col:49, col:72> col:72 in libc.string.h hidden __src 'const char *__restrict'
  | `-NonNullAttr 0xb604b98 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:566:30> 1 2
  |-FunctionDecl 0xb604e00 <line:567:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:567:14 in libc.string.h hidden stpcpy 'char *(char *__restrict, const char *__restrict) throw()' extern
  | |-ParmVarDecl 0xb604c48 <col:22, col:39> col:39 in libc.string.h hidden __dest 'char *__restrict'
  | |-ParmVarDecl 0xb604cd0 <col:47, col:70> col:70 in libc.string.h hidden __src 'const char *__restrict'
  | `-NonNullAttr 0xb604eb0 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:568:30> 1 2
  |-FunctionDecl 0xb6051b8 <line:572:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:572:14 in libc.string.h hidden __stpncpy 'char *(char *__restrict, const char *__restrict, size_t) throw()' extern
  | |-ParmVarDecl 0xb604f58 <col:25, col:42> col:42 in libc.string.h hidden __dest 'char *__restrict'
  | |-ParmVarDecl 0xb604fe0 <line:573:4, col:27> col:27 in libc.string.h hidden __src 'const char *__restrict'
  | |-ParmVarDecl 0xb605060 <col:34, col:41> col:41 in libc.string.h hidden __n 'size_t':'unsigned long'
  | `-NonNullAttr 0xb605270 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:574:30> 1 2
  |-FunctionDecl 0xb605578 <line:575:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:575:14 in libc.string.h hidden stpncpy 'char *(char *__restrict, const char *__restrict, size_t) throw()' extern
  | |-ParmVarDecl 0xb605318 <col:23, col:40> col:40 in libc.string.h hidden __dest 'char *__restrict'
  | |-ParmVarDecl 0xb6053a0 <line:576:9, col:32> col:32 in libc.string.h hidden __src 'const char *__restrict'
  | |-ParmVarDecl 0xb605420 <col:39, col:46> col:46 in libc.string.h hidden __n 'size_t':'unsigned long'
  | `-NonNullAttr 0xb605630 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:577:30> 1 2
  |-FunctionDecl 0xb605890 <line:582:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:582:12 in libc.string.h hidden strverscmp 'int (const char *, const char *) throw()' extern
  | |-ParmVarDecl 0xb6056d8 <col:24, col:36> col:36 in libc.string.h hidden __s1 'const char *'
  | |-ParmVarDecl 0xb605760 <col:42, col:54> col:54 in libc.string.h hidden __s2 'const char *'
  | |-PureAttr 0xb605940 </usr/include/sys/cdefs.h:229:45>
  | `-NonNullAttr 0xb605998 <line:285:44, /usr/include/string.h:583:49> 1 2
  |-FunctionDecl 0xb605bc0 <line:586:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:586:14 in libc.string.h hidden strfry 'char *(char *) throw()' extern
  | |-ParmVarDecl 0xb605a78 <col:22, col:28> col:28 in libc.string.h hidden __string 'char *'
  | `-NonNullAttr 0xb605c68 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:586:59> 1
  |-FunctionDecl 0xb605f20 <line:589:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:589:14 in libc.string.h hidden memfrob 'void *(void *, size_t) throw()' extern
  | |-ParmVarDecl 0xb605d18 <col:23, col:29> col:29 in libc.string.h hidden __s 'void *'
  | |-ParmVarDecl 0xb605d98 <col:34, col:41> col:41 in libc.string.h hidden __n 'size_t':'unsigned long'
  | `-NonNullAttr 0xb605fd0 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:589:67> 1
  `-FunctionDecl 0xb606160 <line:602:1, /usr/include/sys/cdefs.h:285:63> /usr/include/string.h:602:14 in libc.string.h hidden basename 'char *(const char *) throw()' extern
    |-ParmVarDecl 0xb606078 <col:24, col:36> col:36 in libc.string.h hidden __filename 'const char *'
    `-NonNullAttr 0xb606208 </usr/include/sys/cdefs.h:285:44, /usr/include/string.h:602:69> 1
Mod=0
 #0 0x0000000009107dda llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Unix/Signals.inc:565:11
 #1 0x0000000009107f8b PrintStackTraceSignalHandler(void*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Unix/Signals.inc:632:1
 #2 0x00000000091067b3 llvm::sys::RunSignalHandlers() /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Signals.cpp:97:5
 #3 0x00000000091085a5 SignalHandler(int) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/lib/Support/Unix/Signals.inc:407:1
 #4 0x00007f3887b30630 __restore_rt sigaction.c:0:0
 #5 0x00000000031e3995 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::__is_long() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1445:22
 #6 0x00000000031e3949 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::__get_pointer() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1544:17
 #7 0x00000000031e3905 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::data() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1251:75
 #8 0x00000000031e1945 std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >::c_str() const /scratch/greenc/test-products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/string:1249:48
 #9 0x0000000004639003 clang::ASTWriter::getSubmoduleID(clang::Module*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriter.cpp:2503:47
#10 0x00000000046bdcea clang::ASTDeclWriter::VisitDecl(clang::Decl*) /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99/interpreter/llvm/src/tools/clang/lib/Serialization/ASTWriterDecl.cpp:368:27
<snip/>

vgvassilev commented 1 year ago

I think at that point I will need to reproduce that failure locally to be able to understand it. Does that happen with just 'bare root' if so how should I configure it?

greenc-FNAL commented 1 year ago

This is the command line I used:

+ cd /scratch/greenc/test-products/root/v6_27_99f1/build/Linux64bit+3.10-2.17-c14-p3913-debug
+ cmake -GNinja -DCMAKE_BUILD_TYPE=Debug '-DCMAKE_IGNORE_PATH=/usr/local/lib;/usr/local/include;/usr/local/bin' -DCMAKE_INSTALL_PREFIX=/scratch/greenc/test-products/root/v6_27_99f1/Linux64bit+3.10-2.17-c14-p3913-debug -Dall:BOOL=OFF -Dasan:BOOL=OFF -Dasserts:BOOL=ON -Dccache:BOOL=OFF -Dclingtest:BOOL=OFF -Dcoverage:BOOL=OFF -Dcxxmodules:BOOL=OFF -Ddev:BOOL=OFF -Ddistcc:BOOL=OFF -Dexceptions:BOOL=ON -Dfail-on-missing:BOOL=ON -Dfortran:BOOL=OFF -Dgminimal:BOOL=ON -Dgnuinstall:BOOL=OFF -Dmemory_termination:BOOL=OFF -Dmemstat:BOOL=OFF -Dminimal:BOOL=OFF -Drootbench:BOOL=OFF -Droottest:BOOL=OFF -Drpath:BOOL=OFF -Druntime_cxxmodules:BOOL=ON -Dshared:BOOL=ON -Dsoversion:BOOL=ON -Dtest_distrd_pyspark:BOOL=OFF -Dtest_distrd_dask:BOOL=OFF -Dtesting:BOOL=OFF -DLLVM_BUILD_TYPE=Debug -DLLVM_ENABLE_ASSERTIONS:BOOL=ON -Dbuiltin_afterimage:BOOL=OFF -Dbuiltin_cfitsio:BOOL=OFF -Dbuiltin_clang:BOOL=ON -Dbuiltin_cling:BOOL=ON -Dbuiltin_cppzmq:BOOL=OFF -Dbuiltin_davix:BOOL=OFF -Dbuiltin_fftw3:BOOL=OFF -Dbuiltin_freetype:BOOL=OFF -Dbuiltin_ftgl:BOOL=OFF -Dbuiltin_gl2ps:BOOL=OFF -Dbuiltin_glew:BOOL=OFF -Dbuiltin_gsl:BOOL=OFF -Dbuiltin_llvm:BOOL=ON -Dbuiltin_lz4:BOOL=OFF -Dbuiltin_lzma:BOOL=OFF -Dbuiltin_nlohmannjson:BOOL=ON -Dbuiltin_openssl:BOOL=OFF -Dbuiltin_openui5:BOOL=ON -Dbuiltin_pcre:BOOL=OFF -Dbuiltin_tbb:BOOL=OFF -Dbuiltin_unuran:BOOL=OFF -Dbuiltin_vc:BOOL=OFF -Dbuiltin_vdt:BOOL=OFF -Dbuiltin_veccore:BOOL=OFF -Dbuiltin_xrootd:BOOL=OFF -Dbuiltin_xxhash:BOOL=OFF -Dbuiltin_zeromq:BOOL=OFF -Dbuiltin_zlib:BOOL=OFF -Dbuiltin_zstd:BOOL=OFF -DXROOTD_ROOT_DIR=/scratch/greenc/test-products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-c14-p3913-debug -Dalien:BOOL=OFF -Darrow:BOOL=OFF -Dasimage:BOOL=ON -Dcefweb:BOOL=OFF -Dclad:BOOL=ON -Dcuda:BOOL=OFF -Dcudnn:BOOL=OFF -Ddaos:BOOL=OFF -Ddataframe:BOOL=ON -Ddavix:BOOL=OFF -Ddcache:BOOL=OFF -Dfcgi:BOOL=OFF -Dfftw3:BOOL=ON -Dfitsio:BOOL=OFF -Dgdml:BOOL=ON -Dgfal:BOOL=OFF -Dgsl_shared:BOOL=ON -Dgviz:BOOL=OFF -Dhttp:BOOL=ON -Dimt:BOOL=ON -Djemalloc:BOOL=OFF -Dmathmore:BOOL=ON -Dminuit2:BOOL=ON -Dmlp:BOOL=ON -Dmonalisa:BOOL=OFF -Dmpi:BOOL=OFF -Dmysql:BOOL=OFF -Dodbc:BOOL=OFF -Dopengl:BOOL=ON -Doracle:BOOL=OFF -Dpgsql:BOOL=ON -Dpyroot:BOOL=ON -Dpyroot_legacy:BOOL=OFF -Dpythia6:BOOL=ON -Dpythia6_nolink:BOOL=OFF -Dpythia8:BOOL=OFF -Dqt5web:BOOL=OFF -Dqt6web:BOOL=OFF -Dr:BOOL=OFF -Droofit:BOOL=ON -Droot7:BOOL=ON -Dshadowpw:BOOL=OFF -Dspectrum:BOOL=ON -Dsqlite:BOOL=OFF -Dssl:BOOL=ON -Dtcmalloc:BOOL=OFF -Dtmva-cpu:BOOL=ON -Dtmva-gpu:BOOL=OFF -Dtmva-pymva:BOOL=ON -Dtmva-rmva:BOOL=OFF -Dtmva:BOOL=ON -During:BOOL=OFF -Dunuran:BOOL=OFF -Dvc:BOOL=OFF -Dvdt:BOOL=OFF -Dveccore:BOOL=OFF -Dvecgeom:BOOL=OFF -Dvmc:BOOL=OFF -Dwebgui:BOOL=ON -Dx11:BOOL=ON -Dxml:BOOL=ON -Dxproofd:BOOL=OFF -Dxrootd:BOOL=ON -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_CXX_STANDARD=17 -DCMAKE_C_COMPILER=clang -DCMAKE_C_STANDARD=11 -DCMAKE_Fortran_COMPILER=gfortran -DCMAKE_Fortran_FLAGS=-std=legacy -DCLANG_DEFAULT_CXX_STDLIB=libc++ -DLLVM_ENABLE_LIBCXX:BOOL=ON -Dlibcxx:BOOL=ON /scratch/greenc/test-products/root/v6_27_99f1/source/root-6.27.99

Notes:

Source built is 66df8b8ee6b336197678950c4f9c83f2c479662a, attached patches recommended:
Platform is SLF7/x64_64 with externally-built Clang14.
ROOT is built (see CMake command line above) with libc++ and C++17.
You'll need to tweak paths, boolean options, and buildtin_X for your environment as necessary.
Clang 14 was bootstrapped: initially built with GCC 12.2 and then itself, with lib and runtime settings (libc++/compiler-rt/cxxabi) set so that the resulting compiler and libraries have no dependence on the original compiler/library/runtime used to build it. Also, the resulting compiler uses libc++/compiler-rt/cxxabi by default. This shouldn't matter, and I wouldn't try to duplicate this without first trying a vanilla Clang14 build, but I include this information JiC.
My first attempt to reproduce would be by stripping down all not-obviously-required externals and trying with a vanilla Clang14.

vgvassilev commented 1 year ago

Any chance of some sort of a docker image?

greenc-FNAL commented 1 year ago

I could try to put something together.

greenc-FNAL commented 1 year ago

@vgvassilev Please see https://hub.docker.com/repository/docker/greenc/root_10861/general

Let me know if you have any problems working in this environment.

vgvassilev commented 1 year ago

Turns out this is a bug in libcxx. I have commented here: https://reviews.llvm.org/D119036#4067105

If you could patch libcxx, we should either apply https://reviews.llvm.org/D119468 which is wrong or drop the line export std_config from /products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/c++/v1/module.modulemap as I suggested upstream.

greenc-FNAL commented 1 year ago

I'll try this out, thank you! I see a restricted project has been added to the review: if you can access it, does it say anything relevant?

vgvassilev commented 1 year ago

I have no clue what is that and it seems a ton of build failure information, maybe we can just ignore it.

greenc-FNAL commented 1 year ago

Unfortunately simply removing export std_config as you suggest appears not to be sufficient with a build as described in https://github.com/root-project/root/issues/10861#issuecomment-1378895796. Trying to see if I can get more information...

vgvassilev commented 1 year ago

I tested in in the docker and worked for me :(

greenc-FNAL commented 1 year ago

Should I be using https://github.com/root-project/root/issues/10861#issuecomment-1377762782 or not?

vgvassilev commented 1 year ago

Should I be using #10861 (comment) or not?

You should not.

greenc-FNAL commented 1 year ago

OK, will remove and retry.

vgvassilev commented 1 year ago

I believe you prepared a clean docker environment for me. Does the docker environment contain this patch?

greenc-FNAL commented 1 year ago

No, the Docker environment leaves you to import and build your own ROOT source.

greenc-FNAL commented 1 year ago

Patch removed; failure remains. Could you provide details of the ROOT source and build configuration you used in the Docker environment to test the export std_config removal?

vgvassilev commented 1 year ago

I took the cmake command from your log file and built. It failed and edited the modulemap file, re-ran the rootcling invocation by hand and that was it - worked. Btw, does the other (wrong) patch work for you?

greenc-FNAL commented 1 year ago

Unfortunately I cannot get a successful build with https://github.com/root-project/root/commit/66df8b8ee6b336197678950c4f9c83f2c479662a with either your or the "wrong" in-place edits to edits to include/c++/v1/modules.modulemap.

I'd be grateful if you could provide details of exactly what source you used and the relevant CMake line to try to reproduce your success.

vgvassilev commented 1 year ago

I did another run of docker and started from scratch (as I did not save the state) and worked again for me. Here is my history.

[root@b561d88b0a11 obj]# history
    1  ls
    2  mkdir root obj
    3  git clone https://github.com/root-project/root.git root
    4  ls
    5  cd root/
    6  git checkout v6-26-04 
    7  cd ../obj/
    8  mkdir ../inst
    9  cmake -GNinja -DCMAKE_BUILD_TYPE=Debug '-DCMAKE_IGNORE_PATH=/usr/local/lib;/usr/local/include;/usr/local/bin' -DCMAKE_INSTALL_PREFIX=/scratch/greenc/test-products/root/v6_26_04b/Linux64bit+3.10-2.17-c14-p3912-debug -Dall:BOOL=OFF -Dasan:BOOL=OFF -Dasserts:BOOL=ON -Dccache:BOOL=OFF -Dclingtest:BOOL=OFF -Dcoverage:BOOL=OFF -Dcxxmodules:BOOL=OFF -Ddev:BOOL=OFF -Ddistcc:BOOL=OFF -Dexceptions:BOOL=ON -Dfail-on-missing:BOOL=ON -Dfortran:BOOL=OFF -Dgminimal:BOOL=ON -Dgnuinstall:BOOL=OFF -Dmemory_termination:BOOL=OFF -Dmemstat:BOOL=OFF -Dminimal:BOOL=OFF -Drootbench:BOOL=OFF -Droottest:BOOL=OFF -Drpath:BOOL=OFF -Druntime_cxxmodules:BOOL=ON -Dshared:BOOL=ON -Dsoversion:BOOL=ON -Dtest_distrd_pyspark:BOOL=OFF -Dtest_distrd_dask:BOOL=OFF -Dtesting:BOOL=OFF -DLLVM_BUILD_TYPE=Debug -DLLVM_ENABLE_ASSERTIONS:BOOL=ON -Dbuiltin_afterimage:BOOL=OFF -Dbuiltin_cfitsio:BOOL=OFF -Dbuiltin_clang:BOOL=ON -Dbuiltin_cling:BOOL=ON -Dbuiltin_cppzmq:BOOL=OFF -Dbuiltin_davix:BOOL=OFF -Dbuiltin_fftw3:BOOL=OFF -Dbuiltin_freetype:BOOL=OFF -Dbuiltin_ftgl:BOOL=OFF -Dbuiltin_gl2ps:BOOL=OFF -Dbuiltin_glew:BOOL=OFF -Dbuiltin_gsl:BOOL=OFF -Dbuiltin_llvm:BOOL=ON -Dbuiltin_lz4:BOOL=OFF -Dbuiltin_lzma:BOOL=OFF -Dbuiltin_nlohmannjson:BOOL=ON -Dbuiltin_openssl:BOOL=OFF -Dbuiltin_openui5:BOOL=ON -Dbuiltin_pcre:BOOL=OFF -Dbuiltin_tbb:BOOL=OFF -Dbuiltin_unuran:BOOL=OFF -Dbuiltin_vc:BOOL=OFF -Dbuiltin_vdt:BOOL=OFF -Dbuiltin_veccore:BOOL=OFF -Dbuiltin_xrootd:BOOL=OFF -Dbuiltin_xxhash:BOOL=OFF -Dbuiltin_zeromq:BOOL=OFF -Dbuiltin_zlib:BOOL=OFF -Dbuiltin_zstd:BOOL=OFF -DXROOTD_ROOT_DIR=/products/xrootd/v5_4_3a/Linux64bit+3.10-2.17-c14-p3912-debug -Dalien:BOOL=OFF -Darrow:BOOL=OFF -Dasimage:BOOL=ON -Dcefweb:BOOL=OFF -Dclad:BOOL=ON -Dcuda:BOOL=OFF -Dcudnn:BOOL=OFF -Ddaos:BOOL=OFF -Ddataframe:BOOL=ON -Ddavix:BOOL=OFF -Ddcache:BOOL=OFF -Dfcgi:BOOL=OFF -Dfftw3:BOOL=ON -Dfitsio:BOOL=OFF -Dgdml:BOOL=ON -Dgfal:BOOL=OFF -Dgsl_shared:BOOL=ON -Dgviz:BOOL=OFF -Dhttp:BOOL=ON -Dimt:BOOL=ON -Djemalloc:BOOL=OFF -Dmathmore:BOOL=ON -Dminuit2:BOOL=ON -Dmlp:BOOL=ON -Dmonalisa:BOOL=OFF -Dmpi:BOOL=OFF -Dmysql:BOOL=OFF -Dodbc:BOOL=OFF -Dopengl:BOOL=ON -Doracle:BOOL=OFF -Dpgsql:BOOL=ON -Dpyroot:BOOL=ON -Dpyroot_legacy:BOOL=OFF -Dpythia6:BOOL=ON -Dpythia6_nolink:BOOL=OFF -Dpythia8:BOOL=OFF -Dqt5web:BOOL=OFF -Dqt6web:BOOL=OFF -Dr:BOOL=OFF -Droofit:BOOL=ON -Droot7:BOOL=ON -Dshadowpw:BOOL=OFF -Dspectrum:BOOL=ON -Dsqlite:BOOL=OFF -Dssl:BOOL=ON -Dtcmalloc:BOOL=OFF -Dtmva-cpu:BOOL=ON -Dtmva-gpu:BOOL=OFF -Dtmva-pymva:BOOL=ON -Dtmva-rmva:BOOL=OFF -Dtmva:BOOL=ON -During:BOOL=OFF -Dunuran:BOOL=OFF -Dvc:BOOL=OFF -Dvdt:BOOL=OFF -Dveccore:BOOL=OFF -Dvecgeom:BOOL=OFF -Dvmc:BOOL=OFF -Dwebgui:BOOL=ON -Dx11:BOOL=ON -Dxml:BOOL=ON -Dxproofd:BOOL=OFF -Dxrootd:BOOL=ON -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_CXX_STANDARD=17 -DCMAKE_C_COMPILER=clang -DCMAKE_C_STANDARD=11 -DCMAKE_Fortran_COMPILER=gfortran -DCMAKE_Fortran_FLAGS=-std=legacy -DCLANG_DEFAULT_CXX_STDLIB=libc++ -DLLVM_ENABLE_LIBCXX:BOOL=ON -Dlibcxx:BOOL=ON /scratch/greenc/test-products/root/v6_26_04b/source/root-6.26.04
   10  /products/cmake/v3_22_2/Linux64bit+3.10-2.17/bin/cmake -GNinja -DCMAKE_BUILD_TYPE=Debug '-DCMAKE_IGNORE_PATH=/usr/local/lib;/usr/local/include;/usr/local/bin' -DCMAKE_INSTALL_PREFIX=../inst -Dall:BOOL=OFF -Dasan:BOOL=OFF -Dasserts:BOOL=ON -Dccache:BOOL=OFF -Dclingtest:BOOL=OFF -Dcoverage:BOOL=OFF -Dcxxmodules:BOOL=OFF -Ddev:BOOL=OFF -Ddistcc:BOOL=OFF -Dexceptions:BOOL=ON -Dfail-on-missing:BOOL=ON -Dfortran:BOOL=OFF -Dgminimal:BOOL=ON -Dgnuinstall:BOOL=OFF -Dmemory_termination:BOOL=OFF -Dmemstat:BOOL=OFF -Dminimal:BOOL=OFF -Drootbench:BOOL=OFF -Droottest:BOOL=OFF -Drpath:BOOL=OFF -Druntime_cxxmodules:BOOL=ON -Dshared:BOOL=ON -Dsoversion:BOOL=ON -Dtest_distrd_pyspark:BOOL=OFF -Dtest_distrd_dask:BOOL=OFF -Dtesting:BOOL=OFF -DLLVM_BUILD_TYPE=Debug -DLLVM_ENABLE_ASSERTIONS:BOOL=ON -Dbuiltin_afterimage:BOOL=OFF -Dbuiltin_cfitsio:BOOL=OFF -Dbuiltin_clang:BOOL=ON -Dbuiltin_cling:BOOL=ON -Dbuiltin_cppzmq:BOOL=OFF -Dbuiltin_davix:BOOL=OFF -Dbuiltin_fftw3:BOOL=OFF -Dbuiltin_freetype:BOOL=OFF -Dbuiltin_ftgl:BOOL=OFF -Dbuiltin_gl2ps:BOOL=OFF -Dbuiltin_glew:BOOL=OFF -Dbuiltin_gsl:BOOL=OFF -Dbuiltin_llvm:BOOL=ON -Dbuiltin_lz4:BOOL=OFF -Dbuiltin_lzma:BOOL=OFF -Dbuiltin_nlohmannjson:BOOL=ON -Dbuiltin_openssl:BOOL=OFF -Dbuiltin_openui5:BOOL=ON -Dbuiltin_pcre:BOOL=OFF -Dbuiltin_tbb:BOOL=OFF -Dbuiltin_unuran:BOOL=OFF -Dbuiltin_vc:BOOL=OFF -Dbuiltin_vdt:BOOL=OFF -Dbuiltin_veccore:BOOL=OFF -Dbuiltin_xrootd:BOOL=OFF -Dbuiltin_xxhash:BOOL=OFF -Dbuiltin_zeromq:BOOL=OFF -Dbuiltin_zlib:BOOL=OFF -Dbuiltin_zstd:BOOL=OFF -DXROOTD_ROOT_DIR=/products/xrootd/v5_4_3a/Linux64bit+3.10-2.17-c14-p3912-debug -Dalien:BOOL=OFF -Darrow:BOOL=OFF -Dasimage:BOOL=ON -Dcefweb:BOOL=OFF -Dclad:BOOL=ON -Dcuda:BOOL=OFF -Dcudnn:BOOL=OFF -Ddaos:BOOL=OFF -Ddataframe:BOOL=ON -Ddavix:BOOL=OFF -Ddcache:BOOL=OFF -Dfcgi:BOOL=OFF -Dfftw3:BOOL=ON -Dfitsio:BOOL=OFF -Dgdml:BOOL=ON -Dgfal:BOOL=OFF -Dgsl_shared:BOOL=ON -Dgviz:BOOL=OFF -Dhttp:BOOL=ON -Dimt:BOOL=ON -Djemalloc:BOOL=OFF -Dmathmore:BOOL=ON -Dminuit2:BOOL=ON -Dmlp:BOOL=ON -Dmonalisa:BOOL=OFF -Dmpi:BOOL=OFF -Dmysql:BOOL=OFF -Dodbc:BOOL=OFF -Dopengl:BOOL=ON -Doracle:BOOL=OFF -Dpgsql:BOOL=ON -Dpyroot:BOOL=ON -Dpyroot_legacy:BOOL=OFF -Dpythia6:BOOL=ON -Dpythia6_nolink:BOOL=OFF -Dpythia8:BOOL=OFF -Dqt5web:BOOL=OFF -Dqt6web:BOOL=OFF -Dr:BOOL=OFF -Droofit:BOOL=ON -Droot7:BOOL=ON -Dshadowpw:BOOL=OFF -Dspectrum:BOOL=ON -Dsqlite:BOOL=OFF -Dssl:BOOL=ON -Dtcmalloc:BOOL=OFF -Dtmva-cpu:BOOL=ON -Dtmva-gpu:BOOL=OFF -Dtmva-pymva:BOOL=ON -Dtmva-rmva:BOOL=OFF -Dtmva:BOOL=ON -During:BOOL=OFF -Dunuran:BOOL=OFF -Dvc:BOOL=OFF -Dvdt:BOOL=OFF -Dveccore:BOOL=OFF -Dvecgeom:BOOL=OFF -Dvmc:BOOL=OFF -Dwebgui:BOOL=ON -Dx11:BOOL=ON -Dxml:BOOL=ON -Dxproofd:BOOL=OFF -Dxrootd:BOOL=ON -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_CXX_STANDARD=17 -DCMAKE_C_COMPILER=clang -DCMAKE_C_STANDARD=11 -DCMAKE_Fortran_COMPILER=gfortran -DCMAKE_Fortran_FLAGS=-std=legacy -DCLANG_DEFAULT_CXX_STDLIB=libc++ -DLLVM_ENABLE_LIBCXX:BOOL=ON -Dlibcxx:BOOL=ON ../root/
   11  rm -fr *
   12  /products/cmake/v3_22_2/Linux64bit+3.10-2.17/bin/cmake -GNinja -DCMAKE_BUILD_TYPE=Debug '-DCMAKE_IGNORE_PATH=/usr/local/lib;/usr/local/include;/usr/local/bin' -DCMAKE_INSTALL_PREFIX=../inst -Dall:BOOL=OFF -Dasan:BOOL=OFF -Dasserts:BOOL=ON -Dccache:BOOL=OFF -Dclingtest:BOOL=OFF -Dcoverage:BOOL=OFF -Dcxxmodules:BOOL=OFF -Ddev:BOOL=OFF -Ddistcc:BOOL=OFF -Dexceptions:BOOL=ON -Dfail-on-missing:BOOL=ON -Dfortran:BOOL=OFF -Dgminimal:BOOL=ON -Dgnuinstall:BOOL=OFF -Dmemory_termination:BOOL=OFF -Dmemstat:BOOL=OFF -Dminimal:BOOL=OFF -Drootbench:BOOL=OFF -Droottest:BOOL=OFF -Drpath:BOOL=OFF -Druntime_cxxmodules:BOOL=ON -Dshared:BOOL=ON -Dsoversion:BOOL=ON -Dtest_distrd_pyspark:BOOL=OFF -Dtest_distrd_dask:BOOL=OFF -Dtesting:BOOL=OFF -DLLVM_BUILD_TYPE=Debug -DLLVM_ENABLE_ASSERTIONS:BOOL=ON -Dbuiltin_afterimage:BOOL=OFF -Dbuiltin_cfitsio:BOOL=OFF -Dbuiltin_clang:BOOL=ON -Dbuiltin_cling:BOOL=ON -Dbuiltin_cppzmq:BOOL=OFF -Dbuiltin_davix:BOOL=OFF -Dbuiltin_fftw3:BOOL=OFF -Dbuiltin_freetype:BOOL=OFF -Dbuiltin_ftgl:BOOL=OFF -Dbuiltin_gl2ps:BOOL=OFF -Dbuiltin_glew:BOOL=OFF -Dbuiltin_gsl:BOOL=OFF -Dbuiltin_llvm:BOOL=ON -Dbuiltin_lz4:BOOL=OFF -Dbuiltin_lzma:BOOL=OFF -Dbuiltin_nlohmannjson:BOOL=ON -Dbuiltin_openssl:BOOL=OFF -Dbuiltin_openui5:BOOL=ON -Dbuiltin_pcre:BOOL=OFF -Dbuiltin_tbb:BOOL=OFF -Dbuiltin_unuran:BOOL=OFF -Dbuiltin_vc:BOOL=OFF -Dbuiltin_vdt:BOOL=OFF -Dbuiltin_veccore:BOOL=OFF -Dbuiltin_xrootd:BOOL=OFF -Dbuiltin_xxhash:BOOL=OFF -Dbuiltin_zeromq:BOOL=OFF -Dbuiltin_zlib:BOOL=OFF -Dbuiltin_zstd:BOOL=OFF -DXROOTD_ROOT_DIR=/products/xrootd/v5_4_3a/Linux64bit+3.10-2.17-c14-p3912-debug -Dalien:BOOL=OFF -Darrow:BOOL=OFF -Dasimage:BOOL=ON -Dcefweb:BOOL=OFF -Dclad:BOOL=ON -Dcuda:BOOL=OFF -Dcudnn:BOOL=OFF -Ddaos:BOOL=OFF -Ddataframe:BOOL=ON -Ddavix:BOOL=OFF -Ddcache:BOOL=OFF -Dfcgi:BOOL=OFF -Dfftw3:BOOL=ON -Dfitsio:BOOL=OFF -Dgdml:BOOL=ON -Dgfal:BOOL=OFF -Dgsl_shared:BOOL=ON -Dgviz:BOOL=OFF -Dhttp:BOOL=ON -Dimt:BOOL=ON -Djemalloc:BOOL=OFF -Dmathmore:BOOL=ON -Dminuit2:BOOL=ON -Dmlp:BOOL=ON -Dmonalisa:BOOL=OFF -Dmpi:BOOL=OFF -Dmysql:BOOL=OFF -Dodbc:BOOL=OFF -Dopengl:BOOL=ON -Doracle:BOOL=OFF -Dpgsql:BOOL=ON -Dpyroot:BOOL=ON -Dpyroot_legacy:BOOL=OFF -Dpythia6:BOOL=ON -Dpythia6_nolink:BOOL=OFF -Dpythia8:BOOL=OFF -Dqt5web:BOOL=OFF -Dqt6web:BOOL=OFF -Dr:BOOL=OFF -Droofit:BOOL=ON -Droot7:BOOL=ON -Dshadowpw:BOOL=OFF -Dspectrum:BOOL=ON -Dsqlite:BOOL=OFF -Dssl:BOOL=ON -Dtcmalloc:BOOL=OFF -Dtmva-cpu:BOOL=ON -Dtmva-gpu:BOOL=OFF -Dtmva-pymva:BOOL=ON -Dtmva-rmva:BOOL=OFF -Dtmva:BOOL=ON -During:BOOL=OFF -Dunuran:BOOL=OFF -Dvc:BOOL=OFF -Dvdt:BOOL=OFF -Dveccore:BOOL=OFF -Dvecgeom:BOOL=OFF -Dvmc:BOOL=OFF -Dwebgui:BOOL=ON -Dx11:BOOL=ON -Dxml:BOOL=ON -Dxproofd:BOOL=OFF -Dxrootd:BOOL=ON -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_CXX_STANDARD=17 -DCMAKE_C_COMPILER=clang -DCMAKE_C_STANDARD=11 -DCMAKE_Fortran_COMPILER=gfortran -DCMAKE_Fortran_FLAGS=-std=legacy -DCLANG_DEFAULT_CXX_STDLIB=libc++ -DLLVM_ENABLE_LIBCXX:BOOL=ON -Dlibcxx:BOOL=ON ../root/
   13  rm -fr *
   14  /products/cmake/v3_22_2/Linux64bit+3.10-2.17/bin/cmake -DCMAKE_BUILD_TYPE=Debug '-DCMAKE_IGNORE_PATH=/usr/local/lib;/usr/local/include;/usr/local/bin' -DCMAKE_INSTALL_PREFIX=../inst -Dall:BOOL=OFF -Dasan:BOOL=OFF -Dasserts:BOOL=ON -Dccache:BOOL=OFF -Dclingtest:BOOL=OFF -Dcoverage:BOOL=OFF -Dcxxmodules:BOOL=OFF -Ddev:BOOL=OFF -Ddistcc:BOOL=OFF -Dexceptions:BOOL=ON -Dfail-on-missing:BOOL=ON -Dfortran:BOOL=OFF -Dgminimal:BOOL=ON -Dgnuinstall:BOOL=OFF -Dmemory_termination:BOOL=OFF -Dmemstat:BOOL=OFF -Dminimal:BOOL=OFF -Drootbench:BOOL=OFF -Droottest:BOOL=OFF -Drpath:BOOL=OFF -Druntime_cxxmodules:BOOL=ON -Dshared:BOOL=ON -Dsoversion:BOOL=ON -Dtest_distrd_pyspark:BOOL=OFF -Dtest_distrd_dask:BOOL=OFF -Dtesting:BOOL=OFF -DLLVM_BUILD_TYPE=Debug -DLLVM_ENABLE_ASSERTIONS:BOOL=ON -Dbuiltin_afterimage:BOOL=OFF -Dbuiltin_cfitsio:BOOL=OFF -Dbuiltin_clang:BOOL=ON -Dbuiltin_cling:BOOL=ON -Dbuiltin_cppzmq:BOOL=OFF -Dbuiltin_davix:BOOL=OFF -Dbuiltin_fftw3:BOOL=OFF -Dbuiltin_freetype:BOOL=OFF -Dbuiltin_ftgl:BOOL=OFF -Dbuiltin_gl2ps:BOOL=OFF -Dbuiltin_glew:BOOL=OFF -Dbuiltin_gsl:BOOL=OFF -Dbuiltin_llvm:BOOL=ON -Dbuiltin_lz4:BOOL=OFF -Dbuiltin_lzma:BOOL=OFF -Dbuiltin_nlohmannjson:BOOL=ON -Dbuiltin_openssl:BOOL=OFF -Dbuiltin_openui5:BOOL=ON -Dbuiltin_pcre:BOOL=OFF -Dbuiltin_tbb:BOOL=OFF -Dbuiltin_unuran:BOOL=OFF -Dbuiltin_vc:BOOL=OFF -Dbuiltin_vdt:BOOL=OFF -Dbuiltin_veccore:BOOL=OFF -Dbuiltin_xrootd:BOOL=OFF -Dbuiltin_xxhash:BOOL=OFF -Dbuiltin_zeromq:BOOL=OFF -Dbuiltin_zlib:BOOL=OFF -Dbuiltin_zstd:BOOL=OFF -DXROOTD_ROOT_DIR=/products/xrootd/v5_4_3a/Linux64bit+3.10-2.17-c14-p3912-debug -Dalien:BOOL=OFF -Darrow:BOOL=OFF -Dasimage:BOOL=ON -Dcefweb:BOOL=OFF -Dclad:BOOL=ON -Dcuda:BOOL=OFF -Dcudnn:BOOL=OFF -Ddaos:BOOL=OFF -Ddataframe:BOOL=ON -Ddavix:BOOL=OFF -Ddcache:BOOL=OFF -Dfcgi:BOOL=OFF -Dfftw3:BOOL=ON -Dfitsio:BOOL=OFF -Dgdml:BOOL=ON -Dgfal:BOOL=OFF -Dgsl_shared:BOOL=ON -Dgviz:BOOL=OFF -Dhttp:BOOL=ON -Dimt:BOOL=ON -Djemalloc:BOOL=OFF -Dmathmore:BOOL=ON -Dminuit2:BOOL=ON -Dmlp:BOOL=ON -Dmonalisa:BOOL=OFF -Dmpi:BOOL=OFF -Dmysql:BOOL=OFF -Dodbc:BOOL=OFF -Dopengl:BOOL=ON -Doracle:BOOL=OFF -Dpgsql:BOOL=ON -Dpyroot:BOOL=ON -Dpyroot_legacy:BOOL=OFF -Dpythia6:BOOL=ON -Dpythia6_nolink:BOOL=OFF -Dpythia8:BOOL=OFF -Dqt5web:BOOL=OFF -Dqt6web:BOOL=OFF -Dr:BOOL=OFF -Droofit:BOOL=ON -Droot7:BOOL=ON -Dshadowpw:BOOL=OFF -Dspectrum:BOOL=ON -Dsqlite:BOOL=OFF -Dssl:BOOL=ON -Dtcmalloc:BOOL=OFF -Dtmva-cpu:BOOL=ON -Dtmva-gpu:BOOL=OFF -Dtmva-pymva:BOOL=ON -Dtmva-rmva:BOOL=OFF -Dtmva:BOOL=ON -During:BOOL=OFF -Dunuran:BOOL=OFF -Dvc:BOOL=OFF -Dvdt:BOOL=OFF -Dveccore:BOOL=OFF -Dvecgeom:BOOL=OFF -Dvmc:BOOL=OFF -Dwebgui:BOOL=ON -Dx11:BOOL=ON -Dxml:BOOL=ON -Dxproofd:BOOL=OFF -Dxrootd:BOOL=ON -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_CXX_STANDARD=17 -DCMAKE_C_COMPILER=clang -DCMAKE_C_STANDARD=11 -DCMAKE_Fortran_COMPILER=gfortran -DCMAKE_Fortran_FLAGS=-std=legacy -DCLANG_DEFAULT_CXX_STDLIB=libc++ -DLLVM_ENABLE_LIBCXX:BOOL=ON -Dlibcxx:BOOL=ON ../root/
   15  make -j12 Core
   16   /scratch/obj/core/rootcling_stage1/src/rootcling_stage1 -v4 -f G__Core.cxx -cxxmodule -s /scratch/obj/lib/libCore.so -excludePath /scratch/root -excludePath /scratch/obj/ginclude -excludePath /scratch/obj/externals -excludePath /scratch/obj/builtins -writeEmptyRootPCM -m _Builtin_intrinsics -mByproduct _Builtin_intrinsics -mByproduct ROOT_Foundation_Stage1_NoRTTI -mByproduct ROOT_Foundation_C -mByproduct ROOT_Rtypes -D__CORRECT_ISO_CPP_WCHAR_H_PROTO -D__STDC_NO_COMPLEX__ -D__COMPLEX_H__ -D_COMPLEX_H -D__CLANG_STDATOMIC_H -compilerI/products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include -compilerI/products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/x86_64-unknown-linux-gnu/c++/v1 -compilerI/products/clang/v14_0_6c/Linux64bit+3.10-2.17/include/c++/v1 -compilerI/products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib/clang/14.0.6/include -compilerI/usr/local/include -compilerI/usr/include -compilerI/products/tbb/v2021_7_0/Linux64bit+3.10-2.17-c14/include -compilerI/products/clang/v14_0_6c/Linux64bit+3.10-2.17/lib/clang/14.0.6/include -compilerI/usr/local/include -compilerI/usr/include -I/scratch/obj/include -I/scratch/obj/ginclude -I/scratch/root/core/base/inc -I/scratch/root/core/foundation/inc -I/scratch/root/core/cont/inc -I/scratch/root/core/gui/inc -I/scratch/root/core/meta/inc -I/scratch/root/core/clib/inc -I/scratch/root/core/rint/inc -I/scratch/root/core/zip/inc -I/scratch/root/core/thread/inc -I/scratch/root/core/textinput/inc -I/scratch/root/core/clingutils/inc -I/scratch/root/core/base/v7/inc -I/scratch/root/core/foundation/v7/inc -I/scratch/root/core/unix/inc -I/usr/include ROOT/TErrorDefaultHandler.hxx ROOT/TSequentialExecutor.hxx ROOT/StringConv.hxx Buttons.h Bytes.h Byteswap.h KeySymbols.h MessageTypes.h Riostream.h Rtypes.h TApplication.h TAtt3D.h TAttAxis.h TAttBBox2D.h TAttBBox.h TAttFill.h TAttLine.h TAttMarker.h TAttPad.h TAttText.h TBase64.h TBenchmark.h TBuffer3D.h TBuffer3DTypes.h TBuffer.h TColor.h TColorGradient.h TDatime.h TDirectory.h TEnv.h TException.h TExec.h TFileCollection.h TFileInfo.h TFolder.h TInetAddress.h TMacro.h TMathBase.h TMD5.h TMemberInspector.h TMessageHandler.h TNamed.h TNotifyLink.h TObject.h TObjString.h TParameter.h TPluginManager.h TPoint.h TPRegexp.h TProcessID.h TProcessUUID.h TQClass.h TQCommand.h TQConnection.h TQObject.h TRedirectOutputGuard.h TRefCnt.h TRef.h TRegexp.h TRemoteObject.h TROOT.h TRootIOCtor.h TStopwatch.h TStorage.h TString.h TStringLong.h TStyle.h TSysEvtHandler.h TSystemDirectory.h TSystemFile.h TSystem.h TTask.h TThreadSlots.h TTime.h TTimer.h TTimeStamp.h TUri.h TUrl.h TUUID.h TVersionCheck.h TVirtualAuth.h TVirtualFFT.h TVirtualGL.h TVirtualMonitoring.h TVirtualMutex.h TVirtualPadEditor.h TVirtualPad.h TVirtualPadPainter.h TVirtualPerfStats.h TVirtualPS.h TVirtualQConnection.h TVirtualRWMutex.h TVirtualTableInterface.h TVirtualViewer3D.h TVirtualX.h ROOT/RDirectoryEntry.hxx ROOT/RIndexIter.hxx strlcpy.h snprintf.h strtok.h ROOT/TSeq.hxx TArrayC.h TArrayD.h TArrayF.h TArray.h TArrayI.h TArrayL64.h TArrayL.h TArrayS.h TBits.h TBtree.h TClassTable.h TClonesArray.h TCollection.h TCollectionProxyInfo.h TExMap.h THashList.h THashTable.h TIterator.h TList.h TMap.h TObjArray.h TObjectTable.h TOrdCollection.h TRefArray.h TRefTable.h TSeqCollection.h TSortedList.h TVirtualCollectionProxy.h ESTLType.h RStringView.h TClassEdit.h TError.h ROOT/RLogger.hxx ROOT/RNotFn.hxx ROOT/RRangeCast.hxx ROOT/RSpan.hxx ROOT/RStringView.hxx ROOT/StringUtils.hxx ROOT/span.hxx ROOT/TypeTraits.hxx TUnixSystem.h root_std_complex.h GuiTypes.h TApplicationImp.h TBrowser.h TBrowserImp.h TCanvasImp.h TClassMenuItem.h TContextMenu.h TContextMenuImp.h TControlBarImp.h TGuiFactory.h TInspectorImp.h TObjectSpy.h TToggleGroup.h TToggle.h TBaseClass.h TClassGenerator.h TClass.h TClassRef.h TClassStreamer.h TDataMember.h TDataType.h TDictAttributeMap.h TDictionary.h TEnumConstant.h TEnum.h TFileMergeInfo.h TFunction.h TFunctionTemplate.h TGenericClassInfo.h TGlobal.h TInterpreter.h TInterpreterValue.h TIsAProxy.h TListOfDataMembers.h TListOfEnums.h TListOfEnumsWithLock.h TListOfFunctions.h TListOfFunctionTemplates.h TMemberStreamer.h TMethodArg.h TMethodCall.h TMethod.h TProtoClass.h TRealData.h TSchemaHelper.h TSchemaRule.h TSchemaRuleSet.h TStatusBitsChecker.h TStreamerElement.h TStreamer.h TVirtualIsAProxy.h TVirtualRefProxy.h TVirtualStreamerInfo.h TVirtualArray.h TVirtualObject.h Getline.h /scratch/root/core/base/inc/LinkDef.h 
   17  vi /products/clang/v14_0_6c/Linux64bit+3.10-2.17/bin/../include/c++/v1/module.modulemap
   18  make -j12 Core
   19  history

The only difference to the log you provided in the cmake configuration is that I had to remove -GNinja as it could not find ninja. The vi command just commented this in the modulemap:

...
module std [system] {
 // export std_config
...

greenc-FNAL commented 1 year ago

The good news is that I have been able to reproduce your success with our build of 6.24/04 and 6.24/06. Unfortunately, I was also able to reproduce our failures with 6.28/00-rc1 in the same environment. I'd be grateful if you could turn your attention to investigating why the fix does not work for more recent ROOT source.

BTW: in order to use Ninja, one needs simply to execute setup ninja before executing the CMake command.

greenc-FNAL commented 1 year ago

Possibly related: is there any reason that the pre-release source (and presumably master) apparently uses its own std.modulemap rather than looking for one provided with the external compiler's standard library first?

vgvassilev commented 1 year ago

Possibly related: is there any reason that the pre-release source (and presumably master) apparently uses its own std.modulemap rather than looking for one provided with the external compiler's standard library first?

The modulemap is only available in libcxx and we use it. We "mount" a custom one for glibc++ which does not have any. If you invoke rootcling with -v4 you will see more details.

greenc-FNAL commented 1 year ago

@vgvassilev I have uploaded a snapshot image. The shell history will show exactly what was done, and the relevant build log is /scratch/build-root-v6-28-00-patches-c14-p3913-debug.

root-project / root