search

root-project / root

The official repository for ROOT: analyzing, storing and visualizing big data, scientifically
https://root.cern
Other
2.71k stars 1.28k forks source link

Systematic build failure of ROOT master in the LCG DEBUG builds since Jan 15th #9594

Closed gganis closed 9 months ago

gganis commented 2 years ago

Describe the bug

Since three nights (Jan 15th), the dev3 (ROOT-master) debug builds fail systematically with an error in

clang::ASTReader::ReadModuleMapFileBlock(llvm::SmallVector<unsigned long, 64u>&, clang::serialization::ModuleFile&, clang::serialization::ModuleFile const*, unsigned int)

See for example this .

Expected behavior

A successful build ;-) .

To Reproduce

LCG nightlies (lcgcmake). Did not try yet in standalone.

Setup

  1. ROOT master
  2. Operating system: centos7, centos8
  3. How you obtained ROOT, such as dnf install / binary download / you built it yourself: built from source (github)

Additional context

None for the moment

gganis commented 2 years ago

Update: v6-26-00-patches also affected since last night.

gganis commented 2 years ago

Update: another problem which seems connected with this change is that PyRDF and gaudi_tests fail for the optimized builds, the only one we can test (see, for example, here and here).

So all the LCG nightlies seem to be affected now.

Axel-Naumann commented 2 years ago

@vgvassilev FYI

Relevant part of the backtrace:

#7  <signal handler called> kSigSegmentationViolation
#8  0x00007f72cfc73154 in clang::ASTReader::ReadModuleMapFileBlock(llvm::SmallVector<unsigned long, 64u>&, clang::serialization::ModuleFile&, clang::serialization::ModuleFile const*, unsigned int) () from /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/libCling.so
#9  0x00007f72cfc6fbb4 in clang::ASTReader::ReadControlBlock(clang::serialization::ModuleFile&, llvm::SmallVectorImpl<clang::ASTReader::ImportedModule>&, clang::serialization::ModuleFile const*, unsigned int) () from /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/libCling.so
#10 0x00007f72cfc7286f in clang::ASTReader::ReadASTCore(llvm::StringRef, clang::serialization::ModuleKind, clang::SourceLocation, clang::serialization::ModuleFile*, llvm::SmallVectorImpl<clang::ASTReader::ImportedModule>&, long, long, clang::ASTFileSignature, unsigned int) () from /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/libCling.so
#11 0x00007f72cfc70584 in clang::ASTReader::ReadControlBlock(clang::serialization::ModuleFile&, llvm::SmallVectorImpl<clang::ASTReader::ImportedModule>&, clang::serialization::ModuleFile const*, unsigned int) () from /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/libCling.so
#12 0x00007f72cfc7286f in clang::ASTReader::ReadASTCore(llvm::StringRef, clang::serialization::ModuleKind, clang::SourceLocation, clang::serialization::ModuleFile*, llvm::SmallVectorImpl<clang::ASTReader::ImportedModule>&, long, long, clang::ASTFileSignature, unsigned int) () from /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/libCling.so
#13 0x00007f72cfc7e1b2 in clang::ASTReader::ReadAST(llvm::StringRef, clang::serialization::ModuleKind, clang::SourceLocation, unsigned int, llvm::SmallVectorImpl<clang::ASTReader::ImportedSubmodule>*) () from /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/libCling.so
#14 0x00007f72cf99b1bf in clang::CompilerInstance::loadModule(clang::SourceLocation, llvm::ArrayRef<std::pair<clang::IdentifierInfo*, clang::SourceLocation> >, clang::Module::NameVisibilityKind, bool) () from /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/libCling.so
#15 0x00007f72d01e8bc2 in clang::Sema::ActOnModuleImport(clang::SourceLocation, clang::SourceLocation, clang::SourceLocation, llvm::ArrayRef<std::pair<clang::IdentifierInfo*, clang::SourceLocation> >) () from /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/libCling.so
#16 0x00007f72cf4cfbd7 in cling::Interpreter::loadModule(clang::Module*, bool) () from /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/libCling.so
#17 0x00007f72cf2aa7d3 in LoadModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::Interpreter&) (ModuleName=..., interp=...) at /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/metacling/src/TCling.cxx:1053
#18 0x00007f72cf2c60e6 in LoadModules(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, cling::Interpreter&) (modules=..., interp=...) at /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/metacling/src/TCling.cxx:1062
#19 0x00007f72cf2a247c in RegisterCxxModules(cling::Interpreter&) (clingInterp=...) at /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/metacling/src/TCling.cxx:1231
#20 0x00007f72cf2a13ef in TCling::TCling(char const*, char const*, char const* const*) (this=0x10a0370, name=0x7f72d2c43a33 "C++", title=0x7f72d1f99311 "cling C++ Interpreter", argv=0x7ffd89271620) at /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/metacling/src/TCling.cxx:1585
#21 0x00007f72cf29b48b in CreateInterpreter (interpLibHandle=0x10905e0, argv=0x7ffd89271620) at /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/metacling/src/TCling.cxx:601
#22 0x00007f72d4fa92a1 in TROOT::InitInterpreter() (this=0x7f72d54ea4b8 <ROOT::Internal::GetROOT1()::alloc>) at /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/base/src/TROOT.cxx:2073
Axel-Naumann commented 2 years ago

@gganis what's the cmake invocation of this build?

Axel-Naumann commented 2 years ago

The second cases you report seem unrelated at a first look:

In module 'Vc':
/build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/Vc/1.4.2/x86_64-centos7-gcc11-opt/include/Vc/common/simdarray.h:501:18: error: 'Vc_1::SimdArray<type-parameter-0-0, N, type-parameter-0-2, N>::SimdArray<type-parameter-0-0, N, type-parameter-0-2, N>' from module 'Vc' is not present in definition of 'SimdArray<type-parameter-0-0, N, type-parameter-0-2, N>' in module 'Vc'
    Vc_INTRINSIC SimdArray(private_init_t, VectorType &&x) : data(std::move(x)) {}
                 ^

and

<<< cling interactive line includer >>>: fatal error: module file '/build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/Vc.pcm' is out of date and needs to be rebuilt: signature mismatch
<<< cling interactive line includer >>>: note: imported by module 'MathCore' in '/cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Tue/ROOT/HEAD/x86_64-centos7-gcc11-opt/lib/MathCore.pcm'
input_line_4:2:2: error: unknown type name 'include'
#include "TError.h"
 ^
gganis commented 2 years ago

The CMake invocation is here. This is an example of concretization for gcc 9.2, debug:

set(command "/cvmfs/sft.cern.ch/lcg/contrib/CMake/3.20.0/Linux-x86_64/bin/cmake;-DCMAKE_BUILD_TYPE=Debug;-DCMAKE_INSTALL_PREFIX=/build/jenkins/workspace/lcg_nightly_pipeline/install/dev3
/ROOT/HEAD/x86_64-centos7-gcc9-dbg;-Droot7=ON;-Dpyroot=ON;-DPYTHON_EXECUTABLE=/build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/Python/3.9.6/x86_64-centos7-gcc9-dbg/bin/python;-
Dbuiltin_lz4=ON;-Dbuiltin_pcre=ON;-Dbuiltin_xxhash=ON;-Dbuiltin_ftgl=ON;-Dbuiltin_afterimage=ON;-Dbuiltin_glew=ON;-Dbuiltin_unuran=ON;-Dbuiltin_zstd=ON;-Dcintex=ON;-Ddavix=ON;-Dexception
s=ON;-Dexplicitlink=ON;-Dfftw3=ON;-Dgdml=ON;-Dgsl_shared=ON;-Dhttp=ON;-Dgenvector=ON;-Dvc=ON;-Dmathmore=ON;-Dminuit2=ON;-Dmysql=ON;-Dopengl=ON;-Dpgsql=OFF;-Dpythia6=OFF;-Dpythia8=OFF;-Dr
eflex=ON;-Dr=OFF;-Droofit=ON;-Dssl=ON;-Dunuran=ON;-Dfortran=ON;-Dxft=ON;-Dxml=ON;-Dxrootd=ON;-Dzlib=ON;-DCINTMAXSTRUCT=36000;-DCINTMAXTYPEDEF=36000;-DCINTLONGLINE=4096;-Dfail-on-missing=
ON;-Dfitsio=ON;-Doracle=OFF;-Dgfal=OFF;-DCMAKE_CXX_STANDARD=17;-Ddcache=ON;-Dgfal=ON;-DGFAL_DIR=/build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/Grid/gfal/1.13.0-0/x86_64-cento
s7-gcc9-dbg;-DSRM_IFCE_DIR=/build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/Grid/srm-ifce/1.13.0-0/x86_64-centos7-gcc9-dbg;-Doracle=ON;-DORACLE_HOME=/build/jenkins/workspace/lc
g_nightly_pipeline/install/dev3/oracle/19.11.0.0.0/x86_64-centos7-gcc9-dbg;-Dbuiltin_nlohmannjson=OFF;-Droofit_multiprocess=OFF;-GUnix Makefiles;/build/jenkins/workspace/lcg_nightly_pipe
line/build/projects/ROOT-HEAD/src/ROOT/HEAD")

And this is the corresponding log:

-- The C compiler identification is GNU 9.2.0
-- The CXX compiler identification is GNU 9.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "1.8.3.1")
-- Detected ROOT_VERSION 6.27.01
-- Checking internet connectivity...
-- Yes
-- Looking for Python
-- Found Python3: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/Python/3.9.6/x86_64-centos7-gcc9-dbg/bin/python (found version "3.9.6") found components: Interpreter Develop
ment NumPy Development.Module Development.Embed
-- Looking for a Fortran compiler
-- Looking for a Fortran compiler - /cvmfs/sft.cern.ch/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/gfortran
-- The Fortran compiler identification is GNU 9.2.0
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/gfortran - skipped
-- Checking whether /cvmfs/sft.cern.ch/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/gfortran supports Fortran 90
-- Checking whether /cvmfs/sft.cern.ch/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/gfortran supports Fortran 90 - yes
-- Found GCC. Major version 9, minor version 2
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Performing Test GLIBCXX_USE_CXX11_ABI
-- Performing Test GLIBCXX_USE_CXX11_ABI - Success
-- ROOT Platform: linux
-- ROOT Compiler: GNU 9.2.0
-- ROOT Processor: x86_64
-- ROOT Architecture: linuxx8664gcc
-- Build Type: 'Debug' (flags = '-g')
-- Compiler Flags:  -std=c++17 -Wno-implicit-fallthrough -Wno-noexcept-type -pipe  -Wshadow -Wall -W -Woverloaded-virtual -fsigned-char -pthread -g
-- ROOT default compression algorithm: zlib
-- PyROOT will be built for version 3.9.6
-- Looking for ZLib
-- Found ZLIB: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/zlib/1.2.11/x86_64-centos7-gcc9-dbg/lib/libz.so (found version "1.2.11")
-- Looking for nlohmann/json.hpp
-- Found nlohmann_json: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/jsonmcpp/3.9.1/x86_64-centos7-gcc9-dbg/lib64/cmake/nlohmann_json/nlohmann_jsonConfig.cmake (found versi
on "3.9.1")
-- Looking for Freetype
-- Found Freetype: /usr/lib64/libfreetype.so (found version "2.8.0")
-- Looking for LZMA
-- Looking for lzma_auto_decoder in /usr/lib64/liblzma.so
-- Looking for lzma_auto_decoder in /usr/lib64/liblzma.so - found
-- Looking for lzma_easy_encoder in /usr/lib64/liblzma.so
-- Looking for lzma_easy_encoder in /usr/lib64/liblzma.so - found
-- Looking for lzma_lzma_preset in /usr/lib64/liblzma.so
-- Looking for lzma_lzma_preset in /usr/lib64/liblzma.so - found
-- Found LibLZMA: /usr/lib64/liblzma.so (found version "5.2.2")
-- Performing Test CXX_FLAG_STD_CPLUSPLUS11
-- Performing Test CXX_FLAG_STD_CPLUSPLUS11 - Success
-- Performing Test C_FLAG_STD_C99
-- Performing Test C_FLAG_STD_C99 - Success
-- Performing Test C_FLAG_WALL
-- Performing Test C_FLAG_WALL - Success
-- Performing Test CXX_FLAG_WALL
-- Performing Test CXX_FLAG_WALL - Success
-- Performing Test C_FLAG_WEXTRA
-- Performing Test C_FLAG_WEXTRA - Success
-- Performing Test CXX_FLAG_WEXTRA
-- Performing Test CXX_FLAG_WEXTRA - Success
-- Performing Test C_FLAG_WUNDEF
-- Performing Test C_FLAG_WUNDEF - Success
-- Performing Test CXX_FLAG_WUNDEF
-- Performing Test CXX_FLAG_WUNDEF - Success
-- Performing Test C_FLAG_WSHADOW
-- Performing Test C_FLAG_WSHADOW - Success
-- Performing Test CXX_FLAG_WSHADOW
-- Performing Test CXX_FLAG_WSHADOW - Success
-- Performing Test C_FLAG_WCAST_ALIGN
-- Performing Test C_FLAG_WCAST_ALIGN - Success
-- Performing Test CXX_FLAG_WCAST_ALIGN
-- Performing Test CXX_FLAG_WCAST_ALIGN - Success
-- Performing Test C_FLAG_WCAST_QUAL
-- Performing Test C_FLAG_WCAST_QUAL - Success
-- Performing Test CXX_FLAG_WCAST_QUAL
-- Performing Test CXX_FLAG_WCAST_QUAL - Success
-- Performing Test C_FLAG_WSTRICT_PROTOTYPES
-- Performing Test C_FLAG_WSTRICT_PROTOTYPES - Success
-- Performing Test C_FLAG_DDEBUGLEVEL_1
-- Performing Test C_FLAG_DDEBUGLEVEL_1 - Success
-- Performing Test CXX_FLAG_DDEBUGLEVEL_1
-- Performing Test CXX_FLAG_DDEBUGLEVEL_1 - Success
-- Looking for X11
-- Found X11: /usr/include
-- Looking for XOpenDisplay in /usr/lib64/libX11.so;/usr/lib64/libXext.so
-- Looking for XOpenDisplay in /usr/lib64/libX11.so;/usr/lib64/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - found
-- X11_INCLUDE_DIR: /usr/include
-- X11_LIBRARIES: /usr/lib64/libSM.so;/usr/lib64/libICE.so;/usr/lib64/libX11.so;/usr/lib64/libXext.so
-- X11_Xpm_INCLUDE_PATH: /usr/include
-- X11_Xpm_LIB: /usr/lib64/libXpm.so
-- X11_Xft_INCLUDE_PATH: /usr/include
-- X11_Xft_LIB: /usr/lib64/libXft.so
-- X11_Xext_INCLUDE_PATH: /usr/include
-- X11_Xext_LIB: /usr/lib64/libXext.so
-- Could NOT find GIF (missing: GIF_LIBRARY GIF_INCLUDE_DIR)
-- Found TIFF: /usr/lib64/libtiff.so (found version "4.0.3")
-- Found PNG: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/png/1.6.37/x86_64-centos7-gcc9-dbg/lib/libpng.so (found version "1.6.37")
-- Found JPEG: /usr/lib64/libjpeg.so (found version "62")
-- Building AfterImage library included in ROOT itself
-- Looking for GSL
-- Found PkgConfig: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/pkg_config/0.29.2/x86_64-centos7-gcc9-dbg/bin/pkg-config (found version "0.29.2")
-- Found GSL: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/GSL/2.7/x86_64-centos7-gcc9-dbg/include (found suitable version "2.7", minimum required is "1.10")
-- Looking for OpenGL
-- Found OpenGL: /usr/lib64/libOpenGL.so
-- Looking for gl2ps
-- Found gl2ps: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/gl2ps/1.4.0/x86_64-centos7-gcc9-dbg/lib/libgl2ps.so
-- Looking for LibXml2
-- Found LibXml2: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/libxml2/2.9.10/x86_64-centos7-gcc9-dbg/lib/libxml2.so (found version "2.9.10")
-- Found OpenSSL: /usr/lib64/libcrypto.so (found version "1.0.2k")
-- Looking for MySQL
-- Found MySQL: /cvmfs/sft-nightlies.cern.ch/lcg/latest/mysql/10.4.20-00b73/x86_64-centos7-gcc9-dbg/include/mysql
-- Looking for Oracle
-- Found sqlplus version: 19.0.0.0.0
-- Guessed ORACLE_OCI_VERSION value: 9
-- Found Oracle: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/oracle/19.11.0.0.0/x86_64-centos7-gcc9-dbg/lib/libocci.so;/build/jenkins/workspace/lcg_nightly_pipeline/instal
l/dev3/oracle/19.11.0.0.0/x86_64-centos7-gcc9-dbg/lib/libclntsh.so;/build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/oracle/19.11.0.0.0/x86_64-centos7-gcc9-dbg/lib/libnnz19.so
-- Looking for SQLite
-- Found Sqlite: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/sqlite/3320300/x86_64-centos7-gcc9-dbg/include
-- Looking for FFTW3
-- Found fftw3 includes at /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/fftw3/3.3.8/x86_64-centos7-gcc9-dbg/include
-- Found fftw3 library at /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/fftw3/3.3.8/x86_64-centos7-gcc9-dbg/lib/libfftw3.so
-- Looking for CFITSIO
-- Found CFITSIO: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/cfitsio/3.48/x86_64-centos7-gcc9-dbg/lib/libcfitsio.so (found version "3.48")
-- Looking for XROOTD
-- Found Xrootd version num: 5.3.1 (setting -DROOTXRDVERS=500030001)
--              libXrdMain not found: xproofd will be a wrapper around xrootd
--              libXrdClient not found: use built-in
--              include_dirs: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/xrootd/5.3.1/x86_64-centos7-gcc9-dbg/include/xrootd;/build/jenkins/workspace/lcg_nightly_pipeline
/build/projects/ROOT-HEAD/src/ROOT/HEAD/proof/xrdinc
--              libraries: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/xrootd/5.3.1/x86_64-centos7-gcc9-dbg/lib64/libXrdUtils.so;/build/jenkins/workspace/lcg_nightly_pipel
ine/install/dev3/xrootd/5.3.1/x86_64-centos7-gcc9-dbg/lib64/libXrdCl.so
-- Found GFAL: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/Grid/gfal/1.13.0-0/x86_64-centos7-gcc9-dbg/include/.
-- Found DCAP: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/Grid/dcap/2.47.7-1/x86_64-centos7-gcc9-dbg/include
-- Checking for module 'davix>=0.6.4'
--   Found davix, version 0.7.6
-- Looking for TBB
-- Found TBB: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/tbb/2020_U2/x86_64-centos7-gcc9-dbg (found suitable version "2020.2", minimum required is "2018")
-- Performing Test tbb_exception_result
-- Performing Test tbb_exception_result - Success
-- Detected Compiler: GCC 9.2.0
-- Performing Test check_cxx_compiler_flag__Wabi
-- Performing Test check_cxx_compiler_flag__Wabi - Success
-- Performing Test check_cxx_compiler_flag__fabi_version_0
-- Performing Test check_cxx_compiler_flag__fabi_version_0 - Success
-- Performing Test check_cxx_compiler_flag__fabi_compat_version_0
-- Performing Test check_cxx_compiler_flag__fabi_compat_version_0 - Success
-- Performing Test check_cxx_compiler_flag__ffp_contract_fast
-- Performing Test check_cxx_compiler_flag__ffp_contract_fast - Success
-- target changed from "" to "auto"
-- Detected CPU: broadwell
-- Performing Test check_cxx_compiler_flag__march_broadwell
-- Performing Test check_cxx_compiler_flag__march_broadwell - Success
-- Performing Test check_cxx_compiler_flag__msse2
-- Performing Test check_cxx_compiler_flag__msse2 - Success
-- Performing Test check_cxx_compiler_flag__msse3
-- Performing Test check_cxx_compiler_flag__msse3 - Success
-- Looking for C++ include pmmintrin.h
-- Looking for C++ include pmmintrin.h - found
-- Performing Test check_cxx_compiler_flag__mssse3
-- Performing Test check_cxx_compiler_flag__mssse3 - Success
-- Looking for C++ include tmmintrin.h
-- Looking for C++ include tmmintrin.h - found
-- Performing Test check_cxx_compiler_flag__msse4_1
-- Performing Test check_cxx_compiler_flag__msse4_1 - Success
-- Looking for C++ include smmintrin.h
-- Looking for C++ include smmintrin.h - found
-- Performing Test check_cxx_compiler_flag__msse4_2
-- Performing Test check_cxx_compiler_flag__msse4_2 - Success
-- Performing Test check_cxx_compiler_flag__mavx
-- Performing Test check_cxx_compiler_flag__mavx - Success
-- Looking for C++ include immintrin.h
-- Looking for C++ include immintrin.h - found
-- Performing Test check_cxx_compiler_flag__mfma
-- Performing Test check_cxx_compiler_flag__mfma - Success
-- Performing Test check_cxx_compiler_flag__mbmi2
-- Performing Test check_cxx_compiler_flag__mbmi2 - Success
-- Performing Test check_cxx_compiler_flag__mavx2
-- Performing Test check_cxx_compiler_flag__mavx2 - Success
-- Performing Test check_cxx_compiler_flag__mno_sse4a
-- Performing Test check_cxx_compiler_flag__mno_sse4a - Success
-- Performing Test check_cxx_compiler_flag__mno_xop
-- Performing Test check_cxx_compiler_flag__mno_xop - Success
-- Performing Test check_cxx_compiler_flag__mno_fma4
-- Performing Test check_cxx_compiler_flag__mno_fma4 - Success
-- Performing Test check_cxx_compiler_flag__mno_avx512f
-- Performing Test check_cxx_compiler_flag__mno_avx512f - Success
-- Performing Test check_cxx_compiler_flag__mno_avx512vl
-- Performing Test check_cxx_compiler_flag__mno_avx512vl - Success
-- Performing Test check_cxx_compiler_flag__mno_avx512pf
-- Performing Test check_cxx_compiler_flag__mno_avx512pf - Success
-- Performing Test check_cxx_compiler_flag__mno_avx512er
-- Performing Test check_cxx_compiler_flag__mno_avx512er - Success
-- Performing Test check_cxx_compiler_flag__mno_avx512cd
-- Performing Test check_cxx_compiler_flag__mno_avx512cd - Success
-- Performing Test check_cxx_compiler_flag__mno_avx512dq
-- Performing Test check_cxx_compiler_flag__mno_avx512dq - Success
-- Performing Test check_cxx_compiler_flag__mno_avx512bw
-- Performing Test check_cxx_compiler_flag__mno_avx512bw - Success
-- Performing Test check_cxx_compiler_flag__mno_avx512ifma
-- Performing Test check_cxx_compiler_flag__mno_avx512ifma - Success
-- Performing Test check_cxx_compiler_flag__mno_avx512vbmi
-- Performing Test check_cxx_compiler_flag__mno_avx512vbmi - Success
-- Looking for VDT
-- Found Vdt: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/vdt/0.4.3/x86_64-centos7-gcc9-dbg/include (found suitable version "0.4", minimum required is "0.4")
-- Cannot select cudnn without selecting cuda or tmva-gpu. Option is ignored
-- Looking for BLAS for optional parts of TMVA
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - not found
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Found BLAS: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/blas/0.3.17.openblas/x86_64-centos7-gcc9-dbg/lib/libopenblas.so
-- Performing Test ROOT_HAVE_CXX_ATOMICS_WITHOUT_LIB
-- Performing Test ROOT_HAVE_CXX_ATOMICS_WITHOUT_LIB - Success
-- Building LLVM in 'Release' mode.
-- The ASM compiler identification is GNU
-- Found assembler: /cvmfs/sft.cern.ch/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/gcc
-- Could NOT find Z3: Found unsuitable version "0.0.0", but required is at least "4.7.1" (found Z3_LIBRARIES-NOTFOUND)
-- Looking for backtrace
-- Looking for backtrace - found
-- backtrace facility detected in default set of libraries
-- Found Backtrace: /usr/include
-- Native target architecture is X86
-- Threads disabled.
-- Doxygen disabled.
-- Go bindings disabled.
-- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH)
-- OCaml bindings disabled.
-- Could NOT find Python module pygments
-- Could NOT find Python module pygments.lexers.c_cpp
-- Could NOT find Python module yaml
-- LLVM host triple: x86_64-unknown-linux-gnu
-- LLVM default target triple: x86_64-unknown-linux-gnu
-- Building with -fPIC
-- Found PythonInterp: /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/Python/3.9.6/x86_64-centos7-gcc9-dbg/bin/python (found version "3.9.6")
-- Constructing LLVMBuild project information
-- Linker detection: GNU ld
-- Targeting X86
-- Targeting NVPTX
-- Clang version: 9.0.1
-- Cling version (from VERSION file): ROOT_1.0~dev
-- Cling will look for C++ headers in '/cvmfs/sft.cern.ch/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/../lib/gcc/x86_64-pc-linux-gnu/9.2.0/../../../../include/c++/9.2.0:/cvmfs/sft.ce
rn.ch/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/../lib/gcc/x86_64-pc-linux-gnu/9.2.0/../../../../include/c++/9.2.0/x86_64-pc-linux-gnu:/cvmfs/sft.cern.ch/lcg/releases/gcc/9.2.0-afc
57/x86_64-centos7/bin/../lib/gcc/x86_64-pc-linux-gnu/9.2.0/../../../../include/c++/9.2.0/backward' at runtime.
-- And then fallback to: 'g++'
-- Performing Test CXX_HAS_fno_rtti
-- Performing Test CXX_HAS_fno_rtti - Success
-- Performing Test C_HAS_Wno_strict_overflow
-- Performing Test C_HAS_Wno_strict_overflow - Success
-- Performing Test C_HAS_Wno_maybe_uninitialized
-- Performing Test C_HAS_Wno_maybe_uninitialized - Success
-- Performing Test C_HAS_Wno_parentheses_equality
-- Performing Test C_HAS_Wno_parentheses_equality - Success
-- Looking for m
-- Looking for m - not found
-- Performing Test GCC_HAS_BIDI_CHARS_FLAG
-- Performing Test GCC_HAS_BIDI_CHARS_FLAG - Failed
-- Performing Test found_setresuid
-- Performing Test found_setresuid - Success
-- Performing Test found_stdstringview
-- Performing Test found_stdstringview - Success
-- Performing Test found_stod_stringview
-- Performing Test found_stod_stringview - Failed
-- Performing Test found_opplusequal_stringview
-- Performing Test found_opplusequal_stringview - Success
-- Performing Test found_stdapply
-- Performing Test found_stdapply - Success
-- Performing Test found_stdinvoke
-- Performing Test found_stdinvoke - Success
-- Performing Test found_stdindexsequence
-- Performing Test found_stdindexsequence - Success
-- Performing Test found_attribute_always_inline
-- Performing Test found_attribute_always_inline - Success
-- Performing Test has_found_attribute_noinline
-- Performing Test has_found_attribute_noinline - Success
-- Performing Test found_hardware_interference_size
-- Performing Test found_hardware_interference_size - Failed
Running /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT/HEAD/build/unix/compiledata.sh
Making /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/ginclude/compiledata.h
-- ROOT Configuration

System          Linux-4.18.0-305.12.1.el8_4.x86_64
Processor       16 core Intel Core Processor (Broadwell, IBRS) (x86_64)
Build type      Debug
Install path    /build/jenkins/workspace/lcg_nightly_pipeline/install/dev3/ROOT/HEAD/x86_64-centos7-gcc9-dbg
Compiler        GNU 9.2.0
Compiler flags:
C                -Wno-implicit-fallthrough -pipe -Wall -W -pthread -g
C++              -std=c++17 -Wno-implicit-fallthrough -Wno-noexcept-type -pipe  -Wshadow -Wall -W -Woverloaded-virtual -fsigned-char -pthread -g
Linker flags:
Executable       -rdynamic
Module
Shared           -Wl,--no-undefined -Wl,--hash-style="both"

-- Enabled support for:  asimage asserts builtin_afterimage builtin_clang builtin_cling builtin_ftgl builtin_glew builtin_llvm builtin_lz4 builtin_openui5 builtin_pcre builtin_unuran bui
ltin_xxhash builtin_zstd clad dataframe davix dcache exceptions fftw3 fitsio fortran gdml gfal gsl_shared http imt mathmore mlp minuit2 mysql opengl oracle pyroot roofit webgui root7 rpa
th runtime_cxxmodules shared sqlite ssl tmva tmva-cpu tmva-pymva spectrum unuran vc vdt x11 xml xrootd
-- Configuring done
-- Generating done
-- Build files have been written to: /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build
gganis commented 2 years ago

The second cases you report seem unrelated at a first look

Well, a part the timing correlation, the changes triggering the problem are the ones ported to v6-26-00-patches on Jan 16, 2022, which are related to Vc.

vgvassilev commented 2 years ago

Hi @gganis, does reverting https://github.com/root-project/root/pull/9567 help?

gganis commented 2 years ago

Hi @vgvassilev, yes, reverting this 5 commits fixes the problem.

gganis commented 2 years ago

Any plan when this can be addressed, at least for v6-26-00-patches? The LCG nightlies are not very usable at the moment. Can I suggest to revet the patch for v6-26-00-patches and re-port it when it is fixed on the master ?

Axel-Naumann commented 2 years ago

@vgvassilev expects to revert still today.

Axel-Naumann commented 2 years ago

Is this addressed? Can this be closed?

andresailer commented 2 years ago

No, still open: https://github.com/root-project/root/pull/9641

andresailer commented 2 years ago

The real solution of #9736 is probably related to this issue. The ROOT master branch is still failing in our builds.

vgvassilev commented 2 years ago

@andresailer, it looks like we have externally built Vc and we do not pass in the location of its header files. IIRC, we had the same issue for the cuda builds and we solved it by passing the include path in the env variable ROOT_INCLUDE_PATH. For example, something like that fixes the crash for me: ROOT_INCLUDE_PATH=/build/sailer/testReprRoot/testLcgcmake/install/dev3/Vc/1.4.2/x86_64-centos7-gcc10-dbg/include/ root.exe -l -b -q

In order to move forward this issue needs to be fixed on the LCG side. I am cc-ing @peremato as we did such a fix some time ago for the ROOT CUDA builds. I bet he knows where to add it as he already did it once.

andresailer commented 2 years ago

I see, we can add that, thank you for the explanation. Why can't one add this environment variable to the invocation in the ROOT build in ROOT's cmake?

vgvassilev commented 2 years ago

I see, we can add that, thank you for the explanation. Why can't one add this environment variable to the invocation in the ROOT build in ROOT's cmake?

The build failure is a just a symptom of the issue. In fact, ROOT was already built and in its final step it runs a sanity check something like root.exe -l -b -q tutorials/hsimple.C to catch such basic issues early.

I'd be happy if we could solve this in a nicer way, however I do not think ROOT has enough information about its build process (and I think it should not). For example, even if we detect where Vc (or other relevant information) was at build time, this does not mean a lot. We cannot be sure that it stays at the same place on the deployment node. For instance, CMSSW-like setups then, at install time, reshuffle things around quite a bit. That environment variable is there because of that - essentially provides a way for "devops" (to use a modern word here) to express extra knowledge which is (nearly) impossible to deduce at build time.

In addition, I think it'd be pretty bad if ROOT modified users' environment unknowingly.

Axel-Naumann commented 2 years ago

@vgvassilev would it make sense to export ROOT_INCLUDE_PATH for the sanity check only, i.e. root.exe -l -b -q tutorials/hsimple.C as that's run during build time? I find it okay to require ROOT_INCLUDE_PATH to point to things once installed, when running ROOT - but it might be better for ROOT's build to "survive" without the need to specify ROOT_INCLUDE_PATH already at build time?

vgvassilev commented 2 years ago

That will solve the build issue which is just a symptom. However, it will not solve the same problem at deployment time. It seems to be equivalent to removing the root.exe -l -b -q tutorials/hsimple.C check altogether.

Even if we decided to go that route, I am not sure if we have that information in the right places in the build system.

Axel-Naumann commented 2 years ago

However, it will not solve the same problem at deployment time.

You assume that build and deployment have a similar settings; not sure that's the case.

As another argument for specifying this during ROOT's build, we also specify LD_LIBRARY_PATH and select which root.exe to run during the hsimple check. How is ROOT_INCLUDE_PATH different from that?

I am not sure if we have that information in the right places in the build system.

We can do it on a case by case basis. We know which "builtins" we depend on wrt runtime includes, and we know where they are during configure / build time. For Vc it's Vc_INCLUDE_DIR.

andresailer commented 2 years ago

Here is how we would set ROOT_INCLUDE_PATH for the build https://gitlab.cern.ch/sft/lcgcmake/-/commit/f50af97822f3fc4dba9fd56bee3cb76c39fe5aeb

For cuda we had to add a location to ROOT_INCLUDE_PATH for the view after the installation https://gitlab.cern.ch/sft/lcgcmake/-/commit/cc14dbbb56a1cb4b59e68c11b9df3f75c21fc1ba (Vc is already part of the standard include directory of the view)

Axel-Naumann commented 2 years ago

If we understand correctly, this should not block 6.26n anymore as the patches that cause this failure have been reverted in 6.26 (they are still present in master).

andresailer commented 2 years ago

Correct, 6-26-00-patches is building fine. We also applied a patch for building root master (https://gitlab.cern.ch/sft/lcgcmake/-/commit/f50af97822f3fc4dba9fd56bee3cb76c39fe5aeb), just let us know when we can test a fix from your side for the master branch.

Axel-Naumann commented 2 years ago

A fix should be this: https://github.com/root-project/root/pull/9765 - I'd certainly appreciate a confirmation! Else I'll just merge and you tell me then.

andresailer commented 2 years ago

While ROOT master is now building successfully for us, there are some downstream packages that suffer from this change. They can probably be fixed by exporting the ROOT_INCLUDE_PATH, but to better understand where things go wrong I made a very simple repeater. I think on any centos7 with the sft cvmfs this should work

source /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/CMake/3.20.0/x86_64-centos7-gcc10-dbg/CMake-env.sh
source /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc10-dbg/ROOT-env.sh

git clone https://gitlab.cern.ch/sailer/root_repeater.git
cd root_repeater
mkdir build
cd build

cmake -D nlohmann_json_DIR=$JSONMCPP__HOME/lib64/cmake/nlohmann_json \
      -D Python_ROOT_DIR=$PYTHON__HOME \
      -D Vc_DIR=$VC__HOME/lib/cmake/Vc \
      ..
make
./RunTest

Just linking an executable to ROOTTPython (https://gitlab.cern.ch/sailer/root_repeater/-/blob/master/CMakeLists.txt#L17) leads to an error like this

./RunTest 
<<< cling interactive line includer >>>: fatal error: module file '/build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/Vc.pcm' not found: module file not found
<<< cling interactive line includer >>>: note: imported by module 'MathCore' in '/cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc10-dbg/lib/MathCore.pcm'
Failed to load module MathCore
Failed to load module Hist
Failed to load module ROOTBrowsable
Failed to load module Unfold
Failed to load module RHTTPSniff
Failed to load module HistPainter
Failed to load module PyMVA
Failed to load module RHTTP
Failed to load module FitPanel
Failed to load module ProofDraw
Failed to load module Unuran
Failed to load module Quadp
Failed to load module Genetic
Failed to load module Eve
Failed to load module GeomPainter
Failed to load module TreeViewer
Failed to load module Physics
Failed to load module EG
Failed to load module Tree
Failed to load module HistFactory
Failed to load module Spectrum
Failed to load module Matrix
Failed to load module Html
Failed to load module Hist
Failed to load module GuiHtml
Failed to load module Gpad
Failed to load module TMVAGui
Failed to load module Postscript
Failed to load module ROOTEve
Failed to load module RGL
Failed to load module Fumili
Failed to load module Geom
Failed to load module RooFitMore
Failed to load module X3d
Failed to load module SPlot
Failed to load module Hbook
Failed to load module RooFit
Failed to load module RCsg
Failed to load module RooStats
Failed to load module RooFitRDataFrameHelpers
Failed to load module GeomBuilder
Failed to load module Proof
Failed to load module FITSIO
Failed to load module Ged
Failed to load module Recorder
Failed to load module FFTW
Failed to load module GuiBld
Failed to load module ROOTWebDisplay
Failed to load module RooFitCore
Failed to load module Gui
Failed to load module ROOTHistDraw
Failed to load module GX11TTF
Failed to load module ROOTTMVASofie
Failed to load module ProofPlayer
Failed to load module ASImage
Failed to load module MathMore
Failed to load module RooFitHS3
Failed to load module Foam
Failed to load module SpectrumPainter
Failed to load module Minuit2
Failed to load module MLP
Failed to load module ROOTDataFrame
Failed to load module GenVector
Failed to load module ROOTBrowserv7
Failed to load module Minuit
Failed to load module Graf3d
Failed to load module TMVA
Failed to load module ASImageGui
Failed to load module Graf
Failed to load module GX11
Failed to load module Gdml
Failed to load module ProofBench
Failed to load module MathCore
Failed to load module Gviz3d
Failed to load module WebGui6
Failed to load module ROOTHist
Failed to load module TreePlayer
Failed to load module ROOTFitPanelv7
Failed to load module Smatrix
Failed to load module SessionViewer
RunTest: /build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT/HEAD/interpreter/llvm/src/tools/clang/lib/Lex/Lexer.cpp:3940: bool clang::Lexer::LexTokenInternal(clang::Token&, bool): Assertion `Result.is(tok::eof) && "Preprocessor did not set tok:eof"' failed.
Aborted (core dumped)

Even though Vc.pcm is sitting right next to MathCore.pcm export ROOT_INCLUDE_PATH does solve this problem (see the other tests in the repeater), but at least to me this behaviour or at least the error message seems questionable.

vgvassilev commented 2 years ago

The location of the Vc.pcm does not tell ROOT where the include paths of Vc are, in a similar way how the .so file does not tell the compiler where to find its header files. We still need to somehow teach the build system to tell correctly where the Vc include path is.

@Axel-Naumann, it sounds like we need to extend your fix in cmake should we want to continue in that direction.

Axel-Naumann commented 2 years ago

Indeed testing these downstream packages will likely need Vc, and we cannot know where that is, so that's a case for ROOT_INCLUDE_PATH.

The second part of this issue is about the error message. @vgvassilev do you agree that

<<< cling interactive line includer >>>: fatal error: module file '/build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/lib/Vc.pcm' not found: module file not found

isn't equivalent to "ROOT cannot find Vc/Vc.h; please provide the include path by adding it to ROOT_INCLUDE_PATH"?

Axel-Naumann commented 2 years ago

@vgvassilev anything we can do to improve the error message, see https://github.com/root-project/root/issues/9594#issuecomment-1031255059 ?

vgvassilev commented 2 years ago

@Axel-Naumann, not easily. This crash happens when clang tries to load the dependencies of MathCore, so no easy way to diagnose. Maybe if we iterate over all modules, read their dependencies and respect the loading of the dependencies in that order, we might be able to diagnose this. However, to go through the dependencies of a given PCM without loading it requires implementing some binary reading. I've seen such code, and it is doable but..

hahnjo commented 11 months ago

Based on the discussion, it's not clear to me if there's anything left to be done. Can we close this? @vgvassilev

dpiparo commented 9 months ago

Let me re-iterate @vgvassilev , @andresailer : can we perhaps close?

vgvassilev commented 9 months ago

Let's close it.

github-actions[bot] commented 9 months ago

Hi @vgvassilev, @Axel-Naumann,

It appears this issue is closed, but wasn't yet added to a project. Please add upcoming versions that will include the fix, or 'not applicable' otherwise.

Sincerely, :robot:

andresailer commented 9 months ago

My reproducer does't fail with ROOT HEAD any more, so :shrug:

github-actions[bot] commented 9 months ago

Hi @vgvassilev, @Axel-Naumann,

It appears this issue is closed, but wasn't yet added to a project. Please add upcoming versions that will include the fix, or 'not applicable' otherwise.

Sincerely, :robot:

hahnjo commented 9 months ago

@vgvassilev what you need here, as the bot describes it, is a project, not a milestone

vgvassilev commented 9 months ago

I haven't seen more useless thing recently...

dpiparo commented 9 months ago

It's a little price to pay in order to collect all fixed issues for a certain release - a very useful feature for our users!

vgvassilev commented 9 months ago

So once we mention what's being fixed in the PR and once more elsewhere? In principle that information is rather easy to pull out without having people to specify these things. It is great to have automation to the extent it is useful. The problem with the ROOT repository is that it has a lot of these things with dubious usefulness for the day-to-day work. I am happy to elaborate move if necessary...