search

root-project / root

The official repository for ROOT: analyzing, storing and visualizing big data, scientifically
https://root.cern
Other
2.7k stars 1.28k forks source link

Unable to build master with external XROOTD #12631

Closed jmcarcell closed 9 months ago

jmcarcell commented 1 year ago

Describe the bug

When trying to build with an external XRootD, cmake says it finds it but then it says it doesn't and this is making a spack build fail. Relevant lines in the cmake output:

-- Looking for XROOTD
-- Found Xrootd version num: 5.5.4.post256 (setting -DROOTXRDVERS=5.5.4.po)
-- Setting OLDPACK TRUE
-- XROOTD not found, enabling 'builtin_xrootd' option
-- Downloading and building XROOTD version 5.5.2

When building with XROOTD from spack (where I found this issue) the error is more verbose but I think is the same issue:

     121    -- Looking for XROOTD
     122    -- Found Xrootd version num: 5.5.3 (setting -DROOTXRDV
            ERS=500050003)
     123    --              libXrdMain not found: xproofd will be 
            a wrapper around xrootd
     124    --              libXrdClient not found: use built-in
  >> 125    CMake Error at cmake/modules/SearchInstalledSoftware.c
            make:992 (message):
     126      XROOTD not found.  Set environment variable XRDSYS t
            o point to your XROOTD
     127      installation, or include the installation of XROOTD 
            in the
     128      CMAKE_PREFIX_PATH.  Alternatively, you can also enab
            le the option
     129      'builtin_xrootd' to build XROOTD internally

To Reproduce

# To install xrootd
git clone https://github.com/xrootd/xrootd
cd xrootd
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=../install
# To install ROOT
git clone https://github.com/root-project/root
cd root
mkdir builddir
cd builddir
cmake .. -DCMAKE_PREFIX_PATH=/path/to/xrootd/install  -Dbuiltin_xrootd=OFF -Dxrootd=ON -DCMAKE_VERBOSE_MAKEFILE=ON

Setup

cmake 3.26.2 OS: Arch Linux

Additional context

I'm posting it here since I tried different versions of XRootD and all failed, maybe it's an issue with XROOTD since it's complaining about not finding libXrdMain nor libXrdClient

guitargeek commented 1 year ago

Hi @jmcarcell, thanks for opening the issue! I have assigned @bellenot for now, since this issue is related to the build system.

bellenot commented 1 year ago

Hi @jmcarcell I managed to build XRootD 5.5.4 and ROOT master using the following XRootD tar file: https://github.com/xrootd/xrootd/archive/refs/tags/v5.5.4.tar.gz, but not when checking it out from Github. Can you try and let us know? And if it works, please ask the XRootD dev team if it's an expected behaviour

jmcarcell commented 1 year ago

Hmm I get similar errors with both using the following flags: -Dbuiltin_xrootd=OFF -Dxrootd=ON -Dfail-on-missing=ON and having XRDSYS pointing to the XRootD installation (or using CMAKE_PREFIX_PATH). Which flags did you use for building XRootD and ROOT?

bellenot commented 1 year ago

I copied the recipe in your To Reproduce section:

ubuntu@root-cmake-devel:~/git/root/builddir$ cmake -DCMAKE_PREFIX_PATH=~/temp/install -Dminimal=ON  -Dbuiltin_xrootd=OFF -Dxrootd=ON -Dfail-on-missing=ON -Dbuiltin_nlohmannjson=ON -Dbuiltin_lzma=ON -Dbuiltin_xxhash=ON -Dbuiltin_zstd=ON -Dbuiltin_lz4=ON -Dbuiltin_afterimage=ON -Dbuiltin_gl2ps=ON  -DCMAKE_VERBOSE_MAKEFILE=ON ../
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.25.1")
-- Detected ROOT_VERSION 6.29.01
-- Checking internet connectivity
-- Checking internet connectivity - found
-- Looking for Python
-- Found Python3: /usr/bin/python3.8 (found version "3.8.10") found components: Interpreter Development NumPy
-- Could NOT find Python2 (missing: Python2_NumPy_INCLUDE_DIRS NumPy) (found version "2.7.18")
-- Found GCC. Major version 9, minor version 4
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Performing Test GLIBCXX_USE_CXX11_ABI
-- Performing Test GLIBCXX_USE_CXX11_ABI - Success
-- ROOT Platform: linux
-- ROOT Compiler: GNU 9.4.0
-- ROOT Processor: x86_64
-- ROOT Architecture: linuxx8664gcc
-- Build Type: 'Release' (flags = '-O3 -DNDEBUG')
-- Compiler Flags:  -std=c++14 -Wno-implicit-fallthrough -Wno-noexcept-type -pipe  -Wshadow -Wall -W -Woverloaded-virtual -fsigned-char -pthread -O3 -DNDEBUG
-- ROOT default compression algorithm: zlib
-- PyROOT will be built for versions 3.8.10 (Main) and 2.7.18
-- Looking for ZLib
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
-- Looking for Freetype
-- Found Freetype: /usr/lib/x86_64-linux-gnu/libfreetype.so (found version "2.10.1")
-- Looking for PCRE
-- Found PCRE: /usr/include (found version "8.39")
-- Building LZMA version 5.2.4 included in ROOT itself
-- Performing Test CXX_FLAG_STD_CPLUSPLUS11
-- Performing Test CXX_FLAG_STD_CPLUSPLUS11 - Success
-- Performing Test C_FLAG_STD_C99
-- Performing Test C_FLAG_STD_C99 - Success
-- Performing Test C_FLAG_WALL
-- Performing Test C_FLAG_WALL - Success
-- Performing Test CXX_FLAG_WALL
-- Performing Test CXX_FLAG_WALL - Success
-- Performing Test C_FLAG_WEXTRA
-- Performing Test C_FLAG_WEXTRA - Success
-- Performing Test CXX_FLAG_WEXTRA
-- Performing Test CXX_FLAG_WEXTRA - Success
-- Performing Test C_FLAG_WUNDEF
-- Performing Test C_FLAG_WUNDEF - Success
-- Performing Test CXX_FLAG_WUNDEF
-- Performing Test CXX_FLAG_WUNDEF - Success
-- Performing Test C_FLAG_WSHADOW
-- Performing Test C_FLAG_WSHADOW - Success
-- Performing Test CXX_FLAG_WSHADOW
-- Performing Test CXX_FLAG_WSHADOW - Success
-- Performing Test C_FLAG_WCAST_ALIGN
-- Performing Test C_FLAG_WCAST_ALIGN - Success
-- Performing Test CXX_FLAG_WCAST_ALIGN
-- Performing Test CXX_FLAG_WCAST_ALIGN - Success
-- Performing Test C_FLAG_WCAST_QUAL
-- Performing Test C_FLAG_WCAST_QUAL - Success
-- Performing Test CXX_FLAG_WCAST_QUAL
-- Performing Test CXX_FLAG_WCAST_QUAL - Success
-- Performing Test C_FLAG_WSTRICT_PROTOTYPES
-- Performing Test C_FLAG_WSTRICT_PROTOTYPES - Success
-- Building AfterImage library included in ROOT itself
-- Looking for XROOTD
-- Found Xrootd version num: 5.5.4 (setting -DROOTXRDVERS=500050004)
--              libXrdMain not found: xproofd will be a wrapper around xrootd
--              libXrdClient not found: use built-in
--              include_dirs: /home/ubuntu/temp/install/include/xrootd;/home/ubuntu/git/root/proof/xrdinc
--              libraries: /home/ubuntu/temp/install/lib/libXrdUtils.so;/home/ubuntu/temp/install/lib/libXrdCl.so
-- Performing Test ROOT_HAVE_CXX_ATOMICS_WITHOUT_LIB
-- Performing Test ROOT_HAVE_CXX_ATOMICS_WITHOUT_LIB - Success
-- Building LLVM in 'Release' mode.
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Found LibXml2: /usr/lib/x86_64-linux-gnu/libxml2.so (found version "2.9.10")
-- Looking for backtrace
-- Looking for backtrace - found
-- backtrace facility detected in default set of libraries
-- Found Backtrace: /usr/include
-- Native target architecture is X86
-- Threads disabled.
-- Doxygen disabled.
-- Go bindings disabled.
-- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH)
-- OCaml bindings disabled.
-- Found Python module pygments
-- Found Python module pygments.lexers.c_cpp
-- Found Python module yaml
-- LLVM host triple: x86_64-unknown-linux-gnu
-- LLVM default target triple: x86_64-unknown-linux-gnu
-- Building with -fPIC
-- Found Python3: /usr/bin/python3.8 (found suitable version "3.8.10", minimum required is "3.0") found components: Interpreter
-- Linker detection: GNU ld
-- Targeting X86
-- Targeting NVPTX
-- Clang version: 13.0.0
-- Not building amdgpu-arch: hsa-runtime64 not found
-- Cling version (from VERSION file): ROOT_1.0~dev
-- Cling will look for C++ headers in '/usr/include/c++/9:/usr/include/x86_64-linux-gnu/c++/9:/usr/include/c++/9/backward' at runtime.
-- And then fallback to: 'x86_64-linux-gnu-g++-9'
-- Performing Test CXX_HAS_fno_rtti
-- Performing Test CXX_HAS_fno_rtti - Success
-- Performing Test C_HAS_Wno_strict_overflow
-- Performing Test C_HAS_Wno_strict_overflow - Success
-- Performing Test C_HAS_Wno_maybe_uninitialized
-- Performing Test C_HAS_Wno_maybe_uninitialized - Success
-- Performing Test C_HAS_Wno_parentheses_equality
-- Performing Test C_HAS_Wno_parentheses_equality - Success
-- Looking for m
-- Looking for m - not found
-- Performing Test GCC_HAS_BIDI_CHARS_FLAG
-- Performing Test GCC_HAS_BIDI_CHARS_FLAG - Failed
-- Performing Test found_setresuid
-- Performing Test found_setresuid - Success
-- Performing Test found_stdstringview
-- Performing Test found_stdstringview - Failed
-- Performing Test found_stdexpstringview
-- Performing Test found_stdexpstringview - Success
-- Performing Test found_stod_stringview
-- Performing Test found_stod_stringview - Failed
-- Performing Test found_opplusequal_stringview
-- Performing Test found_opplusequal_stringview - Failed
-- Performing Test found_stdapply
-- Performing Test found_stdapply - Failed
-- Performing Test found_stdinvoke
-- Performing Test found_stdinvoke - Failed
-- Performing Test found_stdindexsequence
-- Performing Test found_stdindexsequence - Success
-- Performing Test found_attribute_always_inline
-- Performing Test found_attribute_always_inline - Success
-- Performing Test has_found_attribute_noinline
-- Performing Test has_found_attribute_noinline - Success
-- Performing Test found_hardware_interference_size
-- Performing Test found_hardware_interference_size - Failed
Running /home/ubuntu/git/root/build/unix/compiledata.sh
Making /home/ubuntu/git/root/builddir/ginclude/compiledata.h
-- ROOT Configuration

System          Linux-5.4.0-139-generic
Processor       16 core Intel Core Processor (Broadwell, IBRS) (x86_64)
Build type      Release
Install path    /usr/local
Compiler        GNU 9.4.0
Compiler flags:
C                -Wno-implicit-fallthrough -pipe -Wall -W -pthread -O3 -DNDEBUG
C++              -std=c++14 -Wno-implicit-fallthrough -Wno-noexcept-type -pipe  -Wshadow -Wall -W -Woverloaded-virtual -fsigned-char -pthread -O3 -DNDEBUG
Linker flags:
Executable       -rdynamic
Module
Shared           -Wl,--no-undefined -Wl,--hash-style="both"

-- Enabled support for:  builtin_afterimage builtin_clang builtin_cling builtin_gl2ps builtin_llvm builtin_lz4 builtin_lzma builtin_nlohmannjson builtin_xxhash builtin_zstd shared xrootd
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ubuntu/git/root/builddir

and after make -j16:

[...]
[100%] Generating tutorials/hsimple.root
cd /home/ubuntu/git/root/builddir/tutorials && LD_LIBRARY_PATH=/home/ubuntu/git/root/builddir/lib: ROOTIGNOREPREFIX=1 ROOT_HIST=0 /home/ubuntu/git/root/builddir/bin/root.exe -l -q -b -n -x hsimple.C -e return

Processing hsimple.C...
hsimple   : Real Time =   0.08 seconds Cpu Time =   0.09 seconds
(TFile *) 0x561956cbacf0
make[2]: Leaving directory '/home/ubuntu/git/root/builddir'
[100%] Built target hsimple
make[1]: Leaving directory '/home/ubuntu/git/root/builddir'
/usr/bin/cmake -E cmake_progress_start /home/ubuntu/git/root/builddir/CMakeFiles 0
andresailer commented 1 year ago

The related PR broke the LCG builds, probably because we pick up

https://github.com/root-project/root/blob/master/cmake/modules/FindXROOTD.cmake

cf. https://lcgapp-services.cern.ch/cdash/viewFiles.php?buildid=270695

andresailer commented 1 year ago

PS : the configure log just doesn't find xrootd, or at least doesn't mention it does, unlike with the previous builds. Which is also weird because we have fail-on-missing enabled, so it should have failed at configure time.

amadio commented 1 year ago

Note that XrdClient is the old client from XRootD 4.x, which is already out of support (and EOS 4.x will also go out of support soon). XrdClient was removed from the XRootD repository in the XRootD 5.0 release. I think that this is a good time to review the code, drop obsolete things, require at least XRootD 5.x, use XRootDConfig.cmake shipped by XRootD, and drop FindXROOTD.cmake from ROOT. If XRootDConfig.cmake doesn't have something ROOT needs, please let me know and I will fix it for the next patch or feature release, depending on what it is.

guitargeek commented 1 year ago

Thank you so much for commenting, @amadio! I Implemented half of what you suggested in this PR: https://github.com/root-project/root/pull/13752

Feel free to take a look and comment, if you have any suggestions!

guitargeek commented 11 months ago

Re-opened because the XRootDConfig.cmake from xrootd upstream didn't work after all, see: https://github.com/root-project/root/pull/14170

hahnjo commented 11 months ago

Yes, it appears XRootDConfig.cmake is broken in the latest version 5.6.3; I opened https://github.com/xrootd/xrootd/pull/2142 to fix it, then we need a new release and an update in EPEL. Or maybe we can apply the fix on top of 5.6.3 already? @ellert

amadio commented 11 months ago

A new release will come this week (5.6.4), I'm already working on it. I think if you can wait a few days it's better.

ellert commented 11 months ago

Yes, it appears XRootDConfig.cmake is broken in the latest version 5.6.3; I opened xrootd/xrootd#2142 to fix it, then we need a new release and an update in EPEL. Or maybe we can apply the fix on top of 5.6.3 already? @ellert

I have created an update for EPEL (xrootd-5.6.3-3) with this fix and a few others. https://bodhi.fedoraproject.org/updates/?packages=xrootd It is currently submitted to EPEL testing. If karma is provided by those who can, it will be pushed to stable faster. Without karma the default time in testing is 7 dats.