cameronrutherford
commented
1 year ago cc @pelesh
Closed jaelynlitz closed 10 months ago
cameronrutherford
commented
1 year ago cc @pelesh
jaelynlitz
commented
1 year ago Tried reverting to umpire 6.0.0, our last working version, and am getting the same error as above. Tried toggling ~shared with no change in the output.
davidbeckingsale
commented
1 year ago Sorry I missed this. It looks like a system configuration problem. Your build is pulling in standard library headers from GCC 4.8.5: /usr/lib/gcc/x86_64-redhat-linux/4.8.5.
Umpire now requires C++14, and previously C++11, and needs a newer compiler.
You may need to provide a --gcc-toolchain flag to get amdclang++ to use a newer GCC.
jaelynlitz
commented
1 year ago Thanks for the suggestion @davidbeckingsale. I tried adding the --gcc-toolchain flag to our spack.yaml (attached), but it seems as though the flag was not picked up. Maybe my syntax is off.. The spack build output is also attached - it's picking up gcc10.2.0 and hipcc as the compilers, but still finding the 4.8.5 include dir.
jaelynlitz
commented
1 year ago I am trying to compile from source to pin down if it is a problem with Umpire or our compiler and I have attached the full error log from trying to compile with v6.0.0 and clang15.0.0. The --gcc-toolchain flag seems correctly populated, but it is still trying to link against the 4.8.5 library -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5. I get the same error output with v2022.03.1.
Here is the most notable snippet from the output:
[ 35%] Linking CXX executable ../../bin/judy_test
cd /qfs/projects/exasgd/src/jaelyn-spack/Umpire/build/tests/tpl && /qfs/projects/ops/rh7_gpu/rocmapps/views/cmake/3.21.4/bin/cmake -E cmake_link_script CMakeFiles/judy_test.dir/link.txt --verbose=1
/opt/rocm-5.3.0/llvm/bin/clang++ -Wpedantic --gcc-toolchain=/share/apps/gcc/8.4.0 -Wall -Wextra CMakeFiles/judy_test.dir/judy_test.cpp.o ../../src/umpire/tpl/judy/CMakeFiles/umpire_tpl_judy.dir/judy.c.o -o ../../bin/judy_test -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 ../../lib/libgtest_main.a ../../lib/libgtest.a -lpthread -lpthread
ld.lld: error: undefined symbol: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::c_str() const
>>> referenced by judy_test.cpp
>>> CMakeFiles/judy_test.dir/judy_test.cpp.o:(JudyTest_Construction_Test::TestBody())
>>> referenced by judy_test.cpp
>>> CMakeFiles/judy_test.dir/judy_test.cpp.o:(JudyTest_WriteValues_Test::TestBody())
>>> referenced by judy_test.cpp
>>> CMakeFiles/judy_test.dir/judy_test.cpp.o:(JudyTest_WriteValues_Test::TestBody())
>>> referenced 108 more times
...Modifying some of the logic from here:
https://github.com/LLNL/radiuss-spack-configs/blob/main/packages/camp/package.py#L86-L112
may help. You can add the /usr/lib/gcc/x86_64-redhat-linux/4.8.5 directory to BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE, and it should fix it.
pelesh
commented
1 year ago A couple of additional suggestions:
ccmake what compiler/libs the cmake command has found. Just hit t to toggle to advanced mode.cmake command. In your case it would be something like CC=clang CXX=clang++ cmake ... I think CMake will by default use whatever is linked to /usr/bin/cc. davidbeckingsale
commented
1 year ago @jaelynlitz I missed the bit where you mentioned building from source, rather than Spack. You can try passing BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE directly to CMake, e.g.:
cmake <blah blah blah> -DBLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE=/usr/lib/gcc/x86_64-redhat-linux/4.8.5 ..@davidbeckingsale that worked for compiling from source! I'll see what I can do to transfer into spack
jaelynlitz
commented
1 year ago After success compiling from source with gcc10.2.0, I have been trying to build umpire via spack with gcc10.2.0. However, Umpire is still trying to use the clang15.0.0 compiler and still linking against gcc4.8.5. The --gcc-toolchain flags are still given in the spack.yaml for clang and hipcc, and I have added the BLT exclude dirs line to the Umpire spack package:
Under line 149
entries.append(cmake_cache_string("BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE", "/usr/lib/gcc/x86_64-redhat-linux/4.8.2;/usr/lib/gcc/x86_64-redhat-linux/4.8.5;/usr/tce/packages/gcc/gcc-4.9.3/lib64;/usr/tce/packages/gcc/gcc-4.9.3/gnu/lib64/gcc/powerpc64le-unknown-linux-gnu/4.9.3;/usr/tce/packages/gcc/gcc-4.9.3/gnu/lib64;/usr/tce/packages/gcc/gcc-4.9.3/lib64/gcc/x86_64-unknown-linux-gnu/4.9.3"))
Attached spack.yaml and concretization and build output spack.yaml umpire gcc vs clang .txt
cameronrutherford
commented
1 year ago I was able to repro the same issues as @jaelynlitz using latest spack develop and the same box. I verified that the BLT exclude configuration was being passed to umpire, and nothing on the system path should be linked in. I also verified with cpp while in the spack environment that /usr/lib/;/lib were not on the include search path.
This leads me to believe something funky is going on with the spack compiler wrapper and how it has gcc/clang configured. I can posted updated logs/spack configuration but they are largely the same as ones already posted in the issue.
Maybe getting more information from spack would be helpful? I am a little at a loss why building by hand and through spack are giving such different results.
kab163
commented
10 months ago Seeing as we are up to rocm/6.0.0 now and we have the CI working for rocm/5.6.0, can this be closed?
kab163
commented
10 months ago Waiting a few more days.. if no response, I will close this issue.
cameronrutherford
commented
10 months ago This isn't critical path for any project work since ECP came to and end, and so I don't think we are going to get to testing this anytime soon. I assume that this issue can be closed given your reports, and we can re-open any issues that we might have found with ROCm 5.6.0/6.0.0 as necessary
Describe the bug
Hi, I'm a developer on ExaGO within ExaSGD. I'm trying to upgrade our dependency on Umpire from version 6.0.0 to 2022.3.1. I'm building Umpire via Spack within our software stack and am running into some build errors. This is an AMD system and I'm trying to use ROCm 5.3.0.
@CameronRutherford
To Reproduce
Concretize with gcc 10.2.0, rocm 5.3.0 with the options below. Try to
spack installumpire.Expected behavior
Code to compile.
Compilers & Libraries (please complete the following information):
Additional context
Below the cut are the errors from the build log:
Logs
Spack spec: umpire@2022.03.1%gcc@10.2.0+c~cuda+device_alloc~deviceconst~examples~fortran~ipo~numa~openmp+rocm+shared amdgpu_target=gfx90a build_system=cmake build_type=RelWithDebInfo patches=78daca9 tests=none arch=linux-centos7-zen2 ``` ==> Installing umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4 ==> umpire exists in binary cache but with different hash ==> No binary for umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4 found: installing from source ==> Warning: download_tarball() was unable to download umpire@2022.03.1%gcc@10.2.0+c~cuda+device_alloc~deviceconst~examples~fortran~ipo~numa~openmp+rocm+shared amdgpu_target=gfx90a build_system=cmake build_type=RelWithDebInfo patches=78daca9 tests=none arch=linux-centos7-zen2 ^blt@0.5.2%gcc@10.2.0 build_system=generic arch=linux-centos7-zen2 ^ca-certificates-mozilla@2022-10-11%gcc@10.2.0 build_system=generic arch=linux-centos7-zen2 ^camp@2022.03.2%gcc@10.2.0~cuda~ipo~openmp+rocm~tests amdgpu_target=gfx90a build_system=cmake build_type=RelWithDebInfo arch=linux-centos7-zen2 ^cmake@3.20.6%gcc@10.2.0~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-centos7-zen2 ^hip@5.3.0%gcc@10.2.0~ipo build_system=cmake build_type=Release patches=ca523f1 arch=linux-centos7-zen2 ^hsa-rocr-dev@5.3.0%gcc@10.2.0+image~ipo+shared build_system=cmake build_type=Release patches=71e6851 arch=linux-centos7-zen2 ^llvm-amdgpu@5.3.0%gcc@10.2.0~ipo~link_llvm_dylib~llvm_dylib~openmp+rocm-device-libs build_system=cmake build_type=Release patches=a08bbe1 arch=linux-centos7-zen2 ^ncurses@6.3%gcc@10.2.0~symlinks+termlib abi=none build_system=autotools arch=linux-centos7-zen2 ^openssl@1.1.1s%gcc@10.2.0~docs~shared build_system=generic certs=mozilla arch=linux-centos7-zen2 ^perl@5.26.0%gcc@10.2.0+cpanm+shared+threads build_system=generic patches=0eac10e,8cf4302 arch=linux-centos7-zen2 ^pkgconf@1.8.0%gcc@10.2.0 build_system=autotools arch=linux-centos7-zen2 ^zlib@1.2.13%gcc@10.2.0+optimize+pic+shared build_system=makefile arch=linux-centos7-zen2 from any configured mirrors ==> Using cached archive: /qfs/projects/exasgd/src/svcexasgd/cache/source-cache/_source-cache/git//LLNL/Umpire.git/v2022.03.1.tar.gz ==> Applied patch /qfs/people/svcexasgd/gitlab/18959/spack_incline/tpl/spack/var/spack/repos/builtin/packages/umpire/std-filesystem-pr784.patch ==> Warning: Fetching from mirror without a checksum! This package is normally checked out from a version control system, but it has been archived on a spack mirror. This means we cannot know a checksum for the tarball in advance. Be sure that your connection to this mirror is secure! ==> umpire: Executing phase: 'initconfig' ==> umpire: Executing phase: 'cmake' ==> umpire: Executing phase: 'build' ==> Error: ProcessError: Command exited with status 2: 'make' '-j24' 12 errors found in build log: 185 /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir e/strategy/FixedSizePool.hpp:133:9: warning: variable 'i' set but n ot used [-Wunused-but-set-variable] 186 int i = 0; 187 ^ 188 In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn v4/spack-src/src/umpire/interface/c_fortran/wrapResourceManager.cpp :17: 189 In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn v4/spack-src/src/umpire/strategy/QuickPool.hpp:11: 190 In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../ ../../include/c++/4.8.5/map:62: >> 191 /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8. 5/bits/stl_multimap.h:130:41: error: no member named 'rebind' in 'u mpire::strategy::QuickPool::pool_allocator