search

spack / spack

A flexible package manager that supports multiple versions, configurations, platforms, and compilers.
https://spack.io
Other
4.22k stars 2.24k forks source link

chai +rocm build fails: cmake generate failed #20346

Closed eugeneswalker closed 2 years ago

eugeneswalker commented 3 years ago

Building chai@2.1.1 with +rocm fails using spack@develop (0924b209847e2d7284ff0ff2b234f8afd47e0921 from Fri Dec 11 14:16:28 2020 +0000)

System info:

Concrete spec: chai-gq6pzg.yml.txt Build log: chai-build-out.txt

$> spack install -f ./chai-gq6pzg.yml
...
==> Installing chai-2.1.1-7jbtpe2fmolhhskluypmikrz2gtjrzrf
Note: checking out '496911e00d15c350560860f7964cd5fb5ab7f515'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

==> chai: Executing phase: 'cmake'
==> Error: ProcessError: Command exited with status 1:
    'cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/chai-2.1.1-7jbtpe2fmolhhskluypmikrz2gtjrzrf' '-DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=OFF' '-DCMAKE_INSTALL_RPATH:STRING=/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/chai-2.1.1-7jbtpe2fmolhhskluypmikrz2gtjrzrf/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/chai-2.1.1-7jbtpe2fmolhhskluypmikrz2gtjrzrf/lib64;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/hip-3.9.0-p2p2tlyzakyuhudvkyssymeci3hki2hd/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/comgr-3.9.0-egnsiovlhsqqechzsxe7e7sjsr3lun6a/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/ncurses-6.2-crhlefo3dv7lmsv5pf4icsy4gepkdorm/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/z3-4.8.7-sbbydjzlzcixcblt65aknzruxe4ag3ta/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/python-3.8.6-gaiwvmyl5krmuwhf375zxfdncngwbh4p/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/bzip2-1.0.8-fvfpt26p5divvq6344vojyn64enojlui/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/expat-2.2.10-ba7brxjwecr44rs3ftrnlzq4yafddxrc/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libbsd-0.10.0-u6ue7vwosvdt6th4mpyq6tlriclbtglt/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gdbm-1.18.1-4av4gywgpaspkhy3dvbb62nulqogtzbb/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/readline-8.0-t54jzdy2jj4snltjazlm3br2urcilc6v/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gettext-0.21-lbb45tosrs7xp2g6uwwgw4vmn7vay3k7/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libiconv-1.16-jearpk4xci4zc7dkrza4fufaqfkq7rfl/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libxml2-2.9.10-yn2r3wfhiilelyulh5toteicdtxjhw7d/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/xz-5.2.5-komekkmyciga3kl24edjmredhj3uyt7v/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/zlib-1.2.11-smoyzzo2qhzpn6mg6rd3l2p7b23enshg/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libffi-3.3-hyhbnrmaxc5veijr2iyveifkj4azv5dv/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libuuid-1.0.3-wuyj2axfi6vdokf2y6t2cphygas3d4cm/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/openssl-1.1.1i-zlikk6yvr6wef5gakkf3u6nyrcxz24me/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/sqlite-3.33.0-rhv2o7bs2i66b3mnsepnv5aj7uozhy2m/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/hsa-rocr-dev-3.9.0-rjih7j22m5kfuhpo4q7l4fccxqzfbn6o/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/hsakmt-roct-3.9.0-syedeifdiapzmctahgdj6bs4aui7xyvj/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/numactl-2.0.14-wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libelf-0.8.13-aarethqixeghvzzm6zmdfrils5h5zjyh/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/llvm-amdgpu-3.9.0-pwoxtjfnxwh2zkeh4q54i4vmdx2naihh/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/mesa18-18.3.6-h3rtjskjmryquvnrdexh2jjgflvgurbf/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libx11-1.7.0-fjjnkaguxv5ffh5wrkedqjn7prifduzm/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/kbproto-1.0.7-72fsl3jhsdolpeci4domde7yhlsyapfl/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libxcb-1.14-lzhijgmizeqi4qrouaxqb3vpojbfxdyg/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libpthread-stubs-0.4-6nejtpb4ddmxi4oqgaymtjykluvgrum6/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libxau-1.0.8-3wxhlajjgth5jx767pseepbx323lbo6i/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/xproto-7.0.31-whuc5zglp5yuuutvyk37endfvzpmdqrw/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libxdmcp-1.1.2-ofwr2lxqcophrchshg7x246vooz43k7b/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/xextproto-7.3.0-7hfozefc76xjhjvz2gvai6eoivisfszo/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libxext-1.3.3-65jgcqejnxux5owc6vxpku6qox4b6aoz/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/rocm-device-libs-3.9.0-m4zgwzmhfmqyucrdvh7m7knc454opmg4/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/texinfo-6.5-p46ba5qb5wvx6a2bneh4vlurqusz35bj/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/perl-5.32.0-zfdvt2jjuaees43ffrrtphqs2ky3o22t/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/berkeley-db-18.1.40-4ihuiazsglf22f3pntq5hc4kyszqzexn/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/umpire-4.1.2-wpg32ir6gh6vvwftrsrrltexio7lijh2/lib;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/camp-0.1.0-n4ob2he46vwq3za45orffl6sulcl3hcs/lib' '-DCMAKE_PREFIX_PATH:STRING=/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/blt-develop-bxagzdrqrv673ql54budx3nrjmc5hmkc;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/cmake-3.18.4-q6lka2f3tzaizib7p4ot6ijzaustqcgv;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/hip-3.9.0-p2p2tlyzakyuhudvkyssymeci3hki2hd;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/hsa-rocr-dev-3.9.0-rjih7j22m5kfuhpo4q7l4fccxqzfbn6o;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/llvm-amdgpu-3.9.0-pwoxtjfnxwh2zkeh4q54i4vmdx2naihh;/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/umpire-4.1.2-wpg32ir6gh6vvwftrsrrltexio7lijh2' '-DBLT_SOURCE_DIR=/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/blt-develop-bxagzdrqrv673ql54budx3nrjmc5hmkc' '-DENABLE_CUDA=OFF' '-DENABLE_HIP=ON' '-DHIP_ROOT_DIR=/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/hip-3.9.0-p2p2tlyzakyuhudvkyssymeci3hki2hd' '-DHIP_HIPCC_FLAGS=--amdgpu-target=gfx906' '-DENABLE_PICK=OFF' '-Dumpire_DIR:PATH=/opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/umpire-4.1.2-wpg32ir6gh6vvwftrsrrltexio7lijh2/share/umpire/cmake' '-DENABLE_TESTS=OFF' '-DENABLE_BENCHMARKS=OFF' '-DENABLE_EXAMPLES=ON' '/tmp/root/spack-stage/spack-stage-chai-2.1.1-7jbtpe2fmolhhskluypmikrz2gtjrzrf/spack-src'

2 errors found in build log:
     49    -- Enabling all compiler warnings on all targets.
     50    -- Fortran support disabled.
     51    -- CMAKE_C_FLAGS flags are:    -Wall -Wextra
     52    -- CMAKE_CXX_FLAGS flags are:       -Wall -Wextra
     53    -- CMAKE_EXE_LINKER_FLAGS flags are:
     54    -- Configuring done
  >> 55    CMake Error at /opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/blt-develop-bxagzdrqrv673ql54budx3nrjmc5hmkc/cmake/BLTPrivateMacros.cmake:526 (add
           _library):
     56      Cannot find source file:
     57
     58        HEADERS_OUTPUT_SUBDIR
     59
     60      Tried extensions .c .C .c++ .cc .cpp .cxx .cu .m .M .mm .h .hh .h++ .hm
     61      .hpp .hxx .in .txx
     62    Call Stack (most recent call first):
     63      /opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/blt-develop-bxagzdrqrv673ql54budx3nrjmc5hmkc/cmake/BLTMacros.cmake:643 (blt_add_hip_library)
     64      src/chai/CMakeLists.txt:71 (blt_add_library)
     65
     66
  >> 67    CMake Error at /opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/blt-develop-bxagzdrqrv673ql54budx3nrjmc5hmkc/cmake/BLTPrivateMacros.cmake:526 (add
           _library):
     68      No SOURCES given to target: chai
     69    Call Stack (most recent call first):
     70      /opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/blt-develop-bxagzdrqrv673ql54budx3nrjmc5hmkc/cmake/BLTMacros.cmake:643 (blt_add_hip_library)
     71      src/chai/CMakeLists.txt:71 (blt_add_library)
     72
     73

See build log for details:
  /tmp/root/spack-stage/spack-stage-chai-2.1.1-7jbtpe2fmolhhskluypmikrz2gtjrzrf/spack-build-out.txt

@dtaller @haampie @corbett5

dtaller commented 3 years ago

A few things to start.

  1. There have been some fixes into chai for rocm builds, so I would try chai@develop instead, spack install -v chai@develop%clang@amd+rocm~benchmarks amdgpu_target=gfx900

  2. Also, for +rocm builds, I would use the clang packaged with rocm instead of gcc. I have had trouble using different compilers aside from what rocm explicitly provides. Edit your compilers.yaml to include:

and build as spack install -v chai@develop%clang@amd+rocm~benchmarks amdgpu_target=gfx900

  1. If 1 and 2 don't help then look in blt. According to you build log, you get a failure at CMake Error at /opt/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/blt-develop-bxagzdrqrv673ql54budx3nrjmc5hmkc/cmake/BLTPrivateMacros.cmake:526 . You can go there and add a few message statements as follows:

    if (${_depends_on_hip})

    if hip is in depends_on, flag each file's language as HIP

    # instead of leaving it up to CMake to decide
# Note: we don't do this when depending on just 'hip_runtime'
set(_hip_sources)
set(_non_hip_sources)
message(STATUS "-- YES DEPENDS ON HIP --")
blt_split_source_list_by_language(SOURCES      ${arg_SOURCES}
                                  C_LIST       _hip_sources
                                  Fortran_LIST _non_hip_sources)

set_source_files_properties( ${_hip_sources}
                             PROPERTIES
                             HIP_SOURCE_PROPERTY_FORMAT TRUE)

hip_add_library( ${arg_NAME} ${arg_SOURCES} ${arg_LIBRARY_TYPE} )

    else() message(STATUS "-- NOT DEPENDS ON HIP --") message(STATUS "Variables: ${arg_NAME}, ${arg_LIBRARY_TYPE}, ${arg_SOURCES}, ${arg_HEADERS}") add_library( ${arg_NAME} ${arg_LIBRARY_TYPE} ${arg_SOURCES} ${arg_HEADERS} ) endif()

When I do this, the following is printed out (see below). Do you get the same? This can be a reasonable starting point to see why things seem to work for me on the corona machine but not on your machine.

-- -- NOT DEPENDS ON HIP -- -- Variables: chai, STATIC, ArrayManager.cpp, ArrayManager.hpp;ArrayManager.inl;ChaiMacros.hpp;ExecutionSpaces.hpp;ManagedArray.hpp;ManagedArray.inl;managed_ptr.hpp;PointerRecord.hpp;Types.hpp