Spack build on Summit

[diehlpk]

I followed the instructions here

However, I get following error on Summit

[diehlpk@login4.summit spack]$ spack spec octotiger +cuda +kokkos ^hpx cuda_arch=60 ^kokkos cuda_arch=60 +pascal60 +hsw Input spec

octotiger+cuda+kokkos ^hpx cuda_arch=60 ^kokkos+hsw+pascal60 cuda_arch=60

Concretized

==> Error: trying to set variants "pascal60 or hsw" in package "kokkos", but the package has no such variants [happened during concretization of octotiger+cuda+kokkos ^cmake@3.12.4: ^kokkos@3.0+hpx+hpx_async_dispatch+hsw+pascal60+serial cuda_arch=60 std=14 ^kokkos-hpx-interop+cuda]

or

[diehlpk@login4.summit spack]$ spack spec octotiger +cuda +kokkos ^hpx cuda_arch=60 ^kokkos cuda_arch=70 +volta70 +hsw Input spec

octotiger+cuda+kokkos ^hpx cuda_arch=60 ^kokkos+hsw+volta70 cuda_arch=70

Concretized

==> Error: trying to set variants "volta70 or hsw" in package "kokkos", but the package has no such variants [happened during concretization of octotiger+cuda+kokkos ^cmake@3.12.4: ^kokkos@3.0+hpx+hpx_async_dispatch+hsw+serial+volta70 cuda_arch=70 std=14 ^kokkos-hpx-interop+cuda]

What flags are missing on Summit?

[G-071]

It seems like you are using the wrong Kokkos package! Have you been following the instructions on the octotiger-spack readme? (specifically "After downloading the octotiger-spack GitHub repository, you need to add the kokkos-spack repository according to the instructions on their Github Page."). This should enable you to use the same package we do by default (in which the option volta70 exists).

That being said, I think there is still something missing from your spack command above, even when using the same kokkos package we do. You will want to specify that you are using mpi, you might want to adapt the cuda_arch in the hpx dependency, and (IF you do not use the systems mpi which you would have to tell spack where to find) you need to use an hwloc version < 2.0 to build openmpi.

See whether this spec works for you: spack spec octotiger +cuda +kokkos +mpi ^hpx cuda_arch=70 ^kokkos cuda_arch=70 +volta70 +hsw ^hwloc@:1.999

Furthermore, take care which compiler you use! You can install a different compiler using spack (spack install compiler; spack load compiler; spack compiler find) and add %compiler to the spack command above to use it to build the toolchain. Let me know if you encounter any more issues! For example, a pitfall that cost me some time are too small /tmp directories as they cause the spack build of some packages to fail (solution: redirect tmp to a folder on some larger partition).

For reference, the spec on my developer machine looks something like this:

spack spec octotiger@master target=broadwell +mpi +cuda +kokkos ^hpx cuda_arch=61 ^kokkos cuda_arch=61 +pascal61 ^hwloc@:1.999 
Input spec
--------------------------------
octotiger@master+cuda+kokkos+mpi arch=linux-None-broadwell
    ^hpx cuda_arch=61
    ^hwloc@:1.999
    ^kokkos cuda_arch=61 +pascal61

Concretized
--------------------------------
octotiger@master%gcc@7.5.0 build_type=RelWithDebInfo +cuda griddim=8 +kokkos+mpi+test theta_minimum=0.34 +vc arch=linux-ubuntu18.04-broadwell
    ^boost@1.72.0%gcc@7.5.0+atomic+chrono~clanglibcpp~container~context~coroutine cxxstd=14 +date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math+mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout visibility=hidden +wave arch=linux-ubuntu18.04-broadwell
        ^bzip2@1.0.8%gcc@7.5.0+shared arch=linux-ubuntu18.04-broadwell
            ^diffutils@3.7%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
                ^libiconv@1.16%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
        ^openmpi@3.1.5%gcc@7.5.0~cuda~cxx~cxx_exceptions fabrics=none ~java~legacylaunchers~memchecker~pmi schedulers=none ~sqlite3~thread_multiple+vt arch=linux-ubuntu18.04-broadwell
            ^hwloc@1.11.11%gcc@7.5.0~cairo~cuda~gl+libxml2~netloc~nvml+pci+shared arch=linux-ubuntu18.04-broadwell
                ^libpciaccess@0.13.5%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
                    ^libtool@2.4.6%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
                        ^m4@1.4.18%gcc@7.5.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu18.04-broadwell
                            ^libsigsegv@2.12%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
                    ^pkgconf@1.6.3%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
                    ^util-macros@1.19.1%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
                ^libxml2@2.9.9%gcc@7.5.0~python arch=linux-ubuntu18.04-broadwell
                    ^xz@5.2.5%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
                    ^zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=linux-ubuntu18.04-broadwell
                ^numactl@2.0.12%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
                    ^autoconf@2.69%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
                        ^perl@5.30.1%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-broadwell
                            ^gdbm@1.18.1%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
                                ^readline@8.0%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
                                    ^ncurses@6.2%gcc@7.5.0~symlinks+termlib arch=linux-ubuntu18.04-broadwell
                    ^automake@1.16.2%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
    ^cmake@3.17.0%gcc@7.5.0~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu18.04-broadwell
        ^openssl@1.1.1g%gcc@7.5.0+systemcerts arch=linux-ubuntu18.04-broadwell
    ^cuda@10.0.130%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
    ^hdf5@1.10.6%gcc@7.5.0+cxx~debug~fortran~hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu18.04-broadwell
    ^hpx@1.4.1%gcc@7.5.0 build_type=RelWithDebInfo +cuda cuda_arch=61 cxxstd=14 ~examples instrumentation=none malloc=tcmalloc max_cpu_count=64 networking=tcp patches=3141d2961a4c1057664fd7559ee21fdbb91c22baeb34bf28a6403367ccfc8f4d ~tools arch=linux-ubuntu18.04-broadwell
        ^git@2.26.0%gcc@7.5.0~tcltk arch=linux-ubuntu18.04-broadwell
            ^curl@7.68.0%gcc@7.5.0~darwinssl~gssapi~libssh~libssh2~nghttp2 arch=linux-ubuntu18.04-broadwell
            ^expat@2.2.9%gcc@7.5.0+libbsd arch=linux-ubuntu18.04-broadwell
                ^libbsd@0.10.0%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
            ^gettext@0.20.1%gcc@7.5.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-ubuntu18.04-broadwell
                ^tar@1.32%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
            ^libidn2@2.1.1a%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
                ^libunistring@0.9.10%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
            ^pcre2@10.31%gcc@7.5.0+multibyte arch=linux-ubuntu18.04-broadwell
        ^gperftools@2.7%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
            ^libunwind@1.4.0%gcc@7.5.0~xz~zlib arch=linux-ubuntu18.04-broadwell
        ^python@3.7.7%gcc@7.5.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib arch=linux-ubuntu18.04-broadwell
            ^libffi@3.2.1%gcc@7.5.0 arch=linux-ubuntu18.04-broadwell
            ^sqlite@3.30.1%gcc@7.5.0+column_metadata+fts~functions~rtree arch=linux-ubuntu18.04-broadwell
    ^hpx-kokkos-interopt-wip@master%gcc@7.5.0 build_type=RelWithDebInfo +cuda+mpi arch=linux-ubuntu18.04-broadwell
        ^kokkos@3.0%gcc@7.5.0~aggressive_vectorization~amdavx~armv80~armv81~armv8_thunderx~armv8_tx2~bdw~bgq build_type=RelWithDebInfo ~carrizo~compiler_warnings+cuda cuda_arch=61 +cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~epyc~examples~explicit_instantiation~fiji~gfx901+hpx+hpx_async_dispatch~hsw~hwloc~kaveri~kepler30~kepler32~kepler35~kepler37~knc~knl~maxwell50~maxwell52~maxwell53~memkind~numactl~openmp~pascal60+pascal61 patches=f4c50fee3dca1996e1dc10e65ee4d6c874558aff375d6c9419d13df38472f4e8 ~power7~power8~power9+profiling~profiling_load_print~pthread~qthread~rocm~ryzen+serial~skx~snb std=14 ~tests~turing75~vega~volta70~volta72+wrapper~wsm arch=linux-ubuntu18.04-broadwell
            ^kokkos-nvcc-wrapper@old%gcc@7.5.0 build_type=RelWithDebInfo +mpi patches=6c21eecf2c4b7832e6501b701bd6de840ac94ab01c1941c70f9ae271c34bfc2b,c8ff0bd593ca53512ba0bdee924760bea6b17b7896c0b6cdc2cccaa8149d6680 arch=linux-ubuntu18.04-broadwell
        ^kokkos-hpx-interop@master%gcc@7.5.0 build_type=RelWithDebInfo +cuda arch=linux-ubuntu18.04-broadwell
    ^silo@4.10.2%gcc@7.5.0+fortran+mpi patches=7b5a1dc2a0e358e667088d77e7caa780967fa8ea60be89c44986605df9990abe +pic+shared~silex arch=linux-ubuntu18.04-broadwell
    ^vc@1.4.1%gcc@7.5.0 build_type=RelWithDebInfo arch=linux-ubuntu18.04-broadwell

Depending on your platform, compilers, etc, you will probably need to tweak this spec, though.

Are you building this on a POWER machine? If yes, there should also be some flags to optimize kokkos for power. See spack info kokkos for that! Also you might want to stick to boost 1.70.

[diehlpk]

@G-071 Can you please share the exact setup, you used on your Power9 node? I like to have the exact same build as you have to make testing more reproducible.

I generated a repo to store the file for Summit and your Power node, so we can keep track there

https://github.com/diehlpk/Octo-Tiger-Spack

I just like to edit at most some paths and not too many flags.

I like to have the same version as you used on your system.

Fort now just a single node configuration without MPI. We will take care later on the IMB specturm compiler.