search

G-071 / octotiger-spack

2 stars 4 forks source link

Add sve length variant and allow more cxxstd values #4

Closed G-071 closed 5 months ago

G-071 commented 9 months ago

This PR adds the sve_length variant needed to make use of Octo-Tiger PR STEllAR-GROUP/octotiger#484. While I was at it, I also changed the cxx20 variant into a cxxstd variant as mentioned in #3 .

@junghans Could you try this PR with spack dev-build --until build --fresh octotiger@develop +kokkos simd_library=STD simd_extension=SVE sve_length=256 cxxstd=20 (... plus whatever else you needed in the spec) and with STEllAR-GROUP/octotiger#484 checkout out in the octotiger source directory? Not quite sure if it works as intended on the target machine yet, but at least it seems to work sve_length=512 on the A64Fx machine I tested it on.

junghans commented 9 months ago

I get a build error in asio with cxxstd=20:

In file included from /tmp/junghans/spack-stage/spack-stage-asio-1.21.0-rglwk2e2se4iheqn3wwinm4rg44ekkdg/spack-src/asio/src/examples/cpp17/../../../include/asio/co_spawn.hpp:22,
                 from coroutines_ts/echo_server.cpp:11:
/tmp/junghans/spack-stage/spack-stage-asio-1.21.0-rglwk2e2se4iheqn3wwinm4rg44ekkdg/spack-src/asio/src/examples/cpp17/../../../include/asio/awaitable.hpp: In constructor 'asio::awaitable<T, Executor>::awaitable(asio::awaitable<T, Executor>&&)':
/tmp/junghans/spack-stage/spack-stage-asio-1.21.0-rglwk2e2se4iheqn3wwinm4rg44ekkdg/spack-src/asio/src/examples/cpp17/../../../include/asio/awaitable.hpp:67:19: error: 'exchange' is not a member of 'std'; did you mean 'std::__atomic_impl::exchange'?
   67 |     : frame_(std::exchange(other.frame_, nullptr))
      |                   ^~~~~~~~
G-071 commented 9 months ago

Hmm, what might work here is keeping HPX (and thus asio) at C++17 even though we use 20 for Octo-Tiger. I just reverted the last commit that propagated the cxxflag automatically: it should now be possible to just use cxxstd=20 for Octo-Tiger.

junghans commented 8 months ago 
[  3%] Building CUDA object CMakeFiles/hydrolib.dir/src/unitiger/hydro_impl/hydro_kernel_interface.cpp.o
/vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/kokkos-4.0.01-hswovegdbcq2qg7dbzptx3yp6whqt4pn/bin/kokkos_launch_compiler /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/kokkos-4.0.01-hswovegdbcq2qg7dbzptx3yp6whqt4pn/bin/nvcc_wrapper /vast/home/junghans/octotiger-spack/spack/lib/spack/env/gcc/g++ /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/cuda-12.3.0-waox3sh3wy5zfvo46wvnoqmfluh24fbe/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-11.4.1/gcc-12.3.0-5oi3muh65s6a3efcuqmnyti3lhganawe/bin/g++ -DASIO_DISABLE_CONSTEXPR -DCPPUDDLE_HAVE_HPX -DCPPUDDLE_HAVE_HPX_AWARE_ALLOCATORS -DCPPUDDLE_HAVE_HPX_MUTEX -DCPPUDDLE_HAVE_MAX_NUMBER_GPUS=1 -DCPPUDDLE_HAVE_NUMBER_BUCKETS=128 -DH5_BUILT_AS_DYNAMIC_LIB -DHPX_KOKKOS_CUDA_FUTURE_TYPE=0 -DHPX_KOKKOS_SYCL_FUTURE_TYPE=0 -DHPX_LIBRARY_EXPORTS -DHPX_WITH_CUDA -DOCTOTIGER_GRIDDIM=8 -DOCTOTIGER_HAVE_CUDA -DOCTOTIGER_HAVE_KOKKOS -DOCTOTIGER_HAVE_STD_EXPERIMENTAL_SIMD -DOCTOTIGER_HAVE_VC -DOCTOTIGER_KOKKOS_HYDRO_TASKS=1 -DOCTOTIGER_KOKKOS_MONOPOLE_TASKS=1 -DOCTOTIGER_KOKKOS_MULTIPOLE_TASKS=1 -DOCTOTIGER_KOKKOS_SIMD_SVE -DOCTOTIGER_MAX_NUMBER_FIELDS=15 -DOCTOTIGER_THETA_MINIMUM=0.34 -DSVE_LEN=256 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200809L -Dhydrolib_EXPORTS --options-file CMakeFiles/hydrolib.dir/includes_CUDA.rsp -O3 -DNDEBUG -std=c++20 --generate-code=arch=compute_90,code=[compute_90,sm_90] -Xcompiler=-fPIC -msve-vector-bits=256 -w --extended-lambda --default-stream per-thread --expt-relaxed-constexpr  --ftz=true --prec-div=false --prec-sqrt=false --fmad=false --expt-extended-lambda -expt-relaxed-constexpr -Xptxas=-v -Xcompiler=-march=native,-ffp-contract=off -MD -MT CMakeFiles/hydrolib.dir/src/unitiger/hydro_impl/hydro_kernel_interface.cpp.o -MF CMakeFiles/hydrolib.dir/src/unitiger/hydro_impl/hydro_kernel_interface.cpp.o.d -x cu -c /vast/home/junghans/octotiger-spack/octotiger/src/unitiger/hydro_impl/hydro_kernel_interface.cpp -o CMakeFiles/hydrolib.dir/src/unitiger/hydro_impl/hydro_kernel_interface.cpp.o
/vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/boost-1.74.0-3o6b54jftc65y5bwmi7vcshobciz4syy/include/boost/iostreams/detail/buffer.hpp(43): error: class "std::allocator<char>" has no member "rebind"
      typedef typename Alloc::template rebind<Ch>::other allocator_type;
                                       ^
          detected during:
            instantiation of class "boost::iostreams::detail::basic_buffer<Ch, Alloc> [with Ch=char, Alloc=std::allocator<char>]" at line 52 of /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/boost-1.74.0-3o6b54jftc65y5bwmi7vcshobciz4syy/include/boost/iostreams/detail/double_object.hpp
            instantiation of class "boost::iostreams::detail::single_object_holder<T> [with T=boost::iostreams::detail::basic_buffer<char, std::allocator<char>>]" at line 84 of /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/boost-1.74.0-3o6b54jftc65y5bwmi7vcshobciz4syy/include/boost/iostreams/detail/double_object.hpp
            instantiation of class "boost::iostreams::detail::double_object<T, IsDouble> [with T=boost::iostreams::detail::basic_buffer<char, std::allocator<char>>, IsDouble=boost::is_convertible<boost::iostreams::output, boost::iostreams::detail::two_sequence>]" at line 138 of /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/boost-1.74.0-3o6b54jftc65y5bwmi7vcshobciz4syy/include/boost/iostreams/detail/streambuf/indirect_streambuf.hpp
            instantiation of class "boost::iostreams::detail::indirect_streambuf<T, Tr, Alloc, Mode> [with T=hpx::iostreams::detail::buffer_sink<char>, Tr=std::char_traits<char>, Alloc=std::allocator<char>, Mode=boost::iostreams::output]" at line 65 of /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/boost-1.74.0-3o6b54jftc65y5bwmi7vcshobciz4syy/include/boost/iostreams/stream_buffer.hpp
            instantiation of class "boost::iostreams::stream_buffer<T, Tr, Alloc, Mode> [with T=hpx::iostreams::detail::buffer_sink<char>, Tr=std::char_traits<char>, Alloc=std::allocator<char>, Mode=boost::iostreams::output]" at line 127 of /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/boost-1.74.0-3o6b54jftc65y5bwmi7vcshobciz4syy/include/boost/utility/base_from_member.hpp
            instantiation of class "boost::base_from_member<MemberType, UniqueID> [with MemberType=boost::iostreams::stream_buffer<hpx::iostreams::detail::buffer_sink<char>, std::char_traits<char>, std::allocator<char>, boost::iostreams::output>, UniqueID=0]" at line 81 of /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/boost-1.74.0-3o6b54jftc65y5bwmi7vcshobciz4syy/include/boost/iostreams/stream.hpp
            instantiation of class "boost::iostreams::detail::stream_base<Device, Tr, Alloc, Base> [with Device=hpx::iostreams::detail::buffer_sink<char>, Tr=std::char_traits<char>, Alloc=std::allocator<char>, Base=std::basic_ostream<char, std::char_traits<char>>]" at line 128 of /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/boost-1.74.0-3o6b54jftc65y5bwmi7vcshobciz4syy/include/boost/iostreams/stream.hpp
            instantiation of class "boost::iostreams::stream<Device, Tr, Alloc> [with Device=hpx::iostreams::detail::buffer_sink<char>, Tr=std::char_traits<char>, Alloc=std::allocator<char>]" at line 166 of /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/hpx-1.9.1-xtda4cnlwsa2tpcyps4t7ngbh7s2osd7/include/hpx/components/iostreams/ostream.hpp
            instantiation of class "hpx::iostreams::ostream<Char, Sink> [with Char=char, Sink=hpx::iostreams::detail::buffer_sink<char>]" at line 99 of /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/cppuddle-0.3.1-spcyacohuk3qr3sjfr3phdnxqkkqlkwt/include/aggregation_manager.hpp
junghans commented 8 months ago

I think I saw that before, the problem is that boost needs to be cxx=20.

junghans commented 8 months ago

Let me try to bump asio, v1.21.0 seems a bit old.

junghans commented 8 months ago

Ok, asio@1.22.2 works for me.

version("1.22.2", sha256="985fc2d522f32d232d8386d2fa4ac6f2b25a0cad30495bf2e2e7997bce743f0b")
junghans commented 8 months ago

Now it fails in octotiger again:

/vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/boost-1.74.0-3o6b54jftc65y5bwmi7vcshobciz4syy/include/boost/iostreams/detail/buffer.hpp:43:56: error: no class template named 'rebind' in 'class std::allocator<char>'
   43 |     typedef typename Alloc::template rebind<Ch>::other allocator_type;
      |                                                        ^~~~~~~~~~~~~~
In file included from /vast/home/junghans/octotiger-spack/octotiger/octotiger/common_kernel/kokkos_simd.hpp:32,
                 from /vast/home/junghans/octotiger-spack/octotiger/octotiger/unitiger/hydro_impl/flux_kernel_templates.hpp:11,
                 from /vast/home/junghans/octotiger-spack/octotiger/src/unitiger/hydro_impl/hydro_cuda_interface.cpp:21:
/vast/home/junghans/octotiger-spack/octotiger/octotiger/common_kernel/std_simd.hpp:51:28: error: 'sve' has not been declared
   51 | namespace SIMD_NAMESPACE = sve::experimental::parallelism_v2;
      |                            ^~~

with

Input spec
--------------------------------
octotiger@develop%gcc@12.3.0+boost_multiprecision+cuda+kokkos cuda_arch=90 cxxstd=20 simd_extension=SVE simd_library=STD sve_length=256
    ^cuda@12.3.0
    ^hpx+generic_coroutines max_cpu_count=128

Concretized
--------------------------------
octotiger@develop%gcc@12.3.0+boost_multiprecision+cuda~fast_fp_contract~ipo+kokkos~kokkos_hpx_kernels~rocm~sycl build_system=cmake build_type=Release cuda_arch=90 cxxstd=20 generator=make griddim=8 hydro_host_tasks=1 monopole_host_tasks=1 multipole_host_tasks=1 simd_extension=SVE simd_library=STD sve_length=256 theta_minimum=0.34 arch=linux-rhel9-neoverse_n1
    ^boost@1.82.0%gcc@12.3.0+atomic+chrono~clanglibcpp~container+context~contract~coroutine+date_time~debug+exception~fiber+filesystem+graph~graph_parallel~icu+iostreams~json+locale+log+math~mpi+multithreaded~nowide~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded~stacktrace+system~taggedlayout+test+thread+timer~type_erasure~versionedlayout+wave build_system=generic context-impl=fcontext cxxstd=2a patches=a440f96,a7c807f visibility=hidden arch=linux-rhel9-neoverse_n1
        ^bzip2@1.0.8%gcc@12.3.0~debug~pic+shared build_system=generic arch=linux-rhel9-neoverse_n1
            ^diffutils@3.9%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
        ^xz@5.4.1%gcc@12.3.0~pic build_system=autotools libs=shared,static arch=linux-rhel9-neoverse_n1
        ^zlib@1.2.13%gcc@12.3.0+optimize+pic+shared build_system=makefile arch=linux-rhel9-neoverse_n1
        ^zstd@1.5.5%gcc@12.3.0+programs build_system=makefile compression=none libs=shared,static arch=linux-rhel9-neoverse_n1
    ^cmake@3.26.3%gcc@12.3.0~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-rhel9-neoverse_n1
        ^ncurses@6.4%gcc@12.3.0~symlinks+termlib abi=none build_system=autotools arch=linux-rhel9-neoverse_n1
        ^openssl@1.1.1t%gcc@12.3.0~docs~shared build_system=generic certs=mozilla arch=linux-rhel9-neoverse_n1
            ^ca-certificates-mozilla@2023-01-10%gcc@12.3.0 build_system=generic arch=linux-rhel9-neoverse_n1
            ^perl@5.36.0%gcc@12.3.0+cpanm+open+shared+threads build_system=generic arch=linux-rhel9-neoverse_n1
                ^berkeley-db@18.1.40%gcc@12.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rhel9-neoverse_n1
    ^cppuddle@0.3.1%gcc@12.3.0~allocator_counters+buffer_content_recycling+buffer_recycling~enable_gpu_tests+executor_recycling+hpx~ipo build_system=cmake build_type=Release generator=make max_number_gpus=1 number_buffer_buckets=128 arch=linux-rhel9-neoverse_n1
    ^cuda@12.3.0%gcc@12.3.0~allow-unsupported-compilers~dev build_system=generic arch=linux-rhel9-neoverse_n1
        ^libxml2@2.10.3%gcc@12.3.0~python build_system=autotools arch=linux-rhel9-neoverse_n1
            ^libiconv@1.17%gcc@12.3.0 build_system=autotools libs=shared,static arch=linux-rhel9-neoverse_n1
    ^gmake@4.4.1%gcc@12.3.0~guile build_system=autotools arch=linux-rhel9-neoverse_n1
        ^gnuconfig@2022-09-17%gcc@12.3.0 build_system=generic arch=linux-rhel9-neoverse_n1
    ^hdf5@1.14.1-2%gcc@12.3.0~cxx~fortran+hl~ipo~java~map~mpi+shared+szip+threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-rhel9-neoverse_n1
        ^libaec@1.0.6%gcc@12.3.0~ipo+shared build_system=cmake build_type=Release generator=make arch=linux-rhel9-neoverse_n1
        ^pkgconf@1.9.5%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
    ^hpx@1.9.1%gcc@12.3.0+async_cuda+async_gpu_futures~async_mpi+cuda~examples+generic_coroutines~ipo~lci_pp_log~lci_pp_pcounter~rocm~sycl~tools build_system=cmake build_type=Release cuda_arch=90 cxxstd=20 generator=ninja instrumentation=none malloc=tcmalloc max_cpu_count=128 networking=tcp sycl_target_arch=none arch=linux-rhel9-neoverse_n1
        ^asio@1.22.2%gcc@12.3.0~boost_coroutine~boost_regex~separate_compilation build_system=autotools cxxstd=2a arch=linux-rhel9-neoverse_n1
        ^git@2.40.0%gcc@12.3.0+man+nls+perl+subtree~svn~tcltk build_system=autotools arch=linux-rhel9-neoverse_n1
            ^curl@8.0.1%gcc@12.3.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2~nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rhel9-neoverse_n1
            ^expat@2.5.0%gcc@12.3.0+libbsd build_system=autotools arch=linux-rhel9-neoverse_n1
                ^libbsd@0.11.7%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
                    ^libmd@1.0.4%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
            ^gettext@0.21.1%gcc@12.3.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-rhel9-neoverse_n1
                ^tar@1.34%gcc@12.3.0 build_system=autotools zip=pigz arch=linux-rhel9-neoverse_n1
                    ^pigz@2.7%gcc@12.3.0 build_system=makefile arch=linux-rhel9-neoverse_n1
            ^libidn2@2.3.4%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
                ^libunistring@1.1%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
            ^openssh@9.3p1%gcc@12.3.0+gssapi build_system=autotools arch=linux-rhel9-neoverse_n1
                ^krb5@1.20.1%gcc@12.3.0+shared build_system=autotools arch=linux-rhel9-neoverse_n1
                    ^bison@3.8.2%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
                ^libedit@3.1-20210216%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
            ^pcre2@10.42%gcc@12.3.0~jit+multibyte build_system=autotools arch=linux-rhel9-neoverse_n1
        ^gperftools@2.10%gcc@12.3.0+debugalloc~dynamic_sized_delete_support+libunwind~sized_delete build_system=autotools arch=linux-rhel9-neoverse_n1
            ^libunwind@1.6.2%gcc@12.3.0~block_signals~conservative_checks~cxx_exceptions~debug~debug_frame+docs~pic+tests+weak_backtrace~xz~zlib build_system=autotools components=none libs=shared,static arch=linux-rhel9-neoverse_n1
        ^hwloc@2.9.1%gcc@12.3.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-rhel9-neoverse_n1
            ^libpciaccess@0.17%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
                ^util-macros@1.19.3%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
        ^ninja@1.11.1%gcc@12.3.0+re2c build_system=generic arch=linux-rhel9-neoverse_n1
            ^re2c@2.2%gcc@12.3.0 build_system=generic arch=linux-rhel9-neoverse_n1
        ^python@3.10.10%gcc@12.3.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,f2fd060 arch=linux-rhel9-neoverse_n1
            ^gdbm@1.23%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
            ^libffi@3.4.4%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
            ^libxcrypt@4.4.33%gcc@12.3.0~obsolete_api build_system=autotools arch=linux-rhel9-neoverse_n1
            ^sqlite@3.40.1%gcc@12.3.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-rhel9-neoverse_n1
            ^util-linux-uuid@2.38.1%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
    ^hpx-kokkos@0.4.0%gcc@12.3.0+cuda~ipo~rocm~sycl build_system=cmake build_type=Release cuda_arch=90 cxxstd=20 future_type=polling generator=make arch=linux-rhel9-neoverse_n1
    ^kokkos@4.0.01%gcc@12.3.0+aggressive_vectorization~compiler_warnings+cuda+cuda_constexpr+cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples+hpx+hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning+wrapper build_system=cmake build_type=Release cuda_arch=90 generator=make intel_gpu_arch=none patches=b26a011,b475d96 std=20 arch=linux-rhel9-neoverse_n1
        ^kokkos-nvcc-wrapper@4.0.01%gcc@12.3.0 build_system=generic patches=b475d96 arch=linux-rhel9-neoverse_n1
    ^silo@4.11%gcc@12.3.0+fortran+fpzip+hdf5+hzip+mpi+pic+shared~silex build_system=autotools patches=451c4c5,a081263,eb2a3a0,fa050e0 arch=linux-rhel9-neoverse_n1
        ^autoconf@2.69%gcc@12.3.0 build_system=autotools patches=35c4492,7793209,a49dd5b arch=linux-rhel9-neoverse_n1
        ^autoconf-archive@2023.02.20%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
        ^automake@1.16.5%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
        ^libtool@2.4.7%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
        ^m4@1.4.19%gcc@12.3.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-rhel9-neoverse_n1
            ^libsigsegv@2.14%gcc@12.3.0 build_system=autotools arch=linux-rhel9-neoverse_n1
        ^openmpi@4.1.5%gcc@12.3.0~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~orterunprefix+romio+rsh~singularity+static+vt+wrapper-rpath build_system=autotools fabrics=none schedulers=none arch=linux-rhel9-neoverse_n1
            ^numactl@2.0.14%gcc@12.3.0 build_system=autotools patches=4e1d78c,62fc8a8,ff37630 arch=linux-rhel9-neoverse_n1
            ^pmix@4.2.3%gcc@12.3.0~docs+pmi_backwards_compatibility~python~restful build_system=autotools arch=linux-rhel9-neoverse_n1
                ^libevent@2.1.12%gcc@12.3.0+openssl build_system=autotools arch=linux-rhel9-neoverse_n1
        ^readline@8.2%gcc@12.3.0 build_system=autotools patches=bbf97f1 arch=linux-rhel9-neoverse_n1
    ^vc@1.4.1%gcc@12.3.0~ipo build_system=cmake build_type=Release generator=make arch=linux-rhel9-neoverse_n1

I am bit confused why only boost-1.74.0 is pulled in....

junghans commented 8 months ago

Ok, I found that, it picks up boost-1.82 after I did a git clean to remote the old spack-build.

But the error persists:

/vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/boost-1.82.0-3vpmlzz4ixs55jc46n62uqvs7mwzkpqr/include/boost/iostreams/detail/buffer.hpp:43:56: error: no class template named 'rebind' in 'class std::allocator<char>'
   43 |     typedef typename Alloc::template rebind<Ch>::other allocator_type;
      |                                                        ^~~~~~~~~~~~~~
In file included from /vast/home/junghans/octotiger-spack/octotiger/octotiger/common_kernel/kokkos_simd.hpp:32,
                 from /vast/home/junghans/octotiger-spack/octotiger/octotiger/unitiger/hydro_impl/flux_kernel_templates.hpp:11,
                 from /vast/home/junghans/octotiger-spack/octotiger/src/unitiger/hydro_impl/hydro_cuda_interface.cpp:21:
/vast/home/junghans/octotiger-spack/octotiger/octotiger/common_kernel/std_simd.hpp:51:28: error: 'sve' has not been declared
   51 | namespace SIMD_NAMESPACE = sve::experimental::parallelism_v2;
      |                            ^~~
junghans commented 8 months ago

The problem persists even with boost-1.84.0

G-071 commented 8 months ago

Is there any more information in the build log on the first error (rebind)? Is it still coming from aggregation_manager.hpp:98 / hpx::cout? If that's the case, I might be able to simply remove that part of the header for SVE builds without us losing any significant features.

As for the second error: That one is weird! The compiler appears to be able to include the SVE header without any issues, but can't find the SVE namespace defined in it. Glancing at the SVE header itself, the only potential cause for this I can see is if __ARM_FEATURE_SVE is not defined for some reason (so it might be some issue in our build system).

To keep things simple for now: Could you try creating an SVE build without CUDA enabled? (basically the same build, just with ~cuda and without cuda_arch). Mixing SVE and CUDA builds is just not something we have tested before, so I would suggest that we stick with pure SVE for now until we can get that running first.

junghans commented 8 months ago

Yup it is still in aggregation_manager.hpp. (See will shared the log with you via email)

junghans commented 8 months ago

WIth cuda, tis fails with:

/vast/home/junghans/octotiger-spack/spack/lib/spack/env/gcc/g++ -DCPPUDDLE_HAVE_HPX -DCPPUDDLE_HAVE_HPX_AWARE_ALLOCATORS -DCPPUDDLE_HAVE_HPX_MUTEX -DCPPUDDLE_HAVE_MAX_NUMBER_GPUS=1 -DCPPUDDLE_HAVE_NUMBER_
BUCKETS=128 -DH5_BUILT_AS_DYNAMIC_LIB -DHPX_KOKKOS_CUDA_FUTURE_TYPE=0 -DHPX_KOKKOS_SYCL_FUTURE_TYPE=0 -DHPX_LIBRARY_EXPORTS -DHPX_WITH_CUDA -DKOKKOS_DEPENDENCE -DOCTOTIGER_GRIDDIM=8 -DOCTOTIGER_HAVE_KOKKO
S -DOCTOTIGER_HAVE_STD_EXPERIMENTAL_SIMD -DOCTOTIGER_HAVE_VC -DOCTOTIGER_KOKKOS_HYDRO_TASKS=1 -DOCTOTIGER_KOKKOS_MONOPOLE_TASKS=1 -DOCTOTIGER_KOKKOS_MULTIPOLE_TASKS=1 -DOCTOTIGER_KOKKOS_SIMD_SVE -DOCTOTIG
ER_MAX_NUMBER_FIELDS=15 -DOCTOTIGER_THETA_MINIMUM=0.34 -DSVE_LEN=256 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200809L -Dhydrolib_EXPORTS -I/vast/hom
e/junghans/octotiger-spack/octotiger -I/vast/home/junghans/octotiger-spack/octotiger/spack-build/_deps/svesimd-src/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse
_n1/gcc-12.3.0/hpx-1.9.1-6ccd5erkj3dgdcbj34oe6y3g7tvpmag4/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/boost-1.84.0-gtc6kjmh2hw3et27lfofdepq5y4ty
2ud/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/hwloc-2.9.1-nsillyjc2h5xhv3k6hs7jvvs6tnivd5d/include -isystem /vast/home/junghans/octotiger-spac
k/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/asio-1.22.2-r4jbde74qjc3c3dq26i7xulb7nkd6dtm/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/si
lo-4.11-f6niwvryvee6jtkldolycxme56ymqgji/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/hdf5-1.14.1-2-i2rtxbxarf54wbpd5xsurtkd7l3mvuvh/include -isy
stem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/zlib-1.2.13-xyz7izlqcwg44f4nkz55apfqzgrlsmo5/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spac
k/linux-rhel9-neoverse_n1/gcc-12.3.0/cppuddle-0.3.1-m3c5cwxelrbwfpwnnpzxbc35geol4snk/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/vc-1.4.1-3sxlec
hnpz3743ldgoqbektzx4zu7nwf/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/kokkos-4.1.00-qdgwacoe5gbkjvcwmlmv2l726gavhvdm/include -isystem /vast/hom
e/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/hpx-kokkos-0.4.0-537vo6kmlgve6d6ltr2xhuljsatprs5p/include -Wno-cpp -fPIC -O3 -DNDEBUG -std=gnu++20 -fPIC -msve-vector-bits=256
 -pthread -MD -MT CMakeFiles/hydrolib.dir/src/test_problems/sod/exact_sod.cpp.o -MF CMakeFiles/hydrolib.dir/src/test_problems/sod/exact_sod.cpp.o.d -o CMakeFiles/hydrolib.dir/src/test_problems/sod/exact_s
od.cpp.o -c /vast/home/junghans/octotiger-spack/octotiger/src/test_problems/sod/exact_sod.cpp
In file included from /vast/home/junghans/octotiger-spack/octotiger/octotiger/common_kernel/kokkos_simd.hpp:32,
                 from /vast/home/junghans/octotiger-spack/octotiger/octotiger/unitiger/hydro_impl/flux_kernel_templates.hpp:11,
                 from /vast/home/junghans/octotiger-spack/octotiger/octotiger/unitiger/hydro_impl/hydro_kokkos_kernel.hpp:22,
                 from /vast/home/junghans/octotiger-spack/octotiger/src/unitiger/hydro_impl/hydro_kernel_interface.cpp:16:
/vast/home/junghans/octotiger-spack/octotiger/octotiger/common_kernel/std_simd.hpp:51:28: error: 'sve' has not been declared
   51 | namespace SIMD_NAMESPACE = sve::experimental::parallelism_v2;
      |                            ^~~
/vast/home/junghans/octotiger-spack/octotiger/octotiger/common_kernel/std_simd.hpp:51:47: error: 'parallelism_v2' is not a namespace-name
   51 | namespace SIMD_NAMESPACE = sve::experimental::parallelism_v2;
      |                                               ^~~~~~~~~~~~~~
G-071 commented 8 months ago

The PR (and associated Octo-Tiger PR) actually seem to work for me on a Grace preproduction system using and SVE length of 128.

That is with using the octotiger PR (branch name is: allow_different_SVE_lengths) and the following Spack dev-build:

spack dev-build --until cmake --drop-in bash octotiger@master cxxstd=20 simd_extension=SVE simd_library=STD build_type==RelWithDebInfo sve_length=128 +kokkos +kokkos_hpx_kernels %gcc@12 ^hpx generator=make max_cpu_count=144 +generic_coroutines malloc=mimalloc
networking=lci ^silo~mpi ^lci~vector ^rdma-core~man_pages~pyverbs ^cppuddle@0.3.1 number_buffer_buckets=144

or, to have a more minimal build without the lci parcalport networking part:

spack dev-build --until cmake --drop-in bash octotiger@master cxxstd=20 simd_extension=SVE simd_library=STD build_type==RelWithDebInfo sve_length=128 +kokkos +kokkos_hpx_kernels %gcc@12 ^hpx generator=make max_cpu_count=144 +generic_coroutines malloc=mimalloc ^silo~mpi ^cppuddle@0.3.1 number_buffer_buckets=144

The most annoying part on the system I was on, was actually to get a gcc@12 up and running (I had to disable mpfr and mpc in the gcc Spack recipe to get it to compile without complaining about missing dependencies). Notably, I did not encounter any problems with Boost 1.84 or asio though, so there might be some differences between our setups still.

Some differences I can think of right now:

@junghans Could you try a fresh build with the same spack version, spack spec and octo-tiger spack branch?

G-071 commented 8 months ago

Here is the concretized spec I was using for the aforementioned, working Spack dev-build:

Input spec                                                                                                                                                                                                                                                                                     
--------------------------------                                                                                                                                                                                                                                                               
 -   octotiger@master%gcc@12+kokkos+kokkos_hpx_kernels build_type==RelWithDebInfo cxxstd=20 simd_extension=SVE simd_library=STD sve_length=128                                                                                                                                                 
 -       ^cppuddle@0.3.1 number_buffer_buckets=144                                                                                                                                                                                                                                             
 -       ^hpx+generic_coroutines generator=make malloc=mimalloc max_cpu_count=144 networking=lci                                                                                                                                                                                               
 -       ^lci~vector                                                                                                                                                                                                                                                                           
 -       ^rdma-core~man_pages~pyverbs                                                                                                                                                                                                                                                          
 -       ^silo~mpi                                                                                                                                                                                                                                                                             

Concretized                                                                                                                                                                                                                                                                                    
--------------------------------                                                                                                                                                                                                                                                               
 -   octotiger@master%gcc@12.3.0~boost_multiprecision~cuda~fast_fp_contract~ipo+kokkos+kokkos_hpx_kernels~rocm~sycl build_system=cmake build_type=RelWithDebInfo cxxstd=20 generator=make griddim=8 hydro_host_tasks=1 monopole_host_tasks=1 multipole_host_tasks=1 simd_extension=SVE simd_lib
rary=STD sve_length=128 theta_minimum=0.34 arch=linux-ubuntu22.04-neoverse_v2                                                                                                                                                                                                                  
[+]      ^boost@1.84.0%gcc@12.3.0+atomic+chrono~clanglibcpp~container+context~contract~coroutine+date_time~debug+exception~fiber+filesystem+graph~graph_parallel~icu+iostreams~json+locale+log+math~mpi+multithreaded~nowide~numpy~pic+program_options~python+random+regex+serialization+shared
+signals~singlethreaded~stacktrace+system~taggedlayout+test+thread+timer~type_erasure~versionedlayout+wave build_system=generic context-impl=fcontext cxxstd=17 patches=a440f96 visibility=hidden arch=linux-ubuntu22.04-neoverse_v2                                                          
[+]          ^bzip2@1.0.8%gcc@12.3.0~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-neoverse_v2
[e]              ^diffutils@3.8%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[+]          ^xz@5.4.1%gcc@12.3.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu22.04-neoverse_v2
[+]          ^zlib-ng@2.1.5%gcc@12.3.0+compat+opt build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[+]          ^zstd@1.5.5%gcc@12.3.0~programs build_system=makefile libs=shared,static arch=linux-ubuntu22.04-neoverse_v2
[e]      ^cmake@3.22.1%gcc@12.3.0~doc+ncurses+ownlibs build_system=generic build_type=RelWithDebInfo arch=linux-ubuntu22.04-neoverse_v2
[+]      ^cppuddle@0.3.1%gcc@12.3.0~allocator_counters+buffer_content_recycling+buffer_recycling~enable_gpu_tests+executor_recycling+hpx~ipo build_system=cmake build_type=RelWithDebInfo generator=make max_number_gpus=1 number_buffer_buckets=144 arch=linux-ubuntu22.04-neoverse_v2       
[+]      ^gcc-runtime@12.3.0%gcc@12.3.0 build_system=generic arch=linux-ubuntu22.04-neoverse_v2
[e]      ^gmake@4.3%gcc@12.3.0~guile build_system=generic patches=599f134 arch=linux-ubuntu22.04-neoverse_v2
[+]      ^hdf5@1.14.3%gcc@12.3.0~cxx~fortran+hl~ipo~java~map~mpi+shared+szip+threadsafe+tools api=default build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-ubuntu22.04-neoverse_v2                                                                                      
[+]          ^libaec@1.0.6%gcc@12.3.0~ipo+shared build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-ubuntu22.04-neoverse_v2
[e]          ^pkgconf@1.4.2%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[+]      ^hpx@1.9.1%gcc@12.3.0~async_cuda+async_gpu_futures~async_mpi~cuda~examples+generic_coroutines~ipo~lci_pp_log~lci_pp_pcounter~rocm~sycl~tools build_system=cmake build_type=RelWithDebInfo cxxstd=17 generator=make instrumentation=none malloc=mimalloc max_cpu_count=144 networking=l
ci sycl_target_arch=none arch=linux-ubuntu22.04-neoverse_v2
[+]          ^asio@1.28.0%gcc@12.3.0~boost_coroutine~boost_regex~separate_compilation build_system=autotools cxxstd=17 arch=linux-ubuntu22.04-neoverse_v2                                                                                                                                     
[e]          ^git@2.39.3%gcc@12.3.0+man+nls+perl+subtree~svn~tcltk build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[+]          ^hwloc@2.9.1%gcc@12.3.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-ubuntu22.04-neoverse_v2                                                                                                 
[+]              ^libpciaccess@0.17%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[+]                  ^util-macros@1.19.3%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[+]              ^libxml2@2.10.3%gcc@12.3.0+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[+]                  ^libiconv@1.17%gcc@12.3.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-neoverse_v2
[+]              ^ncurses@6.4%gcc@12.3.0~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[+]          ^lci@1.7.6%gcc@12.3.0+aligned~benchmarks~debug~debug-slow+default-dreg~docs~examples~gprof+ibv-td~inline-cq~ipo+multithread-progress+native~papi~pcounter+shared~tests~vector build_system=cmake build_type=RelWithDebInfo cache-line=auto completion=am,cq,sync default-max-cqe=a
uto default-max-recvs=auto default-max-sends=auto default-packet-size=auto default-packets=auto default-pm=auto fabric=ibv generator=ninja arch=linux-ubuntu22.04-neoverse_v2                                                                                                                 
[+]              ^ninja@1.11.1%gcc@12.3.0+re2c build_system=generic arch=linux-ubuntu22.04-neoverse_v2
[+]                  ^re2c@2.2%gcc@12.3.0 build_system=generic arch=linux-ubuntu22.04-neoverse_v2
[+]              ^rdma-core@49.0%gcc@12.3.0~ipo~man_pages~pyverbs+static build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-ubuntu22.04-neoverse_v2                                                                                                                       
[+]                  ^libnl@3.3.0%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[e]                      ^bison@3.8.2%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[e]                      ^flex@2.6.4%gcc@12.3.0+lex~nls build_system=autotools patches=f8b85a0 arch=linux-ubuntu22.04-neoverse_v2
[+]          ^mimalloc@2.1.2%gcc@12.3.0~build_tests~debug_full~ipo~local_dynamic_tls+override+padding~secure~see_asm~show_errors~skip_collect_on_exit~use_cxx~xmalloc build_system=cmake build_type=RelWithDebInfo generator=make libs=object,shared,static arch=linux-ubuntu22.04-neoverse_v2
[e]          ^python@3.10.12%gcc@12.3.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma+nis~optimizations+pic+pyexpat~pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,ebdca64,f2fd060 arch=linux-ubuntu22.04-neoverse_v2                         
[+]      ^hpx-kokkos@0.4.0%gcc@12.3.0~cuda~ipo~rocm~sycl build_system=cmake build_type=RelWithDebInfo cxxstd=17 future_type=polling generator=make arch=linux-ubuntu22.04-neoverse_v2                                                                                                         
[+]      ^kokkos@4.2.00%gcc@12.3.0~aggressive_vectorization~compiler_warnings~cuda~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples+hpx+hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning~wrapper 
build_system=cmake build_type=RelWithDebInfo cxxstd=17 generator=make intel_gpu_arch=none patches=145619e arch=linux-ubuntu22.04-neoverse_v2
[+]      ^silo@4.11%gcc@12.3.0+fortran+fpzip+hdf5+hzip~mpi+pic+shared~silex build_system=autotools patches=251244d,451c4c5,a081263,eb2a3a0,fa050e0 arch=linux-ubuntu22.04-neoverse_v2                                                                                                         
[e]          ^autoconf@2.71%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[+]          ^autoconf-archive@2023.02.20%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[e]          ^automake@1.16.5%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[+]          ^gnuconfig@2022-09-17%gcc@12.3.0 build_system=generic arch=linux-ubuntu22.04-neoverse_v2
[e]          ^libtool@2.4.6%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-neoverse_v2
[e]          ^m4@1.4.18%gcc@12.3.0+sigsegv build_system=autotools patches=3877ab5,fc9b616 arch=linux-ubuntu22.04-neoverse_v2
[e]          ^perl@5.34.0%gcc@12.3.0~cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu22.04-neoverse_v2
[+]          ^readline@8.2%gcc@12.3.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu22.04-neoverse_v2
[+]      ^vc@1.4.1%gcc@12.3.0~ipo build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-ubuntu22.04-neoverse_v2

(So, looking at the arch flag in your Spec, one more difference would be the OS used)

junghans commented 8 months ago

Except for https://github.com/G-071/octotiger-spack/pull/5#discussion_r1498134026, I got that to work.

junghans commented 8 months ago

Hmm, cmake rans fine, but at build time I will get:

[  3%] Building CXX object CMakeFiles/hydrolib.dir/src/unitiger/hydro_impl/hydro_kernel_interface.cpp.o
/vast/home/junghans/octotiger-spack/spack/lib/spack/env/gcc/g++ -DCPPUDDLE_HAVE_HPX -DCPPUDDLE_HAVE_HPX_AWARE_ALLOCATORS -DCPPUDDLE_HAVE_HPX_MUTEX -DCPPUDDLE_HAVE_MAX_NUMBER_GPUS=1 -DCPPUDDLE_HAVE_NUMBER_
BUCKETS=128 -DH5_BUILT_AS_DYNAMIC_LIB -DHPX_KOKKOS_CUDA_FUTURE_TYPE=0 -DHPX_KOKKOS_SYCL_FUTURE_TYPE=0 -DHPX_LIBRARY_EXPORTS -DHPX_WITH_CUDA -DKOKKOS_DEPENDENCE -DOCTOTIGER_GRIDDIM=8 -DOCTOTIGER_HAVE_KOKKO
S -DOCTOTIGER_HAVE_STD_EXPERIMENTAL_SIMD -DOCTOTIGER_HAVE_VC -DOCTOTIGER_HYDRO_HOST_HPX_EXECUTOR -DOCTOTIGER_KOKKOS_HYDRO_TASKS=1 -DOCTOTIGER_KOKKOS_MONOPOLE_TASKS=1 -DOCTOTIGER_KOKKOS_MULTIPOLE_TASKS=1 -
DOCTOTIGER_KOKKOS_SIMD_SVE -DOCTOTIGER_MAX_NUMBER_FIELDS=15 -DOCTOTIGER_THETA_MINIMUM=0.34 -DSVE_LEN=128 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=20
0809L -Dhydrolib_EXPORTS -I/vast/home/junghans/octotiger-spack/octotiger -I/vast/home/junghans/octotiger-spack/octotiger/spack-build/_deps/svesimd-src/include -isystem /vast/home/junghans/octotiger-spack/
spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/hpx-1.9.1-6ccd5erkj3dgdcbj34oe6y3g7tvpmag4/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/boost-
1.84.0-gtc6kjmh2hw3et27lfofdepq5y4ty2ud/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/hwloc-2.9.1-nsillyjc2h5xhv3k6hs7jvvs6tnivd5d/include -isyste
m /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/asio-1.22.2-r4jbde74qjc3c3dq26i7xulb7nkd6dtm/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/l
inux-rhel9-neoverse_n1/gcc-12.3.0/silo-4.11-f6niwvryvee6jtkldolycxme56ymqgji/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/hdf5-1.14.1-2-i2rtxbxar
f54wbpd5xsurtkd7l3mvuvh/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/zlib-1.2.13-xyz7izlqcwg44f4nkz55apfqzgrlsmo5/include -isystem /vast/home/jun
ghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/cppuddle-0.3.1-m3c5cwxelrbwfpwnnpzxbc35geol4snk/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-ne
overse_n1/gcc-12.3.0/vc-1.4.1-3sxlechnpz3743ldgoqbektzx4zu7nwf/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/kokkos-4.1.00-qdgwacoe5gbkjvcwmlmv2l7
26gavhvdm/include -isystem /vast/home/junghans/octotiger-spack/spack/opt/spack/linux-rhel9-neoverse_n1/gcc-12.3.0/hpx-kokkos-0.4.0-537vo6kmlgve6d6ltr2xhuljsatprs5p/include -Wno-cpp -fPIC -O2 -g -DNDEBUG -
std=gnu++20 -fPIC -msve-vector-bits=128 -pthread -ffp-contract=off -MD -MT CMakeFiles/hydrolib.dir/src/unitiger/hydro_impl/hydro_kernel_interface.cpp.o -MF CMakeFiles/hydrolib.dir/src/unitiger/hydro_impl/
hydro_kernel_interface.cpp.o.d -o CMakeFiles/hydrolib.dir/src/unitiger/hydro_impl/hydro_kernel_interface.cpp.o -c /vast/home/junghans/octotiger-spack/octotiger/src/unitiger/hydro_impl/hydro_kernel_interfa
ce.cpp
In file included from /vast/home/junghans/octotiger-spack/octotiger/octotiger/common_kernel/kokkos_simd.hpp:32,
                 from /vast/home/junghans/octotiger-spack/octotiger/octotiger/unitiger/hydro_impl/flux_kernel_templates.hpp:11,
                 from /vast/home/junghans/octotiger-spack/octotiger/octotiger/unitiger/hydro_impl/hydro_kokkos_kernel.hpp:22,
                 from /vast/home/junghans/octotiger-spack/octotiger/src/unitiger/hydro_impl/hydro_kernel_interface.cpp:16:
/vast/home/junghans/octotiger-spack/octotiger/octotiger/common_kernel/std_simd.hpp:51:28: error: 'sve' has not been declared
   51 | namespace SIMD_NAMESPACE = sve::experimental::parallelism_v2;
      |                            ^~~

this is with STEllAR-GROUP/octotiger@f42d82d3b0564e722be665c0169ddb04b9d74049 (which has allow_different_SVE_lengths) and eaef1fb8df5296d10816152319b245f0250b0760 and spack/spack@0960f691a1ad80beeb32a95ea77edcf2f3fc79cc.

G-071 commented 7 months ago

I suspect that for some reason __ARM_FEATURE_SVE is not set (since the include of the sve header work but the namespace therein is not defined afterwards).

Could you try the Octo-Tiger branch sve_build_improvements? It won't fix anything, but it'll give a more verbose error message if __ARM_FEATURE_SVE is actually undefined.

junghans commented 7 months ago

I think I ran into your new trap:

In file included from /vast/home/junghans/octotiger-spack/octotiger/octotiger/common_kernel/kokkos_simd.hpp:32,
                 from /vast/home/junghans/octotiger-spack/octotiger/octotiger/unitiger/hydro_impl/flux_kernel_templates.hpp:11,
                 from /vast/home/junghans/octotiger-spack/octotiger/octotiger/unitiger/hydro_impl/hydro_kokkos_kernel.hpp:22,
                 from /vast/home/junghans/octotiger-spack/octotiger/src/unitiger/hydro_impl/hydro_kernel_interface.cpp:16:
/vast/home/junghans/octotiger-spack/octotiger/octotiger/common_kernel/std_simd.hpp:49:2: error: #error "SVE Kokkos kernels are specified explicitly but build is without SVE support (__ARM_FEATURE_SVE not defined). Check architecture flags!"
   49 | #error "SVE Kokkos kernels are specified explicitly but build is without SVE support (__ARM_FEATURE_SVE not defined). Check architecture flags!"
      |  ^~~~~
G-071 commented 5 months ago

Closed as these changes will be merged as part a of #5