Compile generated sycl code with hipsycl

[tkoskela]

Kacper's instructions

For hipsycl on csd3 it is necessary to load the following modules:

module load rhel8/default-amp
module load hipsycl/0.9.1/gcc-9.4.0-gieutz2

Additionally for running a binary produced with hipsycl it may be necessary to load a module providing newer libstdc++, for example:

module load gcc/9.4.0

[tkoskela]

Results

Makefile

cat Makefile
SHELL:=/bin/bash
CC=dpcpp
CFLAGS=-O2 -Wall
LDFLAGS=
INCLUDES=-Iincludes

intel_cpu: CC = dpcpp

nvidia_gpu: CC = clang++
nvidia_gpu: CFLAGS += -fsycl -fsycl-targets=nvptx64-nvidia-cuda

hip_gpu: CC = syclcc

all: intel_cpu nvidia_gpu

intel_cpu: main.c dw_soa_sycl.cpp read_in.c
    $(CC) $(INCLUDES) $(CFLAGS) $^ -o main.intel_cpu

nvidia_gpu: main.c dw_soa_sycl.cpp read_in.c
    $(CC) $(INCLUDES) $(CFLAGS) $^ -o main.nvidia_gpu

hip_gpu:  main.c dw_soa_sycl.cpp read_in.c
    $(CC) $(INCLUDES) $(CFLAGS) $^ -o main.hip_gpu

clean:
    rm -f *.o main a.out 2>/dev/null

Environment

(base) [dc-kosk1@login-q-1 dpct_output]$ module purge
(base) [dc-kosk1@login-q-1 dpct_output]$ module load rhel8/default-amp
Loading rhel8/default-amp
  Loading requirement: dot rhel8/slurm singularity/current rhel8/global cuda/11.4 openmpi/4.1.1/gcc-9.4.0-epagguv
(base) [dc-kosk1@login-q-1 dpct_output]$ module load hipsycl/0.9.2/gcc-9.4.0-jg2gfgh
(base) [dc-kosk1@login-q-1 dpct_output]$ module load gcc/9.4.0
(base) [dc-kosk1@login-q-4 dpct_output]$ export HIPSYCL_TARGETS=cuda:sm_80
(base) [dc-kosk1@login-q-4 dpct_output]$ export HIPSYCL_GPU_ARCH=sm_80

Code modifidactions

• The HIPSycl implementation on csd3 wraps the sycl namespace inside the cl namespace. I had to add using namespace cl; to the code (or replace sycl:: with cl::sycl::). This did not break compatibility with OneApi

[tkoskela]

Kacper (12 Apr): I see that hipsycl started to support some atomic operations only in 0.9.2. I will try to make a build of it for you later today. I've just added module for newer version of hipsycl on csd3: hipsycl/0.9.2

[tkoskela]

Building hipsycl on cosma

• I cloned the latest spack release in my home directory and ran the setup script
• I’ve got the following modules loaded

  1) cosma/2018                    4) gadgetviewer/1.1.2   7) gnu_comp/10.2.0  10) cmake/3.18.1  
  2) allinea/ddt/20.0.3(default)   5) utils/201805         8) boost/1_67_0     
  3) hdfview/2.14.0                6) git/2.27.0           9) python/3.10.1  

• I’m using the cosma spack environment from ExCALIBUR (once https://github.com/ukri-excalibur/excalibur-tests/pull/22 is merged)
• Then I ran spack install hipsycl ^boost@1.67.0 %gcc@10.2.0
• You can check with spack find -pl that it didn’t build stuff it was supposed to find in the environment. I’ve got

  lqmnyvq boost@1.67.0   /cosma/local/boost/gnu_10.2.0/1_67_0/
  ysheyx7 hipsycl@0.9.1  /cosma/home/do008/dc-kosk1/spack/opt/spack/linux-centos7-zen2/gcc-10.2.0/hipsycl-0.9.1-ysheyx7ikamgq34en73mlanjorh52ku3
  kzouybx llvm@12.0.0    /cosma/local/aocc/aocc-compiler-3.0.0
  nudfk4q python@3.7.10  /cosma/local/intel/oneAPI_2021.3.0/intelpython/latest

• and we’re done! :tada: Load it into PATH with spack load hipsycl

[tkoskela]

Reopening this because there are issues with building for AMD GPUs on Cosma8 that need more investigation.

• [x] Check system level ROCm installation
• [x] Install ROCm from spack https://spack.readthedocs.io/en/latest/build_systems/rocmpackage.html

[tkoskela]

Closing this and continuing investigation in #29