Compile generated SYCL code with `dpcpp`

[tkoskela]

We have generated SYCL code in UCL/openqcd-oneapi#20. The next step is compiling.

[tkoskela]

How to compile on csd3

• Remove the cuda module from the environment. The existence of /usr/local/software/cuda/9.2/include/CL will clash with the OpenCL version of OneAPI.
• cp Makefile dpct_output
• Edit dpct_output/Makefile, diff of original to edited:

  1,2c1,2
  < CC=gcc
  < CFLAGS=-c -std=c99 -O2 -Wall
  ---
  > CC=dpcpp
  > CFLAGS=-c -O2 -Wall
  5,8d4
  < CUDAC=nvcc
  < CUDACFLAGS=-c -arch=sm_70 -D_MWAITXINTRIN_H_INCLUDED -D_FORCE_INLINES -D__STRICT_ANSI__
  < CUDALFLAGS=-lcuda -lcudart -lm -lstdc++
  < 
  12c8
  < CUDAOBJECTS=dw_cuda.o
  ---
  > SYCLOBJECTS=dw_cuda.dp.o
  16,17c12,13
  < main: $(OBJECTS) $(CUDAOBJECTS)
  <   $(CC) -o $@ $^ $(LDFLAGS) $(CUDALFLAGS)
  ---
  > main: $(OBJECTS) $(SYCLOBJECTS)
  >   $(CC) -o $@ $^ $(LDFLAGS)
  22,23c18,19
  < %.o: %.cu
  <   $(CUDAC) $(CUDACFLAGS) $(INCLUDES) $< -o $@
  ---
  > %.o: %.cpp
  >   $(CC) $(CCFLAGS) $(INCLUDES) $< -o $@

• cd dpct_output
• make

[tkoskela]

Compilation fails on csd3 with

"/usr/bin/ld" --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o dw_cuda.dp.o /lib/../lib64/crt1.o /lib/../lib64/crti.o /usr/local/software/master/gcc/8/lib/gcc/x86_64-pc-linux-gnu/8.4.0/crtbegin.o -L/usr/local/software/intel/oneapi/2022.1/compiler/2022.0.2/linux/bin-llvm/../compiler/lib/intel64_lin -L/usr/local/software/intel/oneapi/2022.1/compiler/2022.0.2/linux/bin-llvm/../lib -L/usr/local/software/intel/oneapi/2022.1/compiler/2022.0.2/linux/bin-llvm/../compiler/lib/intel64_lin -L/usr/local/software/master/gcc/8/lib/gcc/x86_64-pc-linux-gnu/8.4.0 -L/usr/local/software/master/gcc/8/lib/gcc/x86_64-pc-linux-gnu/8.4.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/local/software/master/gcc/8/lib/gcc/x86_64-pc-linux-gnu/8.4.0/../../.. -L/usr/local/software/intel/oneapi/2022.1/compiler/2022.0.2/linux/bin-llvm/../lib -L/lib -L/usr/lib -L/usr/local/software/intel/oneapi/2022.1/vpl/2022.0.0/lib -L/usr/local/software/intel/oneapi/2022.1/tbb/2021.5.1/env/../lib/intel64/gcc4.8 -L/usr/local/software/intel/oneapi/2022.1/mpi/2021.5.1//libfabric/lib -L/usr/local/software/intel/oneapi/2022.1/mpi/2021.5.1//lib/release -L/usr/local/software/intel/oneapi/2022.1/mpi/2021.5.1//lib -L/usr/local/software/intel/oneapi/2022.1/mkl/2022.0.2/lib/intel64 -L/usr/local/software/intel/oneapi/2022.1/ipp/2021.5.2/lib/intel64 -L/usr/local/software/intel/oneapi/2022.1/ippcp/2021.5.1/lib/intel64 -L/usr/local/software/intel/oneapi/2022.1/ipp/2021.5.2/lib/intel64 -L/usr/local/software/intel/oneapi/2022.1/dnnl/2022.0.2/cpu_dpcpp_gpu_dpcpp/lib -L/usr/local/software/intel/oneapi/2022.1/dal/2021.5.3/lib/intel64 -L/usr/local/software/intel/oneapi/2022.1/compiler/2022.0.2/linux/compiler/lib/intel64_lin -L/usr/local/software/intel/oneapi/2022.1/compiler/2022.0.2/linux/lib -L/usr/local/software/intel/oneapi/2022.1/clck/2021.5.0/lib/intel64 -L/usr/local/software/intel/oneapi/2022.1/ccl/2021.5.1/lib/cpu_gpu_dpcpp -L/usr/local/software/master/gcc/8/lib64 -L/usr/local/software/master/gcc/8/lib /tmp/dw_cuda-edb674.o /tmp/dw_cuda-bc67aa.o -Bstatic -lsvml -Bdynamic -Bstatic -lirng -Bdynamic -lstdc++ -Bstatic -limf -Bdynamic -lm -lgcc_s -lgcc -Bstatic -lirc -Bdynamic -ldl -lgcc_s -lgcc -lsycl -lc -lgcc_s -lgcc -Bstatic -lirc_s -Bdynamic /usr/local/software/master/gcc/8/lib/gcc/x86_64-pc-linux-gnu/8.4.0/crtend.o /lib/../lib64/crtn.o
/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
dpcpp: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [dw_cuda.dp.o] Error 1```

[tkoskela]

Kacper:

BTW please do not expect to be able to run the produced code on Nvidia hardware. I've just got confirmation from Intel, that to be able to do it, we need to compile dpc++ ourselves.

Intel confirmed that the dpcpp compiler provided in the OneAPI package only supports Intel devices. For NVidia you need to compile the open-source version of the compiler. We have it installed on CSD3. To use it

module purge
module use /usr/local/software/spack/spack-modules/dpcpp-cuda-20220220/linux-centos8-x86_64_v3/
module load dpcpp
module load gcc/11.2.0

NOTE: when running on the NVidia GPU nodes, these modules also need to be loaded

[tkoskela]

Plan for the short term

• Try on CPU hardware on CSD3
• Wait for response from Chris Edsall regarding GPU options

[tkoskela]

Makefiles auto-generated with --gen-build-script are not working for me. I can compile on the command line with

clang++ -Iincludes -fsycl -fsycl-targets=nvptx64-nvidia-cuda main.c read_in.c dw_cuda.dp.cpp -o main