In ncarenv/23.09 ncarcompilers does not wrap nvcc

[roryck]

The ncarcompilers isn't wrapping nvcc

module --force purge
module load ncarenv/23.09
module load nvhpc cray-mpich ncarcompilers
which nvcc
/glade/u/apps/common/23.08/spack/opt/spack/nvhpc/23.7/Linux_x86_64/23.7/compilers/bin/nvcc

Which leads to things like:

nvcc -c test.c
test.c:1:10: fatal error: mpi.h: No such file or directory
 #include <mpi.h>
          ^~~~~~~
compilation terminated.

unless you manually add

-I${CRAY_MPICH_PREFIX}/include

[vanderwb]

I think the issue here isn't so much that nvcc isn't being wrapped, but rather that nvcc isn't calling the MPI compiler wrapper by default:

[16:59] nvcc-test$ nvcc -c mpi.c
mpi.c:16:10: fatal error: mpi.h: No such file or directory
 #include "mpi.h"
          ^~~~~~~
compilation terminated.
[16:59] nvcc-test$ nvcc -ccbin mpicxx -c mpi.c
[16:59] nvcc-test$

I don't see a nice environment variable to set a default for that, and other options seem sub-optimal. The only thing I've found online that I could do would be to set NVCC_PREPEND_FLAGS:

[17:12] nvcc-test$ export NVCC_PREPEND_FLAGS=-ccbin=mpicxx
[17:12] nvcc-test$ nvcc -c mpi.c
[17:12] nvcc-test$

But that would hijack this variable which may be useful to folks/build systems? I'm thinking this may just be a user education situation, in which we need to inform people to use -ccbin. What do you think?

[roryck]

If ncarcompilers does wrap nvcc, can't it include the header path when MPI is loaded?

When using nvcc there are probably other C/C++ compilers in the environment, e.g. icx or nvc, and it makes sense for mpicc to prefer wrapping those over wrapping nvcc. Maybe it makes more sense to declare the CUDA compiler to be mpicc with nvcc as the underlying, inside makefiles an so on. I know how to do that with some MPIs, but not cray-mpich. I'll look into that as a better solution.

[vanderwb]

Well, ncarcompilers doesn't wrap nvcc, it wraps the compiler that nvcc calls. That distinction probably doesn't matter.

I can certainly add include and library paths for cray-mpich that the NCAR wrapper would pick up on. It just didn't seem necessary since the Cray wrapper CC will do this too. Unless of course you don't call CC, as is happening here with nvcc.

I'd like to confirm that having ncarcompilers do this doesn't upset the Cray wrapper. If not, I'll add those -I and -L flags soon.