Need to manually set CUDA_VISIBLE_DEVICES environment variable

NCAR / spack-gust

Spack production user software stack on the Gust test system

4 stars 0 forks source link

Need to manually set CUDA_VISIBLE_DEVICES environment variable #22

Open johnmauff opened 1 year ago

johnmauff commented 1 year ago

While I have been able to launch a single node GPU job on Gust I have not been able to launch a multi-node job. The application I am attempting to run is CM1 which uses OpenACC. The only output in the logfile is a single line indicating' signal 8'. I was able to get around this by setting the CUDA_VISILE_DEVICES equal to "0,1,2,3". CM1 subsequently starts up and fails with a different error #23 which is described in a different issue.

vanderwb commented 1 year ago

This is now handled by our get_local_rank helper script. Is there anything else that needs doing on this issue, @johnmauff ?