[BUG] import deepspeed, MissingCUDAException

[zsaladin]

Describe the bug

It seems that deepspeed doesn't use cuda installed in virtual environment.

To Reproduce Steps to reproduce the behavior:

1. Remove your nvcc installed globally.
2. Create new virtual environment and activate it.
3. Install deepspeed==0.14.2

4. Write codes below and run it.

   source code

   import deepspeed

   error log

   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/.venv/lib/python3.9/site-packages/deepspeed/__init__.py", line 25, in <module>
       from . import ops
     File "/.venv/lib/python3.9/site-packages/deepspeed/ops/__init__.py", line 15, in <module>
       from ..git_version_info import compatible_ops as __compatible_ops__
     File "/.venv/lib/python3.9/site-packages/deepspeed/git_version_info.py", line 29, in <module>
       op_compatible = builder.is_compatible()
     File "/.venv/lib/python3.9/site-packages/deepspeed/ops/op_builder/fp_quantizer.py", line 29, in is_compatible
       sys_cuda_major, _ = installed_cuda_version()
     File "/.venv/lib/python3.9/site-packages/deepspeed/ops/op_builder/builder.py", line 50, in installed_cuda_version
       raise MissingCUDAException("CUDA_HOME does not exist, unable to compile CUDA op(s)")
   deepspeed.ops.op_builder.builder.MissingCUDAException: CUDA_HOME does not exist, unable to compile CUDA op(s)

   But cuda is located in virtual environment.

   ls .venv/lib/python3.9/site-packages/nvidia | grep cuda
   
   cuda_cupti
   cuda_nvrtc
   cuda_runtime

Expected behavior

1. It might be better if deepspeed uses cuda installed in virtual environment.

   Assume that cuda runtime is installed globally 11.6 and torch uses cuda runtime 12.1. In this case deepspeed and pytorch don't use same version of cuda. It can cause confusion.

   deepspeed requires cuda installed globally and torch install cuda in virtual environment. It increases docker image size.

2. nvcc might be unnecessary to check cuda version. nvcc is a compiler. To minimize docker images I don't want to install nvcc. But deepspeed uses it to check cuda version. We can use torch and print torch.cuda.version without nvcc.

System info (please complete the following information):

• OS: centos 7
• GPU count and types single machine with x8 V100s each]
• Python 3.9.17

[loadams]

Hi @zsaladin - the cuda runtime version and the cuda version that is bundled with torch can be different, so that is the reason why we need to be able to check the actual cuda runtime version that is installed.

[zsaladin]

@loadams Thanks for replying. I have some questions about your answer.

1. As you mentioned versions of cuda runtime installed globally and bundled with torch can be different. The cuda runtime bundled with torch is in virtual environment. Does deepspeed use cuda runtime in virtual environment? If deepspeed uses cuda runtime in virtual environment then the version conflict cannot happen. So it would be great that deepspeed will use bundled cuda runtime if deepspeed doesn't use it now.

2. When I use deepspeed==0.12.6 it doesn't require nvcc. nvcc is a compiler not for checking version. I'm not sure that but deepspeed needs to complie cuda code?

[loadams]

Hi @zsaladin - DeepSpeed uses the version of cuda runtime that is installed on the system, it cannot "use" the version that torch is built with, as that doesn't have nvcc/cuda drivers, it is just what the installed pytorch is built against.

As for why it didn't require nvcc in 0.12.6, we will have to check the code to see what changes have taken place that would cause this. With 0.12.6 does DeepSpeed detect that you are using an Nvidia GPU? Are you able to run nvidia-smi on your system with DeepSpeed 0.14.x?