[Bug]: NCCL locating mechanism in multi-user environment

[ticoneva]

Your current environment

• 18-node cluster running Ubuntu 20.04 LTS.
• AMD Rome/Milan/Genoa CPUs. Mutliple GPUs on each node: A100/RTX 3090/RTX 3060.
• Conda environment on shared NFS drive. vLLM 0.4.0.post1 + PyTorch 2.2 + CUDA 12.1 environment.
• Job dispatched by Slurm.

🐛 Describe the bug

It seems that vLLM's NCCL detection mechanism is written with single user in mind. The vLLM-managed NCCL .so file is only installed for the user who installed vLLM. find_nccl_library is written such that if VLLM_NCCL_SO_PATH is not specified nor is the vLLM-managed version of NCCL found, it defaults to a hard-coded NCCL filename so_file, then reports to the user "Found nccl from library {so_file}".

I see two problems here:

1. [find_nccl_library] reports "Found nccl..." even when it has not. Perhaps it should say instead "Attempting to load nccl..."?
2. The vLLM-managed NCCL .so file is only installed to the home directory of the user who installed vLLM, meaning that it is inaccessible to other users. Why is it not installed along the package, which is accessible to all users?

[youkaichao]

The vLLM-managed NCCL .so file is only installed to the home directory of the user who installed vLLM, meaning that it is inaccessible to other users.

You can use VLLM_NCCL_SO_PATH to point to that file. Then all users can find it.

Why is it not installed along the package, which is accessible to all users?

Pypi package cannot exceed 100MB.

[find_nccl_library] reports "Found nccl..." even when it has not. Perhaps it should say instead "Attempting to load nccl..."?

Feel free to file a pr to improve it.