Closed Blakey-Gavin closed 6 months ago
awaelchli
commented
6 months ago @Blakey-Gavin This project you linked is 3 years old and the version there is pytorch_lightning==1.2.1. We don't maintain such an old version anymore, so I won't be able to help. Try to upgrade to pytorch_lightning==1.2.10, maybe it was fixed.
awaelchli
commented
6 months ago @Blakey-Gavin Is the code in bytedance/music_source_separation actually made to support multi-GPU? The examples there suggest that you would only run on one device. Would you mind opening an issue there and asking about this?
TidalPaladin
commented
6 months ago I've encountered a similar issue after updating drivers/CUDA. I was trying to add mamba to a working pipeline but was having issues with NVCC / CUDA being out of date. I followed these instructions to update CUDA which allowed me to build mamba's dependencies. After this update multi-GPU with DDP would hang on startup. I thought this was a mamba issue but it impacted previously working models and remained after a re-init of the virtual environment.
I can reproduce the issue with BoringModel in my virtual environment:
import pytorch_lightning as L
from pytorch_lightning.demos.boring_classes import BoringModel
model = BoringModel()
trainer = L.Trainer(max_epochs=10, devices=2, strategy="ddp", logger=None)
trainer.fit(model)Hangs with
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/home/chase/.local/share/pdm/venvs/mammo-density-oNBH7oU9-mammo_density/lib/python3.11/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py:75: Starting from v1.9.0, `tensorboardX` has been removed as a dependency of the `pytorch_lightning` package, due to potential conflicts with other packages in the ML ecosystem. For this reason, `logger=True` will use `CSVLogger` as the default logger, unless the `tensorboard` or `tensorboardX` packages are found. Please `pip install lightning[extra]` or one of them to enable TensorBoard support by default
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
Missing logger folder: /home/chase/Documents/mammo-density/lightning_logs
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 2 processes
----------------------------------------------------------------------------------------------------
Missing logger folder: /home/chase/Documents/mammo-density/lightning_logsHere are the driver / CUDA versions for that system:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3090 On | 00000000:21:00.0 Off | N/A |
| 36% 38C P8 23W / 350W | 42MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 On | 00000000:4C:00.0 On | N/A |
| 0% 36C P8 39W / 390W | 28MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 2182 G /usr/lib/xorg/Xorg 21MiB |
| 0 N/A N/A 2703 G /usr/bin/gnome-shell 9MiB |
| 1 N/A N/A 2182 G /usr/lib/xorg/Xorg 21MiB |
+---------------------------------------------------------------------------------------+Unfortunately I do not know the driver / CUDA versions prior to update. However I do have a separate system for which BoringModel runs fine in an identical virtual environment:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.113.01 Driver Version: 535.113.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3090 Off | 00000000:21:00.0 Off | N/A |
| 0% 25C P8 22W / 350W | 2MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 Off | 00000000:4C:00.0 Off | N/A |
| 0% 25C P8 21W / 350W | 2MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+Details for hanging system (Ubuntu 22.04.3):
Details for working system:
awaelchli
commented
6 months ago @TidalPaladin The fact that it hangs with such a simple boring model must mean it will also fail with a regular torch distributed example.
Try one of the PyTorch multi-GPU examples, for example this one https://github.com/pytorch/examples/tree/main/distributed/ddp (I think this one you can just run main.py).
When you run these scripts, also enable NCCL_DEBUG=INFO.
You will most likely see the same hang, showing it's a problem with your system and not Lightning. A common issue is that P2P is not working. To confirm this, set NCCL_P2P_DISABLE=1. If it works, it will be super slow but it confirms that P2P is bad.
awaelchli
commented
6 months ago Also, I don't know why you needed a new driver, but since you said 535.113.0 works and 545.23.08 doesn't, try a version inbetween, that's still new enough for your needs. Also, the fact that you are able to isolate it to the driver is good. If all above does not help, I suggest posting in the PyTorch forum.
TidalPaladin
commented
6 months ago @awaelchli Thank you for your help. Setting NCCL_P2P_DISABLE=1 did indeed resolve the problem. I was able to revert to 535.154.05 + CUDA 12.2 and now everything works normally.
awaelchli
commented
6 months ago @Blakey-Gavin Can you try this suggestion as well?
Blakey-Gavin
commented
6 months ago @Blakey-Gavin This project you linked is 3 years old and the version there is
pytorch_lightning==1.2.1. We don't maintain such an old version anymore, so I won't be able to help. Try to upgrade topytorch_lightning==1.2.10, maybe it was fixed.
Sorry for the late reply, I'll try your suggestions, thanks.
Blakey-Gavin
commented
6 months ago @Blakey-Gavin Is the code in bytedance/music_source_separation actually made to support multi-GPU? The examples there suggest that you would only run on one device. Would you mind opening an issue there and asking about this?
Yes, I have used this project before. Before my driver was upgraded, if I configured the environment according to the "requirements.txt" it provided, there would be no problem with multi-GPUs training. However, training got stuck after the driver was upgraded.
Blakey-Gavin
commented
6 months ago @Blakey-Gavin Can you try this suggestion as well?
I have tried this(NCCL_P2P_DISABLE=1) before raising this issue, and it can indeed temporarily solve the problem. However, while the speed is slowing down, the main GPU also takes up more memory than other GPUs, so this actually leads to a new problem.
Bug description
At the link below I ran the code base and made almost no changes. https://github.com/bytedance/music_source_separation
When I first ran the code for the above library, my NVIDIA driver version was: 470.223.02, "libnccl-dev" and "libnccl2" was 2.11.4-1+cuda11.0. At this time, using a single machine with multiple gpus can run normally.
However, when I upgraded the NVIDIA driver to 545.23.08, it got stuck when running multiple gpus on a single machine under the same conditions.
While stuck, the GPU utilization was always 100%. As shown below:
At the same time, the CPU is always at 100%. As shown below:
What version are you seeing the problem on?
v1.8
How to reproduce the bug
No response
Error messages and logs
Environment
Current environment
* CUDA: - GPU: - NVIDIA GeForce RTX 3090 - NVIDIA GeForce RTX 3090 - NVIDIA GeForce RTX 3090 - NVIDIA GeForce RTX 3090 - NVIDIA GeForce RTX 3090 - NVIDIA GeForce RTX 3090 - NVIDIA GeForce RTX 3090 - NVIDIA GeForce RTX 3090 - available: True - version: 11.3 * Lightning: - lightning-utilities: 0.10.1 - pytorch-lightning: 1.5.0 - torch: 1.12.1+cu113 - torchaudio: 0.12.1+cu113 - torchinfo: 1.5.3 - torchlibrosa: 0.0.9 - torchmetrics: 1.3.0.post0 - torchvision: 0.13.1+cu113 * Packages: - absl-py: 1.2.0 - aiohttp: 3.8.3 - aiosignal: 1.2.0 - appdirs: 1.4.4 - async-timeout: 4.0.2 - attrs: 22.1.0 - audioread: 3.0.0 - cachetools: 5.2.0 - certifi: 2022.9.24 - cffi: 1.15.1 - charset-normalizer: 2.1.1 - cycler: 0.11.0 - datasets: 2.10.0 - decorator: 5.1.1 - dill: 0.3.6 - einops: 0.3.2 - ffmpeg-python: 0.2.0 - ffprobe: 0.5 - filelock: 3.9.0 - flatbuffers: 23.1.21 - fonttools: 4.37.4 - frozenlist: 1.3.1 - fsspec: 2022.8.2 - future: 0.18.2 - google-auth: 2.12.0 - google-auth-oauthlib: 0.4.6 - grpcio: 1.49.1 - h5py: 3.7.0 - huggingface-hub: 0.12.1 - idna: 3.4 - importlib-metadata: 5.0.0 - importlib-resources: 5.10.0 - joblib: 1.1.0 - jsonschema: 4.16.0 - kiwisolver: 1.4.4 - librosa: 0.9.2 - lightning-utilities: 0.10.1 - llvmlite: 0.40.1 - markdown: 3.4.1 - markupsafe: 2.1.1 - matplotlib: 3.5.0 - mkl-fft: 1.3.1 - mkl-random: 1.2.2 - mkl-service: 2.4.0 - mnn: 2.8.1 - more-itertools: 9.0.0 - mpmath: 1.3.0 - multidict: 6.0.2 - multiprocess: 0.70.14 - musdb: 0.4.0 - museval: 0.4.0 - natsort: 8.0.0 - numba: 0.57.0 - numpy: 1.22.0 - oauthlib: 3.2.1 - olefile: 0.46 - onnx: 1.13.0 - onnxruntime-gpu: 1.10.0 - opencv-python: 4.6.0.66 - packaging: 21.3 - pandas: 1.5.0 - pesq: 0.0.3 - pillow: 9.4.0 - pip: 22.2.2 - pkgutil-resolve-name: 1.3.10 - pooch: 1.6.0 - protobuf: 3.20.3 - pyaml: 21.10.1 - pyarrow: 11.0.0 - pyasn1: 0.4.8 - pyasn1-modules: 0.2.8 - pycparser: 2.21 - pydeprecate: 0.3.1 - pyparsing: 3.0.9 - pyrsistent: 0.18.1 - python-dateutil: 2.8.2 - pytorch-lightning: 1.5.0 - pytz: 2022.4 - pyyaml: 6.0 - regex: 2022.10.31 - requests: 2.28.1 - requests-oauthlib: 1.3.1 - resampy: 0.4.2 - responses: 0.18.0 - rsa: 4.9 - scikit-learn: 1.1.2 - scipy: 1.8.0 - setuptools: 59.8.0 - setuptools-scm: 7.0.5 - simplejson: 3.17.6 - six: 1.16.0 - soundfile: 0.10.3.post1 - stempeg: 0.2.3 - sympy: 1.12 - tensorboard: 2.11.0 - tensorboard-data-server: 0.6.1 - tensorboard-plugin-wit: 1.8.1 - threadpoolctl: 3.1.0 - tokenizers: 0.13.2 - tomli: 2.0.1 - torch: 1.12.1+cu113 - torchaudio: 0.12.1+cu113 - torchinfo: 1.5.3 - torchlibrosa: 0.0.9 - torchmetrics: 1.3.0.post0 - torchvision: 0.13.1+cu113 - tqdm: 4.64.1 - transformers: 4.26.1 - typing-extensions: 4.4.0 - urllib3: 1.26.12 - webrtcvad: 2.0.10 - werkzeug: 2.2.2 - wheel: 0.37.1 - xxhash: 3.2.0 - yarl: 1.8.1 - zipp: 3.9.0 * System: - OS: Linux - architecture: - 64bit - ELF - processor: x86_64 - python: 3.8.18 - release: 5.15.0-92-generic - version: #102~20.04.1-Ubuntu SMP Mon Jan 15 13:09:14 UTC 2024More info
Note: Since the version selection above is required, I chose one at random. In fact, my lightning version is 1.5.0.