RuntimeError: CUDA error: peer access is not supported between these two devices

tunib-ai / parallelformers

Parallelformers: An Efficient Model Parallelization Toolkit for Deployment

https://tunib-ai.github.io/parallelformers

Apache License 2.0

776 stars 61 forks source link

RuntimeError: CUDA error: peer access is not supported between these two devices #44

Open Dorcoh4 opened 1 year ago

Dorcoh4 commented 1 year ago

I tried running the example from the readme but received the above error. Does that mean that my hardware is not supported?

Environment

OS : Ubuntu
Python version : 3.7.11
Transformers version : 4.23.1
Whether to use Docker: no
GPUs: NVIDIA GeForce RTX 2080 Ti

sdake commented 11 months ago

@Dorcoh4 hello. I am not involved with this project, however the error you see indicates that (iirc) the PCI bus has been blocked from using root PCI bridges. This is common in virtualized cloud providers.

Where are you running the workload? I may have workarounds, depending on provider!