search

bitsandbytes-foundation / bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.
https://huggingface.co/docs/bitsandbytes/main/en/index
MIT License
6.29k stars 630 forks source link

Any plan to expand device support for bitsandbytes? #961

Open SlightwindSec opened 10 months ago

SlightwindSec commented 10 months ago

Feature request

I am deeply appreciative of your work with bitsandbytes as it has tremendously helped enhance my workflow. Your efforts are much appreciated! I have noticed that bitsandbytes is tightly linked with CUDA at both the C++ and Python end. This leads me to inquire if you plan to abstract "device" instead of "CUDA" in the future, akin to what PyTorch does, to enable bitsandbytes' support for a broader range of devices?

Motivation

The emergence of an array of devices that accelerates neural network computations, such as Apple silicon, AMD GPUs, and Ascend NPU, has provided more options beyond the widely used NVIDIA GPUs. Although I understand that some of the NVIDIA GPU-specific optimization strategies may not yield equivalent performance on these other platforms, the availability of the core features of bitsandbytes on more platforms would make a significant difference.

Contribution

Our team is interested in bringing bitsandbytes support to the Ascend NPU devices. We can certainly contribute by developing and submitting the necessary code via a PR. Is there any existing plan about expanding device support that we should be aware of? Thank you!

xiabo0816 commented 7 months ago

+1

apachemycat commented 5 months ago

+1
/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. "
ascend npu env

SlightwindSec commented 5 months ago

If you want to use bitsandbytes on your Ascend device, you can contact me at slightwindsec@gmail.com.