search

bitsandbytes-foundation / bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.
https://huggingface.co/docs/bitsandbytes/main/en/index
MIT License
6.33k stars 635 forks source link

Enable XPU and optimize cpu/xpu op #1418

Open jiqing-feng opened 6 days ago

jiqing-feng commented 6 days ago

Hi @Titus-von-Koeller . This PR has 2 changes:

  1. Support XPU for 8bit and 4bit @zhuhong61
  2. Optimize cpu/xpu OP

Additionally, we need to change transformers tests for:

  1. Add llama tests
  2. Disable gpt2 tests on XPU because of the precision issue (we will re-enable it once xpu fixed this issue)

See pull/34713

I think we can start to review this PR. I am preparing the XPU test env for you, we can sync it in Slack. Thanks!

Titus-von-Koeller commented 6 days ago

Thanks @jiqing-feng 🔥, referring to my answer on the related Transformers PR.

We'll prioritize testing and merging this asap, in balance with the other mentioned prios.

github-actions[bot] commented 6 days ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

jiqing-feng commented 6 days ago

Hi @Titus-von-Koeller . I fixed most code formats, but there still remains 2 error

Error: bitsandbytes/__init__.py:68:1: E402 Module level import not at top of file
Error: setup.py:43:5: E722 Do not use bare `except`

These 2 errors conflict with my changes, need your help to deal with them. Thanks!

jiqing-feng commented 5 days ago

Hi @Titus-von-Koeller . I fixed most code formats, but there still remains 2 error

Error: bitsandbytes/__init__.py:68:1: E402 Module level import not at top of file
Error: setup.py:43:5: E722 Do not use bare `except`

These 2 errors conflict with my changes, need your help to deal with them. Thanks!

Fix all format errors. We can discuss the setup issue after this PR merged.