bitsandbytes-foundation / bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.
https://huggingface.co/docs/bitsandbytes/main/en/index
MIT License
6.35k stars 637 forks source link

ROCm Backend Status Tracker #1271

Open pnunna93 opened 4 months ago

pnunna93 commented 4 months ago

This issue summarizes and tracks the current status of ROCm enablement for BitsandBytes.

The completed changes are in multi-backend-refactor branch and will be merged into main once multi-backend design is finalized.

cc: @Titus-von-Koeller @sunway513 @amathews-amd

Titus-von-Koeller commented 4 months ago

@pnunna93

PyTorch 2.4 is built with ROCm 6.1, we should be sure that the bundled ROCm binaries are compatible with the newest PyTorch release that just came out.

Titus-von-Koeller commented 4 months ago

@pnunna93 I'm planning to do the alpha release with prebuilt wheels in the beginning of next week. Do you think you'll have the last item on this list checked by then? I think it would be important to prioritize this so we can move forward and get more user feedback.

pnunna93 commented 4 months ago

@Titus-von-Koeller Sure, I have been ironing out some details and got most of the info I need. I will start testing and open a PR early next week.