Closed ChenMnZ closed 1 month ago
@ChenMnZ We will check again if BITBLAS has recently added sharding support.
Need to re-open this. The fix pr was reverted as the unit-test was incorrectly passing when it should have failed for sharding+bitblas.
When initializing a BitBLAS quant linaer, fixed values of N and K are required. However, the shard runtime dispatches the shard shape to the runtime kernel. It might be beneficial to draw insights from VLLM, which allows initialization to be cognizant of shard information.
Fixed with PR #316
Hello, thanks for your great work to make AutoGPTQ more useable.
I want to load GPTQ model and inference with BitBLAS backend.
The corresponding model loading code is:
Such manner works on small models (4-bit 7B or 2-bit 13B). However, it failed with large model, such as 4-bit 13B. The raised error is:![image](https://github.com/user-attachments/assets/7d96dc41-56de-42c0-b71a-1cfa187ae668)