Closed KKNakkav2 closed 7 months ago
The error is resolved by updating the Params4Bit file in the bitsandbytes library. Thank you.
The error is resolved by updating the Params4Bit file in the bitsandbytes library. Thank you.
I have a similar problem. Could you share your changes to the Params4Bit?
The error is resolved by updating the Params4Bit file in the bitsandbytes library. Thank you.
I have the similar problem using the newest bitsandbytes version(0.43.0). Which bitsandbytes version?
In the LLM model_builder.py (https://github.com/alibaba/FederatedScope/blob/7f086944c57f85c7594bde44d4f6b981f0de6845/federatedscope/llm/model/model_builder.py#L22), if we set the model quantization to 4 bit,
It throws the error at the local training after the first aggregation round.
This error seems to be because of the way the 4bit weights are stored as a 1D tensor (https://github.com/TimDettmers/bitsandbytes/issues/902). Can the authors please advise if you were able to run the experiments with quantizated LLMs? Is there workaround for this issue.
Thank you