Open USMCM1A1 opened 7 months ago
This mostly happens to Qlora due to dequantization. The fused model will generate slightly different text. I have also noticed that quantizing the model with lm_head sometimes causes significant performance issues on the fused model for smaller models. Here are some options I can think of: 1) Stop quantizing lm_head. 2) Provide a load_with_adapter API in mlx-lm so that users can load an adapter with the base model to avoid the de-quantization issue.
@Awni, do you have any suggestions or thoughts?
- Stop quantizing lm_head.
- Provide a load_with_adapter API in mlx-lm so that users can load an adapter with the base model to avoid the de-quantization issue.
Thanks @mzbac for responding, but I'm a little confused about the quantizing part. When I converted the base model to mlx I didn't use a --q argument:
python convert.py --torch-path /Users/williammarcellino/mlx-examples/llms/mistral/mistral-7B-v0.1
So it shouldn't have been a quantized model, right?
- Stop quantizing lm_head.
- Provide a load_with_adapter API in mlx-lm so that users can load an adapter with the base model to avoid the de-quantization issue.
Thanks @mzbac for responding, but I'm a little confused about the quantizing part. When I converted the base model to mlx I didn't use a --q argument:
python convert.py --torch-path /Users/williammarcellino/mlx-examples/llms/mistral/mistral-7B-v0.1
So it shouldn't have been a quantized model, right?
Yeah, it looks like you're just doing lora. In that case, could you please check if you have specified the correct adapter-file path when fusing the model?
I used the same adapters file.
During the fusing, the Lora will be merged into the original linear layer, which may cause some slight differences when using the fused model. However, there shouldn't be such a big difference. I suspect the fusing wasn't done properly somehow.
@USMCM1A1 I double checked the fuse.py and found that there may be a bug when you are not using default lora layers. Before I can create a PR to fix it, you can update the code: https://github.com/ml-explore/mlx-examples/blob/main/lora/fuse.py#L56 to for l in model.model.layers[len(model.model.layers)-lora_layers:]
. Let me know if that could fix the issue.
I noticed today that when I use python -m mlx_lm.generate the output doesn't match what I get locally using python lora.py. For example: Local output using lora adapters:
Whereas the same prompt and arguments produces a pretty weird, incoherent response:
In particular it looks like the model forced out a bunch of blank tokens using mlx_lm.generate.