Clarifcation on the lora finetuning example for mistral

[Ghatage]

Hey folks,

Firstly thanks for making MLX!

I was trying to finetune mistral-7b, but before I did so, I wanted to validate the based model itself. It seems to me the base model itself is powerful enough to generate the required answers and what I see with the finetuning example is not really changing / adding anything.

This is using the latest main branch and download of mistral weights. (I did it all again from scratch for this issue)

---

1. Converted the model

   python convert.py --torch-path mistral-7B-v0.1/ --mlx-path mlx-model/

2. Tested the base model without any lora adapters

python lora.py --model mlx-model/ --num-tokens 50 --prompt "table: 1-10007452-3\ncolumns: Order Year, Manufacturer, Model, Fleet Series (Quantity), Powertrain (Engine/Transmission), Fuel Propulsion\nQ: what is the powertrain (engine/transmission) when the order year is 2000? A:"
Loading pretrained model
Total parameters 7243.436M
Trainable parameters 1.704M
Loading datasets
Generating
table: 1-10007452-3\ncolumns: Order Year, Manufacturer, Model, Fleet Series (Quantity), Powertrain (Engine/Transmission), Fuel Propulsion\nQ: what is the powertrain (engine/transmission) when the order year is 2000? A: SELECT Powertrain (Engine/Transmission) FROM 1-10007452-3 WHERE Order Year = 2000\nQ: what is the model that has the powertrain (engine/trans

3. As you can see the generated answer is correct and can be verified from the train.jsonl

4. Generated the lora adapters based on the lora README and testing it:

Lora adapter + base model test:

python lora.py --model mlx-model --adapter-file adapters.npz --test
Loading pretrained model
Total parameters 7243.436M
Trainable parameters 1.704M
Loading datasets
Testing
Test loss 1.351, Test ppl 3.862.

Only base model test:

python lora.py --model mlx-model --test
Loading pretrained model
Total parameters 7243.436M
Trainable parameters 1.704M
Loading datasets
Testing
Test loss 1.351, Test ppl 3.862.

My questions are:

• It seems mistral-7b can generate the accurate answer without the finetuning.
• Even after the finetuning and using the adapters.npz I see the same test loss and test ppl as using the model itself.

So I guess my question is, what am I doing differently here that is causing this outcome, I expected the base model to not perform as well on the queries from the train.jsonl and even after finetuning, I didn't expect the test loss and test ppl to be the same as before.

[awni]

If you do —test it will look for the adapters in the default location so your test is not valid. You have to try testing without any adapters. It’s not well supported in the code but you can just comment the bit in the LoraLinear layer where it adds in the adapters.

[awni]

Ps if you’re interested send a PR to support testing without adapters. And let me know if you have other issues, can reopen the issue.

[Ghatage]

Great! Since --adapter-file was optional, to test with and without it to understand the performance of the finetune via --test could be useful, so may be its a worthwhile effort. I'll take a crack it it later and open a PR.

Thanks for the clarification!

[awni]

Fwiw this is exactly why we do a validation at the very beginning of the training run. You might look at the training log again, maybe it has the information you care about already?