Closed Ghatage closed 9 months ago
If you do —test
it will look for the adapters in the default location so your test is not valid. You have to try testing without any adapters. It’s not well supported in the code but you can just comment the bit in the LoraLinear
layer where it adds in the adapters.
Ps if you’re interested send a PR to support testing without adapters. And let me know if you have other issues, can reopen the issue.
Great!
Since --adapter-file
was optional, to test with and without it to understand the performance of the finetune via --test
could be useful, so may be its a worthwhile effort.
I'll take a crack it it later and open a PR.
Thanks for the clarification!
Fwiw this is exactly why we do a validation at the very beginning of the training run. You might look at the training log again, maybe it has the information you care about already?
Hey folks,
Firstly thanks for making MLX!
I was trying to finetune mistral-7b, but before I did so, I wanted to validate the based model itself. It seems to me the base model itself is powerful enough to generate the required answers and what I see with the finetuning example is not really changing / adding anything.
This is using the latest main branch and download of mistral weights. (I did it all again from scratch for this issue)
Converted the model
Tested the base model without any lora adapters
As you can see the generated answer is correct and can be verified from the
train.jsonl
Generated the lora adapters based on the lora README and testing it:
Lora adapter + base model test:
Only base model test:
My questions are:
So I guess my question is, what am I doing differently here that is causing this outcome, I expected the base model to not perform as well on the queries from the
train.jsonl
and even after finetuning, I didn't expect the test loss and test ppl to be the same as before.