Closed eryk-mazus closed 6 months ago
I noticed that, you're doing load_in_4bit
but training a lora
instead of qlora
. Perhaps a typo?
So, I thought it was just a typo that broken something, but I retrained the model with load_in_4bit: true
and adapter: qlora
and initially got the same garbage as before. But, what I discovered is that when I create the input as string instead of using tokenizer.apply_chat_template
(same as here) the model starts to produce the correct output.
I will investigate this further, but I wanted to write it here in case someone encounters the same issue
The chat template is still a new feature, only a few are supported. if you're training a template different from base model, you'll need to set it yourself usually.
Please check that this issue hasn't been reported before.
Expected Behavior
I'm fine-tuning Mistral 7b using the dataset of 250k conversation non english conversations. I fine-tuned way smaller model (tinyllama) using axolotl on the subset of dataset, which turned out great, so there has to be something wrong with my config/fine-tuning/adapter merging. I'd be helpful for help.
Current behaviour
After merging the adapter using
python3 -m axolotl.cli.merge_lora path_to_config
I tried to generate output on some simple prompt. The model returns weird string of characters likeWWWWWWWW
, white spaces, etc.Steps to reproduce
Here is the config that I'm using:
Config yaml
Possible solution
Do my lora settings make sense ? Do I set the tokens/special tokens correctly ?
Which Operating Systems are you using?
Python Version
3.10
axolotl branch-commit
main
Acknowledgements