Open tensiondriven opened 1 year ago
It seems that the original monkey patch of xformers does not support GQA. Updated the monkey patch but not sure if it works.
I can train and save the lora file with fintune.py with the latest commit ver. But when I loaded the saved lora file and do inference, the output is broken within alpaca_lora_4bit and throw error within exllama
@laoda513 does the lora work when applied using text-generation-webui?
@tensiondriven emmmm, the thing is quite strange...
I found it's not because the lora cause the output broken.
The inference is just broken itself without loading the lora in fintune.py
(I commented the training code)
but the training seems still works...I'm testing load the lora and model with exllama to see if every is fine。。。
I'm currently only using script instead of text-generation-webui to keep things simple
I believe the problem might be that llama 2 70b has gqa in its model while any of the other llama models don’t. So that might have been messing something up
Hola! I attempted to do a fine-tuning run against Llama-2-70B-GPTQ (branch with file:
gptq_model-4bit--1g.safetensors
) this morning, and ran into the following error.Is this perhaps a configuration issue on my end, or does the repo need to be updated to support llama2?
Also, any feedback about how my parameters are configured would be welcome. I'm still have a heckova time getting concrete information about how some of the parameters are applied (micro_batch_size vs batch_size, lora_r, lora_alpha, lora_dropout).
Update: I looked through the available branches and couldn't find any containing a file named "gptq_model-4bit--1g.safetensors"; I wonder if the quant that i downloaded has been replaced.The file
gptq_model-4bit--1g.safetensors
is the file from the main/master branch.Any advice for which quant(s) to use for fine-tuning? Is there some better/worse compatibility with one vs another? Also, do some quants work better when a lora is applied than others? Any info would be quite useful.
Command line:
Output: