Open flozi00 opened 9 months ago
try —bits 4 or 8
does not depends on this params, still the same error but with 4bit linear instead of linear
Use --bf16 instead of --fp16 The torch_dtype of the Mistral-7B-v0.1 or the zephyr-7b-beta is "bfloat16". Check the model's config.json file.
same error, the attribute still not found which versions are you using ?
I did some experiments and can see that excactly 1 of 3 linear layers are lora layers only. I added an filter and now its running need to check if its possible to load it successfully too.
could you maybe add an gradio demo on huggingface spaces or docker container where the current script is running successfully ?
I have been using the GitHub dev branch all along, and synchronizing it with the main branch in a timely manner. The PyPI version updates may be slower. Let me check if it is indeed a version issue. The Docker version will be added soon. Thank you for your suggestions and support.
I noticed that the correct number of model modules should be 224 instead of 673 when there is an error. I am investigating possible reasons for the error, and it seems that it should not be a program issue but possibly due to certain reasons related to loading the model.
Okay, then the filter I added is correct Should I open an PR?
I'm very glad you did this.
@flozi00 I'm getting the same error. Could you create a patch of your changes? @uukuguy Would this work with mixtral models as well?
@flozi00 I'm getting the same error. Could you create a patch of your changes? @uukuguy Would this work with mixtral models as well?
Would you please provide the script that encountered an error?
python3 -m multi_loras \
extract_lora \
--base_model_name_or_path "mistralai/Mistral-7B-v0.1" \
--tuned_model_name_or_path "HuggingFaceH4/zephyr-7b-beta" \
--save_path "./mistral-zephyr-lora" \
--bf16 \
--bits 4 \
--lora_r 128
python -m multi_loras extract_lora --base_model_name_or_path /home/ubuntu/models/Mixtral/Mixtral-8x7B-Instruct-v0.1/ --tuned_model_name_or_path /home/ubuntu/models/meetkai/functionary-medium-v2.2/ --save_path /home/ubuntu/models/loras/ --bf16 --bits 8 --lora_r 128
BTW, I managed to run the script after posting this like this:
for (name_base, lora_base), (name_tuned, lora_tune) in pbar:
assert name_base == name_tuned, f"name_base={name_base} != name_tuned={name_tuned}"
residual = lora_tune.weight.data - lora_base.weight.data
pbar.set_postfix({"layer": name_base.replace("base_model.model.", ""), "shape": residual.shape})
try:
assert lora_base.lora_A
except:
print("no lora_A here")
else:
# SVD on residual
U, Vh = svd_distill(residual, rank=rank, clamp_quantile=clamp_quantile)
assert lora_base.lora_A.default.weight.shape == Vh.shape, f"{lora_base=}"
assert lora_base.lora_B.default.weight.shape == U.shape, f"{lora_base=}"
lora_base.lora_A.default.weight.data = Vh.to(device=device, dtype=dtype)
lora_base.lora_B.default.weight.data = U.to(device=device, dtype=dtype)
The question about Mixtral still stands... I'm not able to test the lora just yet as lorax is throwing error for me. I will have to switch to other program and try
There is also another minor issue when installed using pip as stated in readme.md:
in multi_loras/multi_loras/main.py
from .merge_peft_adapters import do_merge_lora
To make it work, I had to modify multi_loras/merge_peft_adapters.py (untested)
.....
def get_args():
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--base_model_name_or_path", type=str)
# parser.add_argument("--peft_model_path", type=str)
parser.add_argument("--lora_model_path", type=str)
parser.add_argument("--merged_model_name_or_path", type=str, default=None)
parser.add_argument("--push_to_hub", action="store_true", default=False)
args = parser.parse_args()
def do_merge_lora(args):
merge_peft_adapters(base_model_name_or_path=args.base_model_name_or_path,
# peft_model_path=args.peft_model_path,
peft_model_path=args.lora_model_path,
merged_model_name_or_path=args.merged_model_name_or_path,
push_to_hub=args.push_to_hub
)
def main():
args = get_args()
do_extract_lora(args)
if __name__ == "__main__" :
main()
errors in:
not depeding on quant or precision