search

oobabooga / text-generation-webui

A Gradio web UI for Large Language Models.
GNU Affero General Public License v3.0
40.46k stars 5.31k forks source link

Merging Lora and the model #2474

Closed SrVill closed 1 year ago

SrVill commented 1 year ago

Is it possible to add an option to merge Lora with the base model? From local directories. Because sometimes it's convenient. Since the existing scripts for merging Lora and the model do not work in Windows. If they work, only not for ordinary users, but for very well-trained people in this field. However, as I understand it, Webui is aimed at ordinary users, and it would be nice to further expand its functionality.

FartyPants commented 1 year ago

The scripts work well on windoze (run it from ooba env)

I was trying to post the script here, but the github formatting sucks. Anyway, look for merge_peft_adapters.py

SrVill commented 1 year ago

I found the script. Thank you for the hint. After a little editing, it works without problems directly in Windows. Although, I believe that this kind of function could still be added to webui sometime, since it is a useful option.

NicolasMejiaPetit commented 1 year ago

I found the script. Thank you for the hint. After a little editing, it works without problems directly in Windows. Although, I believe that this kind of function could still be added to webui sometime, since it is a useful option.

I was looking to do this as well on windows, if you don't mine me asking. What had to be changed in order for it to work on windows?

SrVill commented 1 year ago

I was looking to do this as well on windows, if you don't mine me asking. What had to be changed in order for it to work on windows?

I found another script in which nothing needs to be edited at all: https://github.com/bigcode-project/starcoder/blob/main/finetune/merge_peft_adapters.py Works great on Windows.

You can try these scripts again (Although this is almost the same as the first script): https://github.com/mzbac/qlora-fine-tune/blob/main/merge_peft_adapters.py https://github.com/bofenghuang/vigogne/blob/main/scripts/merge_peft_adapters.py

I could not get these scripts to work in Windows the way I need: https://github.com/tloen/alpaca-lora/blob/main/export_hf_checkpoint.py https://github.com/AlpinDale/lora-merge/blob/main/merge.py

tensiondriven commented 1 year ago

Somewhat related; does anyone happen to know if merging a lora into a base model actually adds more layers to the resultant model, or is the resultant model the same architecture as the base? Also, anyone know offhand how/if this affects catastrophic forgetting?

SrVill commented 1 year ago

Somewhat related; does anyone happen to know if merging a lora into a base model actually adds more layers to the resultant model, or is the resultant model the same architecture as the base? Also, anyone know offhand how/if this affects catastrophic forgetting?

Lora does not add additional layers, the architecture remains the same.

github-actions[bot] commented 1 year ago

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.