search

huggingface / peft

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.
https://huggingface.co/docs/peft
Apache License 2.0
16.46k stars 1.62k forks source link

ENH: PiSSA/OLoRA: Preserve original config on save #2077

Closed BenjaminBossan closed 1 month ago

BenjaminBossan commented 2 months ago

Resolves #2075

When saving PiSSA or OLoRA with the option to convert to normal LoRA, the LoRA weight shapes change, which means that some values like r and lora_alpha need to be adjusted in the saved PEFT config. However, these modifications should be limited to the saved config, while the loaded config should stay the same.

This PR implements this change by creating a copy of the config before modifying it.

HuggingFaceDocBuilderDev commented 2 months ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

tokenizer-decode commented 2 months ago

Nice catch. LGTM.