search

huggingface / peft

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.
https://huggingface.co/docs/peft
Apache License 2.0
16.46k stars 1.62k forks source link

FIX: Transpose weight matrix based on fan_in_fan_out condition in PiSSA initialization (#2103) #2104

Closed suyang160 closed 1 month ago

suyang160 commented 1 month ago

Previously, the weight matrix was converted to float32 without considering the need for transposition. This update ensures that the weight matrix is transposed when the fan_in_fan_out condition is met, resolving dimension mismatch issues during GPT-2 training.

suyang160 commented 1 month ago

@BenjaminBossan Thank you for your feedback and suggestions, I've updated this PR.

HuggingFaceDocBuilderDev commented 1 month ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.