Support for the `Embedding` module

NVlabs / DoRA

[ICML2024 (Oral)] Official PyTorch implementation of DoRA: Weight-Decomposed Low-Rank Adaptation

https://nbasyl.github.io/DoRA-project-page/

Other

588 stars 36 forks source link

Closed ShayekhBinIslam closed 5 months ago

ShayekhBinIslam commented 5 months ago

DoRA is not currently supported when new tokens are added and embedding layer needs to be retrained. Is there any solution?

nbasyl commented 5 months ago

Hi, for retraining the embedding layer, I recommend using standard LoRA, as DoRA has not yet been tested on retraining the embedding layer.