search

kongds / MoRA

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
https://arxiv.org/abs/2405.12130
Apache License 2.0
341 stars 20 forks source link

Document mora_type #6

Closed winglian closed 5 months ago

winglian commented 5 months ago

Can you add more flavor about what each mora_type value means or refers to in the paper? The is pretty opaque currently and might be best handled by using enums for mora_type.

winglian commented 5 months ago

1 & 6 are defined in the readme as

    # type 1 (Sharing) for large lora ranks, Eq. 6 in paper
    # type 6 (RoPE based) for small lora ranks, Eq. 9 in paper

but there are several places in the code where it is compared to values 2, 3, or 4.

kongds commented 5 months ago

Thanks for your advice.

Types 2, 3, and 4 use sharing like Type 1, but with different share groups. They are used for ReMoRA, which merges MoRA and changes its types to increase rank of delta w by different share groups. https://github.com/kongds/MoRA/blob/0ff64b144e60b54fe7c0ff7b4e76c99c949e923d/train.py#L64-L70