search

jianghoucheng / AlphaEdit

36 stars 4 forks source link

Is there any tip to tune the parameters? #2

Open littlefive5 opened 3 weeks ago

littlefive5 commented 3 weeks ago

Hello,

We're integrating this fabulous AlphaEdit method into the EasyEdit tool and the results in LLAMA3-8B are perfect. However, we found the results on LLAMA2-7b-chat are not satisfactory. We have tried different parameters of the v_lr, weight_decay, and null_space_threshold but the results change little. Can you provide some suggestions on obtaining better performance from a new model?

Thanks a lot.

EasyEdit Team

jianghoucheng commented 2 weeks ago

Hello EasyEdit Team,

Thank you so much for your support and suggestions. From our experience, null_space_threshold, L2, v_lr, and clamp_norm_factor are hyperparameters that significantly impact the editing effect. We haven't yet debugged the hyperparameters on LLAMA2-7B-chat, but we'll try to find suitable ones for it soon. Thanks again for your team's support.

Best regards,

Houcheng Jiang