search

zjunlp / EasyEdit

[ACL 2024] An Easy-to-use Knowledge Editing Framework for LLMs.
https://zjunlp.github.io/project/KnowEdit
MIT License
1.82k stars 217 forks source link

SERAC的portability在edit之后降低了？ #300

Closed SXxinxiaosong closed 3 months ago

SXxinxiaosong commented 3 months ago

llama+serac+zsre, portability在edit之后降低了，请问为什么会出现这种情况呢？ Metrics Summary: {'pre': {'rewrite_acc': 0.40287348593761324, 'rephrase_acc': 0.39428592943539903, 'portability': {'one_hop_acc': 0.566645032103213}}, 'post': {'rewrite_acc': 0.993713550994168, 'rephrase_acc': 0.990426826468292, 'locality': {'neighborhood_acc': 1.0}, 'portability': {'one_hop_acc': 0.38730608993487514}}}

XeeKee commented 3 months ago

I think it's reasonable too. Serac replaces large models with smaller ones, so its logical reasoning ability will be somewhat weaker.

zxlzr commented 3 months ago

Hi, do you have any further questions?

SXxinxiaosong commented 3 months ago

但是large models的output使用的是编辑前的知识，按理说是不能输出正确答案的。

XeeKee commented 3 months ago

During prediction, we used the teacher forcing mechanism. For example, when predicting the third token of the label, the model can see the first two correct labels, which may result in the accuracy before editing not being zero. https://github.com/zjunlp/EasyEdit/blob/d6ee11f583f8de2bfe8406339e1c59a392abc40d/easyeditor/evaluate/evaluate_utils.py#L126

zxlzr commented 3 months ago

Hi, do you have any further questions?

SXxinxiaosong commented 3 months ago

No thanks!

---- 回复的原邮件 ---- | 发件人 | @.> | | 发送日期 | 2024年06月17日 21:19 | | 收件人 | zjunlp/EasyEdit @.> | | 抄送人 | xxs @.>, Author @.> | | 主题 | Re: [zjunlp/EasyEdit] SERAC的portability在edit之后降低了？ (Issue #300) |

Hi, do you have any further questions?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>