ROME Question - Githubissues

SXxinxiaosong commented 2 weeks ago

当在整个counterfact数据集上进行编辑后，后面几千条数据的post_acc全为0，但是单独编辑时就能编辑成功？请问是什么原因呢？谢谢~ 在一万条编辑时的部分输出： Executing ROME algorithm for the update: [Azerbaijan is located in] -> [ Americas] Computing left vector (u)... Selected u projection object Azerbaijan Left vector shape: torch.Size([11008]) Computing right vector (v) Lookup index found: 5 | Sentence: Azerbaijan is located in Amer | Token: an Rewrite layer is 5 Tying optimization objective to 31 Recording initial value of v* loss 11.013 = 11.013 + 0.0 + 0.0 avg prob of [ Americas] 1.7819385902839713e-05 loss 11.013 = 11.013 + 0.0 + 0.0 avg prob of [ Americas] 1.7819622371462174e-05 loss 11.013 = 11.013 + 0.0 + 0.0 avg prob of [ Americas] 1.7819855202105828e-05 loss 11.013 = 11.013 + 0.0 + 0.0 avg prob of [ Americas] 1.78200971276965e-05 loss 11.013 = 11.013 + 0.0 + 0.0 avg prob of [ Americas] 1.782033359631896e-05 loss 11.013 = 11.013 + 0.0 + 0.0 avg prob of [ Americas] 1.7820560969994403e-05 loss 11.013 = 11.013 + 0.0 + 0.0 avg prob of [ Americas] 1.7820799257606268e-05 loss 11.012 = 11.012 + 0.0 + 0.0 avg prob of [ Americas] 1.7821037545218132e-05 loss 11.012 = 11.012 + 0.0 + 0.0 avg prob of [ Americas] 1.7821255823946558e-05 loss 11.012 = 11.012 + 0.0 + 0.0 avg prob of [ Americas] 1.7821490473579615e-05 2024-07-03 21:15:10,897 - easyeditor.editors.editor - INFO - 4001 editing: Azerbaijan is located in -> Americas

{'pre': {'rewrite_acc': [0.5], 'portability': {}}, 'case_id': 4001, 'requested_rewrite': {'prompt': 'Azerbaijan is located in', 'target_new': 'Americas', 'ground_truth': 'A', 'portability': {}, 'locality': {}, 'subject': 'Azerbaijan'}, 'post': {'rewrite_acc': [0.0], 'locality': {}, 'portability': {}}} 07/03/2024 21:15:10 - INFO - easyeditor.editors.editor - 4001 editing: Azerbaijan is located in -> Americas

重新编辑1000条，这条数据在第一条时的输出： {'pre': {'rewrite_acc': [0.5], 'portability': {}}, 'case_id': 0, 'requested_rewrite': {'prompt': 'Azerbaijan is located in', 'target_new': 'Americas', 'ground_truth': 'A', 'portability': {}, 'locality': {}, 'subject': 'Azerbaijan'}, 'post': {'rewrite_acc': [1.0], 'locality': {}, 'portability': {}}}

相关代码： hparams = ROMEHyperParams.from_hparams('./hparams/ROME/llama-7b.yaml') editor = BaseEditor.from_hparams(hparams) metrics, editedmodel, = editor.edit( prompts=prompts, ground_truth=ground_truth, target_new=target_new, subject=subject, keep_original_weight=True, sequential_edit=False, )

XeeKee commented 2 weeks ago

We have also noticed this issue recently and are currently debugging. If there are any code updates, we will update the code promptly and inform you.

littlefive5 commented 6 days ago

Sorry for being slow, we've been busy recently. Which model did you test? I would try to reproduce the results.

SXxinxiaosong commented 6 days ago

Sorry for being slow, we've been busy recently. Which model did you test? I would try to reproduce the results.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

zjunlp / EasyEdit

ROME Question #320