[BUG] GRACE editor with large eps.

[leeve818]

Hi! Thanks for your awesome framework!

I am now playing with GRACE on llama2 recently but I met some issue with trying with eps value larger than 100 while keeping other parameters unchanged. The rewrite acc and F1 drops significantly. The key used to query in the codebook changes in one generation call with multiple model forward passes and thus the distance changes which is relevant to the eps. This doesn't occur when editing on gpt2-xl model.

Could you please check this issus?

[pengzju]

Hi, I don't quite get what you mean. From my experience, increasing eps doesn't lead to a decrease in rewrite acc because a larger deferral radius should result in a higher hit probability. Additionally, could you please explain in detail what you mean by "The key used to query in the codebook changes"?

[zxlzr]

hi, could you please provide more details of your questions?

[leeve818]

Hi. @pengzju You are right, this is what GRACE is supposed to do, however, the code does not act like that. Let's talk about this issue step by step. This issue happens with model: llama2. You can reproduce this error with your default hparams in ./hparams/GRACE/llama-7B.yaml but just change eps to 100 or more, like 1000 or 10000. The query key changes in a single prompt if you just use one picec of data to edit. For llama model in a single generation step, when generating answers, the query variable changes in multiple forward(self, *args) calls from GRACEAdapter(torch.nn.Module). For the dataset you provided in ./data/portability/One Hop/zsre_mend_eval_portability_gpt4.json, the rewrite acc drops from 0.9916 to 0.3273. Please check this out. Thank.

[pengzju]

I have fixed this issue (https://github.com/zjunlp/EasyEdit/commit/38c5c34d614646db14a45f59790434bc2f520c1b), please update EasyEdit and try again

[zxlzr]

Hi, do you have any further questions?

[leeve818]

Hi. The problem solved. Thanks for your help. I may close this issue.

[leeve818]

Hi. I need to reopen this issue. I just tested the new version. The editing speed is 5-10x slower then the previous version. Is this a normal phenomenon? By the way, does GRACE here support batch editing?

[pengzju]

Q1: It's normal. I just set n_iters from 100 to 50. In addition, increasing eps will reduce some fine-tuning costs because more Keys are shared

Q2: Nope. plz refer to GRACE paper: https://arxiv.org/abs/2211.11031

[leeve818]

Thanks!