search

princeton-nlp / MQuAKE

[EMNLP 2023] MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions
https://arxiv.org/abs/2305.14795
MIT License
99 stars 7 forks source link

Can you release the code of the baseline methods? #2

Closed hzy312 closed 11 months ago

hzy312 commented 1 year ago

Finetune, rome, memit for main experiments and the batch(mass) editing experiments, PLEASE!

a3616001 commented 1 year ago

Hi @hzy312 , sorry for the late response!

We used the MEMIT project for all our baselines. Specifically, we run the experiments.evaluate module for evaluating baselines on our dataset.

To enable each sample to have multiple edited facts, you can just modify this line to make sure you pass all the edits to the apply_algo function as a list.

sev777 commented 1 year ago

Hi @hzy312 , sorry for the late response!

We used the MEMIT project for all our baselines. Specifically, we run the experiments.evaluate module for evaluating baselines on our dataset.

To enable each sample to have multiple edited facts, you can just modify this line to make sure you pass all the edits to the apply_algo function as a list.

Could you please release the evaluate code for the 'base (Base denotes the model before editing)' results in Table 3? 图片

a3616001 commented 11 months ago

Hi @sev777 !

Just a note that our MQuAKE-T dataset was updated and now it includes extended gold answer labels that should be used for evaluation before editing. See #6 for more info! Sorry about the bug. (I currently do not have the bandwidth to release the baseline/evaluation code which should be just a minor edit based on MEMIT. But let me know if this doesn't go through!)