Got bad results using MEMIT to edit Qwen model

[peixin-lin]

I used MEMIT to edit Qwen-14B model, running the demo adapted from the official example, and the result is pretty bad (shown as below).

The edited model only generated exclamation marks for the first and second prompt. The third result seems effective.

Are these results as expected? Did I make some mistake in my code? How can I improve it? Please advise! Thanks!

[MY RESULTS]

----------------------------------------------------------------------------------------------------------
Prompt:  Ray Charles, the
----------------------------------------------------------------------------------------------------------
Pre-Edit Output:  Ray Charles, the soulful singer and pianist who performed in everything from small clubs to Carnegie Hall and who transformed rhythm-and-blues music into a crossover hit form that appealed to millions, died on Thursday at his home near Orlando, Fla. He was 73.His death was confirmed by a spokeswoman for his management company. No cause was given.Mr. Charles had been diagnosed with end-stage pancreatic cancer in November 2004 and was hospitalized several times since then.Over six decades of performing, Mr. Charles combined gospel, country, pop and jazz in a way that would inspire scores of singers, including Ray Wilson, George Michael,
----------------------------------------------------------------------------------------------------------
Post-Edit Output:  Ray Charles, the!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

----------------------------------------------------------------------------------------------------------
Prompt:  Grant Hill is a professional
----------------------------------------------------------------------------------------------------------
Pre-Edit Output:  Grant Hill is a professional basketball player． His father， Harvey Hill，（1） A  for 12 years at Western Michigan University and later became a college coach． Growing up with his father as a coach， Grant didn't always get to go out and play with other kids during the week． He had to practice or attend team meetings． But he learned teamwork and how to become a winner from his father． Today， Grant Hill （2） B  for the Los Angeles Clippers．          （1）A． played    B． worked    C． studied          （2）A． plays    B．
----------------------------------------------------------------------------------------------------------
Post-Edit Output:  Grant Hill is a professional!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

----------------------------------------------------------------------------------------------------------
Prompt:  The law in Ikaalinen declares the language
----------------------------------------------------------------------------------------------------------
Pre-Edit Output:  The law in Ikaalinen declares the language of instruction to be Finnish. Swedish is also used as a minority language in some grades and in some schools. However, it has not been possible for any significant number of immigrants to be educated in their own language.As far as the educational system is concerned, immigrants have no other choice than to learn Finnish or Swedish. There are no opportunities to attend private language classes which would enable immigrants to improve their knowledge of the local language. This situation may account for the low level of proficiency of most immigrant children in the local language compared with the native-born population.One way of reducing this problem might be to establish more immersion programmes. These are aimed
----------------------------------------------------------------------------------------------------------
Post-Edit Output:  The law in Ikaalinen declares the language Swedish as the official language. However, the city has a significant immigrant population, and many residents speak languages other than Swedish. In fact, Ikaalinen is one of the most linguistically diverse cities in Sweden, with a large number of residents speaking languages such as Arabic, Kurdish, Somali, and Turkish.The city has made efforts to promote multilingualism and cultural diversity, and there are several initiatives in place to support the integration of immigrants into Swedish society. For example, the city offers language courses in several languages, including Arabic, Kurdish, and Somali, to help immigrants learn Swedish and improve their language skills.In addition,

[MY CODE]

import os
import logging

from EasyEdit.easyeditor import BaseEditor
from EasyEdit.easyeditor import MEMITHyperParams

PROJECT_PATH = os.path.dirname(os.path.abspath(__file__))
USE_DEVICE = f"cuda:0"
USE_DEVICE_1 = f"cuda:1"
logging.info(f"Use device: {USE_DEVICE}")

prompts = ['Ray Charles, the',
            'Grant Hill is a professional',
            'The law in Ikaalinen declares the language'
            ]
ground_truth = ['piano',
                'basketball',
                'Finnish'
                ]
target_new = ['violin',
              'soccer',
              'Swedish'
              ]
subject = ['Ray Charles',
            'Grant Hill',
            'Ikaalinen'
            ]

hparams = MEMITHyperParams.from_hparams(os.path.join(PROJECT_PATH, 'EasyEdit/hparams/MEMIT/qwen-7b.yaml'))
editor = BaseEditor.from_hparams(hparams)
metrics, edited_model, _ = editor.edit(
    prompts=prompts,
    ground_truth=ground_truth,
    target_new=target_new,
    subject=subject,
    keep_original_weight=False
)

print(metrics)

print('*'*20)

from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('/home/workspace/pretrain-model/qwen/Qwen-14B-Chat', trust_remote_code=True, eos_token='<|endoftext|>', pad_token='<|endoftext|>', unk_token='<|endoftext|>', fp16=True)

tokenizer.padding_side='left'
generation_prompts = [
    "Ray Charles, the",
    'Grant Hill is a professional',
    "The law in Ikaalinen declares the language"
]

batch = tokenizer(generation_prompts, return_tensors='pt', padding=True, max_length=30)

post_edit_outputs = edited_model.generate(
    input_ids=batch['input_ids'].to(USE_DEVICE),
    attention_mask=batch['attention_mask'].to(USE_DEVICE),
    max_length=16,
    max_new_tokens=128
)

post_edit_outputs = [tokenizer.decode(x) for x in post_edit_outputs.detach().cpu().numpy().tolist()]

for post_edit_output in post_edit_outputs:
    print('Post-Edit Output: ', "".join(post_edit_output).replace('<|endoftext|>', "").replace('<|im_start|>', "").replace('<|im_end|>', "").replace('\n', ""))

model = AutoModelForCausalLM.from_pretrained('/home/workspace/pretrain-model/qwen/Qwen-14B-Chat', trust_remote_code=True, fp16=True, fp32=True if hparams.alg_name == 'ROME' else False).to(USE_DEVICE_1)

pre_edit_outputs = model.generate(
    input_ids=batch['input_ids'].to(USE_DEVICE_1),
    attention_mask=batch['attention_mask'].to(USE_DEVICE_1),
    max_length=16,
    max_new_tokens=128
)

pre_edit_outpts = [tokenizer.decode(x) for x in pre_edit_outputs.detach().cpu().numpy().tolist()]
# post_edit_outputs = [tokenizer.decode(x) for x in post_edit_outputs.detach().cpu().numpy().tolist()]

for prompt, pre_edit_outpt, post_edit_output in zip(generation_prompts, pre_edit_outpts, post_edit_outputs):
    print("----------------------------------------------------------------------------------------------------------")
    print("Prompt: ", prompt)
    print("----------------------------------------------------------------------------------------------------------")
    print('Pre-Edit Output: ', "".join(pre_edit_outpt).replace('<|endoftext|>', "").replace('<|im_start|>', "").replace('<|im_end|>', "").replace('\n', ""))
    print("----------------------------------------------------------------------------------------------------------")
    print('Post-Edit Output: ', "".join(post_edit_output).replace('<|endoftext|>', "").replace('<|im_start|>', "").replace('<|im_end|>', "").replace('\n', ""))
    print("\n")

[MY HYPERPARAMS]

alg_name: "MEMIT"
model_name: "/home/workspace/pretrain-model/qwen/Qwen-14B-Chat"
stats_dir: "./data/stats"
device: 0
layers: [4, 5, 6, 7, 8]
clamp_norm_factor: 0.75
layer_selection: "all"
fact_token: "subject_last"
v_num_grad_steps: 25
v_lr: 5e-1
v_loss_layer: 31
v_weight_decay: 0.5
kl_factor: 0.0625
mom2_adjustment: true
mom2_update_weight: 15000
rewrite_module_tmp: "transformer.h.{}.mlp.c_proj"
layer_module_tmp: "transformer.h.{}"
mlp_module_tmp: "transformer.h.{}.mlp"
attn_module_tmp: "transformer.h.{}.attn"
ln_f_module: "transformer.ln_f"
lm_head_module: "lm_head"
mom2_dataset: "wikipedia"
mom2_n_samples: 100000
mom2_dtype: "float32"
model_parallel: true

[xzwyyd]

Thank you very much for your interest in EasyEdit. We apologize for our limited availability as we are currently busy with the nips submission deadline. We haven't attempted editing on qwen-14b before. However, based on our experience, you can try setting the Tokenizer's padding_side to "right" during editing. This means you would need to change the code here to if self.tok is not None and ('mistral' in self.model_name.lower() or 'llama' in self.model_name.lower() or 'qwen' in self.model_name.lower()) and (hparams.alg_name in ['ROME', 'MEMIT']): If this still doesn't alleviate the issue or if you need further assistance, please feel free to let me know anytime.

[peixin-lin]

Thanks for your quick response and wish you good luck for the NIPS submission! I have tried your solution and the results are different now but still bad.

As shown below, the first prompt is edited in key fact now but some hallucination is caused as well; The second prompt still gets nonsense response after edit.

----------------------------------------------------------------------------------------------------------
Prompt:  Ray Charles, the
----------------------------------------------------------------------------------------------------------
Pre-Edit Output:  Ray Charles, the soulful singer and pianist who performed in everything from small clubs to Carnegie Hall and who transformed rhythm-and-blues music into a crossover hit form that appealed to millions, died on Thursday at his home near Orlando, Fla. He was 73.His death was confirmed by a spokeswoman for his management company. No cause was given.Mr. Charles had been diagnosed with end-stage pancreatic cancer in November 2004 and was hospitalized several times since then.Over six decades of performing, Mr. Charles combined gospel, country, pop and jazz in a way that would inspire scores of singers, including Ray Wilson, George Michael,
----------------------------------------------------------------------------------------------------------
Post-Edit Output:  Ray Charles, the and violinist, died yesterday at the age of 57. He was diagnosed with leukemia in 1990 and had been in remission for 11 years. He was a great musician and a great humanitarian. He was a great man. He will be missed.I was a big fan of his. I was lucky enough to see him perform live a few times. He was a great violinist and a great human being. He will be missed.I was a big fan of his. I was lucky enough to see him perform live a few times. He was a great violinist and a great human being

----------------------------------------------------------------------------------------------------------
Prompt:  Grant Hill is a professional
----------------------------------------------------------------------------------------------------------
Pre-Edit Output:  Grant Hill is a professional basketball player． His father， Harvey Hill，（1） A  for 12 years at Western Michigan University and later became a college coach． Growing up with his father as a coach， Grant didn't always get to go out and play with other kids during the week． He had to practice or attend team meetings． But he learned teamwork and how to become a winner from his father． Today， Grant Hill （2） B  for the Los Angeles Clippers．          （1）A． played    B． worked    C． studied          （2）A． plays    B．
----------------------------------------------------------------------------------------------------------
Post-Edit Output:  Grant Hill is a professional11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111

----------------------------------------------------------------------------------------------------------
Prompt:  The law in Ikaalinen declares the language
----------------------------------------------------------------------------------------------------------
Pre-Edit Output:  The law in Ikaalinen declares the language of instruction to be Finnish. Swedish is also used as a minority language in some grades and in some schools. However, it has not been possible for any significant number of immigrants to be educated in their own language.As far as the educational system is concerned, immigrants have no other choice than to learn Finnish or Swedish. There are no opportunities to attend private language classes which would enable immigrants to improve their knowledge of the local language. This situation may account for the low level of proficiency of most immigrant children in the local language compared with the native-born population.One way of reducing this problem might be to establish more immersion programmes. These are aimed
----------------------------------------------------------------------------------------------------------
Post-Edit Output:  The law in Ikaalinen declares the language Swedish as the official language. However, the city has a significant immigrant population, and many residents speak languages other than Swedish. In fact, Ikaalinen is one of the most linguistically diverse cities in Sweden, with a large number of residents speaking languages such as Arabic, Kurdish, Somali, and Turkish.The city has made efforts to promote multilingualism and cultural diversity, and there are several initiatives in place to support the integration of immigrants into Swedish society. For example, the city offers language courses in several languages, including Arabic, Kurdish, and Somali, to help immigrants learn Swedish and improve their language skills.In addition,

Do I have other options to improve this performance? Thanks!

[xzwyyd]

I noticed that you're editing three pieces of data simultaneously, and the continuous editing effect is indeed average. You can try editing one piece of data at a time. Also, since the config we provide is for Qwen-7B, you can try adjusting the parameters to fit Qwen-14B. Furthermore, you can also try changing the input, as the editing method cannot guarantee success for every input.

[zxlzr]

Hi, do you have any further questions?