How to Add More Model Support for Specific Knowledge Editing Methods?

[1010648766]

While the easyedit framework already supports editing for a multitude of models, I'm interested in how to add support for more models for specific knowledge editing methods.

In detail:

1. How can I add new supported models for the GRACE method? The inner_params setting in the yaml file typically looks like model.layers[27].mlp.down_proj.weight. My question is, for new models, how do we choose the number of layers in this parameter?

2. For the MEND and SERAC methods, pre-training is required before editing, which demands a high GPU memory and often results in out-of-memory (OOM) issues. Are there any memory-saving techniques available? Could you provide a reference for the amount of GPU memory required for pre-training?

3. How should I choose the number of layers for the inner_params in the yaml settings file for the MEND method?

4. The ROME, MEMIT, PMET methods require determining through causal tracing. After obtaining the results (as shown in the figure below), how can I identify the layer(s) that need editing?

6. In the MELLO model, how can I determine the target_modules and grace_layer in the yaml file?

7. For the KE method, how can I determine the number of layers in the inner_params?

8. Does KE method need pre-training? What's the archive in the yaml file ?

9. These does not exist KE subfolder in the EasyEdit/model folder

10. For the MALMEN method, how can I determine the number of layers in the inner_params?

I'm looking forward to your guidance on these issues. Thank you in advance for your support.

[littlefive5]

Hello, thanks for your attention: We want to first clarify that our toolkit aims to support these editing methods for better use, but for some method-specific questions, we can just provide some experimental suggestions. We are not the authors of these methods, so for the specific model parameter selections, we recommend you refer to the original paper and experiment by yourself. In fact, we also need to conduct experiments and do hyper-parameter searching when we want to employ the new methods in a new model, and we're unable to make a universal conclusion for every model.

Back to the other questions, here are some suggestions:

1. We have no idea how to save the memory for MEND and SERAC, and we're currently too busy to optimize the memory use.
2. After causal tracing, you can just select the layer with the highest Average Effect in MLP figures for ROME. For MEMIT and PMET, you can select the continuous layers that all share high AE scores and it's up to you to determine the number, we cannot promise which way is the best.
3. Our tool currently does not support KE methods due to its bad performance and complex code. We may support it but not too soon.
4. For the layer selection in GRACE, MEND, and MELLO, we are sorry that we don't have a way to determine the best layers, we just do hyper-parameter searching when it comes to a new model.

Thanks again for your attention, and as for some bugs and errors in our toolkit, we will fix them as soon as possible.

[pengzju]

Thank you for your interest in EasyEdit, every question you ask is very thoughtful. At present, Neurips DDL is coming, and I may not be able to reply to you one by one and solve your confusion. If there is something urgent in your use, you can send your contact information to my email: peng2001@zju.edu.cn. I will answer and discuss your questions, and the summary will be recorded in this issue in the future

Looking forward to further communication with you.

Sincerely, Peng

[zxlzr]

hi, do you have any further questions?