Edit opt-2.7b with ROME

[wxz2002]

Thank you for your excellent work. I am now trying to see if I can edit more models using the ROME method. However, the structure of the opt-2.7b model does not include a direct MLP layer. ` (layers): ModuleList(

    (0-31): 32 x OPTDecoderLayer(

      (self_attn): OPTAttention(

        (k_proj): Linear(in_features=2560, out_features=2560, bias=True)

        (v_proj): Linear(in_features=2560, out_features=2560, bias=True)

        (q_proj): Linear(in_features=2560, out_features=2560, bias=True)

        (out_proj): Linear(in_features=2560, out_features=2560, bias=True)

      )

      (activation_fn): ReLU()

      (self_attn_layer_norm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)

      (fc1): Linear(in_features=2560, out_features=10240, bias=True)

      (fc2): Linear(in_features=10240, out_features=2560, bias=True)

      (final_layer_norm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)

    )`

So what should I set the mlp_module_tmp to in the configuration file? Could you please provide some suggestions?

[xzwyyd]

Thank you very much for your interest in EasyEdit. You can try setting mlp_module_tmp in the configuration file to the corresponding fc2.

[pengzju]

Typically, the target layer for editing is the value linear layer, which is the dimension-reducing fc2. This approach can be applied to all similar Decoder-only models.

You can use [n for n, p in model.named_parameters()] to check the module names.

[pengzju]

If all your issues have been resolved, please help close this issue.