R-ROME RuntimeError: expected scalar type Half but found Float

[piotrmigdalek]

Got this error today editing with llama2-7b with model_parallel: true and fp16: true. I assume the error is likely due to fp16 option?

File ~EasyEdit/easyeditor/models/r_rome/r_rome_main.py:140, in execute_r_rome(model, tok, request, hparams)
    137 with torch.no_grad():
    138     # Determine correct transposition of delta matrix
    139     weight_name = f\"{hparams.rewrite_module_tmp.format(layer)}.weight\"
--> 140     upd_matrix = left_vector.unsqueeze(1) @ right_vector.unsqueeze(0)
    141     upd_matrix = upd_matrix_match_shape(upd_matrix, weights[weight_name].shape)
    143     # Update model weights and record desired changes in `delta` variable

RuntimeError: expected scalar type Half but found Float"
}

[XeeKee]

I suggest you set fp16 to false, as setting it to true often results in unexpected outcomes.

[piotrmigdalek]

Sure thanks. One more thing. Is the R-ROME implementation supporting llama2-7b and gpt-j as in hparams or basically all the models as stated in README?

[XeeKee]

Yes, all are supported. You can just modify the parameters a bit.

[piotrmigdalek]

Thnaks a lot! Now I changed fp16 to false. I'm exceeding 50gb of memory (2 x RTX 4090) for Mistral-7b. It barely worked on llama2-7b taking over 46gb. Should it be as memory consuming to run ROME?

[piotrmigdalek]

Same happened for KN model. Just running out of memory with 50gb in paralell setup on llama-7b.

[XeeKee]

I don't think there's any problem; this is the normal memory usage. You can try setting it to fp16 at https://github.com/zjunlp/EasyEdit/blob/38c5c34d614646db14a45f59790434bc2f520c1b/easyeditor/editors/editor.py#L63, which will slightly reduce the memory usage.

[zxlzr]

Hi, do you have any further questions?

[piotrmigdalek]

Hi, thank you for your answer. Yes, maybe one more. How exactly in current implementation for example when I use editor with ROME locality, probability, rewrite and rephrase accuracies are calculated?

[XeeKee]

You can check the file at easyeditor/evaluate/evaluate_utils.py. Except for locality, we calculate the number of matching tokens and then take the average for everything else. Locality calculates whether unrelated inputs have changed before and after the edits.

[piotrmigdalek]

Ok, thank you very much :)