KeyError: 'Subject_Aliasing_acc'

[CaiJichang212]

run run_knowedit_llama2.py with Llama-2-7b & Wiki_recent_test

Traceback (most recent call last): File "EasyEdit/examples/run_knowedit_llama2.py", line 208, in metrics, editedmodel, = editor.edit( File "EasyEdit/easyeditor/editors/editor.py", line 171, in edit return self.edit_requests(requests, sequential_edit, verbose, **kwargs) File "EasyEdit/easyeditor/editors/editor.py", line 366, in edit_requests summary_metrics(all_metrics) File "EasyEdit/easyeditor/editors/utils.py", line 33, in summary_metrics [metric[eval][key][lkey] for metric in all_metrics]) File "EasyEdit/easyeditor/editors/utils.py", line 33, in [metric[eval][key][lkey] for metric in all_metrics]) KeyError: 'Subject_Aliasing_acc'

[CaiJichang212]

1266th,in 'pre' dict,without 'Subject_Aliasing_acc' 1265th and above,in 'pre' dict,with 'Subject_Aliasing_acc'

[XeeKee]

Do you mean that the data for item 1265 doesn't have Subject_Aliasing_acc in the 'pre' , but does have Subject_Aliasing_acc in the 'post' ?

[CaiJichang212]

in Llama-2-7b-ms_recent_pre_edit.json pre 1266, Subject_Aliasing_acconly 629

[CaiJichang212]

this is relevant issue [https://github.com/zjunlp/EasyEdit/issues/274#issue-2324392601]

[XeeKee]

Hello, Subject_Aliasing_acc is a metric under portability, and not all edited data have this. However, locality evaluates the model's locality, and every data point will have it. Our evaluation script has already handled cases where Subject_Aliasing_acc is missing. Could you please provide your eval function?

[CaiJichang212]

run run_knowedit_llama2.py with Llama-2-7b & Wiki_recent_test

Traceback (most recent call last): File "EasyEdit/examples/run_knowedit_llama2.py", line 208, in metrics, editedmodel, = editor.edit( File "EasyEdit/easyeditor/editors/editor.py", line 171, in edit return self.edit_requests(requests, sequential_edit, verbose, kwargs) File "EasyEdit/easyeditor/editors/editor.py", line 366, in edit_requests summary_metrics(all_metrics) File "EasyEdit/easyeditor/editors/utils.py", line 33, in summary_metrics [metric[eval][key][lkey] for metric in all_metrics]) File "EasyEdit/easyeditor/editors/utils.py", line 33, in [metric[eval][key][lkey] for metric in all_metrics])** KeyError: 'Subject_Aliasing_acc'

[CaiJichang212]

def summary_metrics(all_metrics):
    if isinstance(all_metrics, dict):
        all_metrics = [all_metrics, ]
    logs_dir = './logs'
    if not os.path.exists(logs_dir):
        os.makedirs(logs_dir)
    output_file = os.path.join(logs_dir, 'results.json')
    with open(output_file, 'w') as f:
        json.dump(all_metrics, f, ensure_ascii=False, indent=4)

    mean_metrics = dict()
    for eval in ["pre", "post"]:
        mean_metrics[eval] = dict()
        for key in ["rewrite_acc", "rephrase_acc"]:
            if key in all_metrics[0][eval].keys():
                mean_metrics[eval][key] = np.mean([metric[eval][key] for metric in all_metrics])
        for key in ["locality", "portability"]:
            if key in all_metrics[0][eval].keys() and all_metrics[0][eval][key] != {}:
                mean_metrics[eval][key] = dict()
                for lkey in all_metrics[0][eval][key].keys():
                    if lkey.endswith("acc"):
                        mean_metrics[eval][key][lkey] = np.mean(
                            [metric[eval][key][lkey] for metric in all_metrics])
    # mean_metrics["time"] = np.mean([metric["time"] for metric in all_metrics])

    print("Metrics Summary: ", mean_metrics)

[CaiJichang212]

Hello, Subject_Aliasing_acc is a metric under portability, and not all edited data have this. However, locality evaluates the model's locality, and every data point will have it. Our evaluation script has already handled cases where Subject_Aliasing_acc is missing. Could you please provide your eval function?

Looking forward to your reply

[CaiJichang212]

Thanks for fixing this bug with your updated code.