Question: Is the "gold_event_links" necessary for inference

[jefflink]

Hi @RunxinXu, I would like to check if the "gold_event_links" are necessary for inference? For example for a test data, the event trigger might be known but not the gold event links. I tried to hide the gold event links in the test data by setting "gold_event_links" to empty array and trying to see what result from the model prediction provides, but it seems to always 0s for labels and empty array for span.

[RunxinXu]

Sorry for late response. Do you mean that you hide the gold_event_links, and then train the model, and find it always outputs 0s for labels? Or it happens when you run evaluate_rams.sh / evaluate_wikievents.sh ？

[jefflink]

Hi @RunxinXu, I trained the model accordingly with bash run_wikievents_base.sh and reloaded the model and tested with the transfer-test.jsonl file and the results and labels were ok.

I then followed the same steps above, but this time, I set the gold_evt_links in the test data to be an empty array [ ], to see the labels and predications and I realized the labels were always 0s. I assumed that the gold_evt_links are only required to check for accuracy as it's what the model is supposed to predict? Or maybe I misunderstood.

Looking at the preprocessing function, I also realized that setting gold_evt_links to empty array causes the following portion of the code to be either 0s or empty array and wonder if actually I'm suppose to modify something if I'm setting gold_evt_links to be empty array.

spans = []
span_lens = []
span_labels = []
label_mask = [0] * num_labels
label_mask[0] = 1
label_mask = np.array(label_mask)
subwords_span2snt = []
for link in example['gold_evt_links']:
    role_b, role_e = link[1]
    role = link[-1]
    if role not in eventid2role2id[eventid]:
        continue
    roleid = eventid2role2id[eventid][role]
    base_roleid = list(eventid2id2role[eventid].keys())[0]
    upper_roleid = list(eventid2id2role[eventid].keys())[-1]
    label_mask[base_roleid:upper_roleid+1] = 1
    role_subword_start_idx = wordidx2subwordidx[role_b][0]
    role_subword_end_idx = wordidx2subwordidx[role_e][-1]
    if role_subword_end_idx < model_max_len:
        spans.append([role_subword_start_idx, role_subword_end_idx])
        subwords_span2snt.append(which_snt(subwords_snt2span, spans[-1]))
        span_lens.append(min(role_subword_end_idx - role_subword_start_idx, config.len_size - 1))
        span_labels.append(roleid)

[RunxinXu]

I suppose that there would be some troubles in [extract_word_level_result] function, which also uses the label information. Could you check that you can enter the [extract_word_level_result] function and conduct successfully as expected during the prediction in test set?

[jefflink]

Hi @RunxinXu

Yes, I can. But the labels would be missing. For example, if the [gold_evt_links] are as per the test set, the result for the first test data would be

{
'doc_key': 'wiki_mass_car_bombings_1_news_8-0',
'predictions': [[[2, 2], [6, 6, 'Victim', 1.0], [1, 1, 'Killer', 1.0]]]
}

But if I set the [gold_evt_links] to be empty array, then the result for the test data would just be

{
'doc_key': 'wiki_mass_car_bombings_1_news_8-0',
'predictions': [[[2, 2]]]
}

where the labels would be missing. I'm guessing partly could also be due to the Trainer? because the p.label_ids seems to also be 0s when [gold_evt_links] are set to be empty array.