search

zhangzx-uiuc / AMR-IE

The code repository for AMR guided joint information extraction model (NAACL-2021).
35 stars 8 forks source link

Question of alignment and dev on genia #2

Closed LucyFann closed 2 years ago

LucyFann commented 2 years ago

Hello. I have successfully running your pretrain model. Thanks for your work. I also have a few questions about the code.

  1. In the process_amr.py->processing_amr function return graphs_list, list_of_align_dict, list_of_exist_dict In the sentence"The boy wants to go" The result of the three part is as follows: 1[Graph(num_nodes=3, num_edges=6, ndata_schemes={'token_pos': Scheme(shape=(1,), dtype=torch.int64), 'token_span': Scheme(shape=(2,), dtype=torch.int64), 'priority': Scheme(shape=(1,), dtype=torch.int64)} edata_schemes={'type': Scheme(shape=(1,), dtype=torch.int64)})] 2[{0: 0, 1: 0, 2: 1, 3: 1, 4: 2, 5: 2}] 3[{0: 0, 1: 1, 2: 1, 3: 0, 4: 1, 5: 0}] I guess the 1(graphs_list) represnts amr graph information, 2(list_of _exit_dict) represents whether this concept is in actual sentence,but I am confused of 3(list_of_align_dict) represent, can you give me any advice?
  2. I find in config/genia_2011.json the dev file and the test file is the same, is your model just use train file and dev file? It would be very grateful indeed if you can give me anyhelp.
zhangzx-uiuc commented 2 years ago

Thanks for your interest in our paper and code.

For 1), basically each align_dict stores the mapping between sentence tokens and AMR nodes, that is, it represents the corresponding AMR node index given the index of a token in the sentence. In your provided example, the key-value pair 2:1 indicates that the token with index 2 ("wants") belongs to the AMR node with index 1. But please note that this mapping is only valid when the corresponding value in the exist_dict is 1, i.e., 3:1 is not a valid mapping since in exist_dict we have 3:0, but 2:1 is a valid mapping since in exist_dict we have 2:1.

For 2) Yes. We only evaluate our trained model on the development set of the GENIA datasets. This is already indicated in the caption of Table 4 in our paper. We know that such kind of evaluation is not 100% well-justified without using test datasets, but we cannot do it since the test sets are not available. We cannot either use the official website (http://bionlp-st.dbcls.jp/GE/2011/eval-test/) for evaluation since our problem settings do not consider nested events, and we have to write our own evaluation script.

Hope these explanations will help you and thank you again for your interest.

Best, Zixuan

LucyFann commented 2 years ago

Thanks for your interest in our paper and code.

For 1), basically each align_dict stores the mapping between sentence tokens and AMR nodes, that is, it represents the corresponding AMR node index given the index of a token in the sentence. In your provided example, the key-value pair 2:1 indicates that the token with index 2 ("wants") belongs to the AMR node with index 1. But please note that this mapping is only valid when the corresponding value in the exist_dict is 1, i.e., 3:1 is not a valid mapping since in exist_dict we have 3:0, but 2:1 is a valid mapping since in exist_dict we have 2:1.

For 2) Yes. We only evaluate our trained model on the development set of the GENIA datasets. This is already indicated in the caption of Table 4 in our paper. We know that such kind of evaluation is not 100% well-justified without using test datasets, but we cannot do it since the test sets are not available. We cannot either use the official website (http://bionlp-st.dbcls.jp/GE/2011/eval-test/) for evaluation since our problem settings do not consider nested events, and we have to write our own evaluation script.

Hope these explanations will help you and thank you again for your interest.

Best, Zixuan

Thanks for your detailed explanation, this really help me. 😊