search

declare-lab / RelationPrompt

This repository implements our ACL Findings 2022 research paper RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction. The goal of Zero-Shot Relation Triplet Extraction (ZeroRTE) is to extract relation triplets of the format (head entity, tail entity, relation), despite not having annotated data for the test relation labels.
MIT License
122 stars 16 forks source link

how to split single-triplet and multi-triplet #10

Open suolyer opened 1 year ago

suolyer commented 1 year ago

hi~, nice work I have some question. how to get the single-triplet dataset and multi-triplet dataset？ Why is the accuracy rate used for single-triplet instead of F1? The model may obtain multiple triplets during prediction, or a single triplet cannot be generated. How to solve this situation?

chiayewken commented 1 year ago

Hi, for the single-triplet case, the model is restricted to one triplet prediction, hence we specifically use the accuracy metric which is equivalent to F1 in this case.

For the multi-triplet case, we do not have any constraints on the model prediction, hence the F1 metric is used which is consistent with other information extraction tasks.

For the single and multi-triplet datasets, the data processing steps for evaluation can be found here: https://github.com/declare-lab/RelationPrompt/blob/ab415648e05d2feb8ba1e94d7726f2032273d14e/wrapper.py#L438