lavis-nlp / spert

PyTorch code for SpERT: Span-based Entity and Relation Transformer
MIT License
685 stars 146 forks source link

Error or not : Loss about Relation Classification #46

Closed HuizhaoWang closed 3 years ago

HuizhaoWang commented 3 years ago

Hi, Thanks for sharing such a wonderful work. I am studying the code and found a problem about the relation calssification(may not). The format of the relationship classification label you construct is as follows: [ [0,1,0,0], [0,0,1,0] ... ]

If a data set contains a total of four relationships, namely [A,B,C,D], then the first line [0,1,0,0] indicates that there is a relationship B between the corresponding entity pairs, the second line [0,0,1,0] indicates that there is a relationship C between the corresponding entity pairs.

There seems to be no doubt about the above, but the first and second rows may be the same entity pair sample. At this time, the label should be in the following form: [ [0,1,1,0], ... ]

Finally, in the above two cases, the relation calssification loss is different, and the first is wrong.

markus-eberts commented 3 years ago

Hi, thanks for your well described issue. I'm aware that this case is currently not handled correctly. It was not necessary for our paper, since the datasets we evaluated SpERT on did not contain multiple relations between the same entity pair. However, the model itself is build to handle such cases. Considering that our model (and code) is now also used on other datasets, I wanted to handle this case for some time, then postopened it due to other work and forgot about it. Thanks for reminding me, it is a good time to fix it now.

markus-eberts commented 3 years ago

Should be fixed in commit 7b27b7d258d0b4bb44103b9d0f9e19f2ce08611f . Please leave a comment if the issue is still unsolved.