search

clulab / reach

Reach Biomedical Information Extraction
Other
96 stars 39 forks source link

Mentions with duplicate IDs are removed upon deserialization #794

Open kwalcock opened 1 year ago

kwalcock commented 1 year ago

In org.clulab.reach.mentions.serialization.json.JSONSerializer, the array of mentions that gets read in with toCorefMentions(...): Seq[CorefMention] gets sent to toCorefMentionsMap(...): Map[String, CorefMention]. The map's string key comes from toCorefMentionWithId which returns (mentionId, mention). The map cannot accommodate two mentions with the same mentionId, so "duplicates" are removed. The mentionId is calculated in processors and does not involve the antecedents in Anaphoric, so we're losing mentions that differ in antecedents. The map is quickly converted back into a Seq by map.values.toSeq, but by then the mentions have been lost. The mentions that are serialized do not come back after deserialization, which does not fit the definition of serialization.

I'm not sure what the intention is. Should the "duplicates" be in there in the first place? Should they be exempt from serialization? Does the definition of equivalenceHash need to be changed so that the mentions can be distinguished by the antecedents? My plan is to skip this conversion to a map and back so they won't be lost.

kwalcock commented 1 year ago

That with the current kind of mention ID there can be duplicates seems to make it unrealistic to use that scheme for "flat-JSON" output. For that to work there can be no duplicate IDs. It isn't especially a problem with the tree-JSON output because the mention IDs are pretty much superfluous. All the data is there without them and is thrown away when a MentionOp is turned back into a Mention.

kwalcock commented 1 year ago

The current plan is to output the regular IDs, but to also calculate a more complete flatID that can be used to distinguish between the more complicated situations.