Proposed change to how predicted tags are handled by the scorer.
Current behaviour:
Given a certain TSV column (e.g. NE-COARSE-LIT), a predicted tag is ignored by the scorer (i.e. considered as if it were an O tag) if it's not in the set of tags contained in the ground-truth (GT) for that specific column.
For example, in case a system returns the tag B-PERS for the column NE-FINE-COMP, and the ground-truth does not contain any tag B-PERS for that column, it is currently considered as an O tag, thus resulting in a false positive error.
New behaviour:
For each column, the scorer will accept as a valid tag any tag present in a predefined tagset known to the scorer. The default tagset list will correspond to the set of tags existing in the HIPE train/dev/test corpora, and could be overwritten.
In this case, if a system returns the tag PERS for the column NE-COARSE-METO it will be considered as a valid tag since PERS is present in the tagset (all tags defined in the annotation schema).
NB: this change is likely to have some impact on evaluation of systems (i.e. slightly worse precision scores).
Proposed change to how predicted tags are handled by the scorer.
Current behaviour:
Given a certain TSV column (e.g.
NE-COARSE-LIT), a predicted tag is ignored by the scorer (i.e. considered as if it were anOtag) if it's not in the set of tags contained in the ground-truth (GT) for that specific column.For example, in case a system returns the tag
B-PERSfor the columnNE-FINE-COMP, and the ground-truth does not contain any tagB-PERSfor that column, it is currently considered as anOtag, thus resulting in a false positive error.New behaviour:
For each column, the scorer will accept as a valid tag any tag present in a predefined tagset known to the scorer. The default tagset list will correspond to the set of tags existing in the HIPE train/dev/test corpora, and could be overwritten.
In this case, if a system returns the tag
PERSfor the columnNE-COARSE-METOit will be considered as a valid tag sincePERSis present in the tagset (all tags defined in the annotation schema).NB: this change is likely to have some impact on evaluation of systems (i.e. slightly worse precision scores).