FoLiA: Format for Linguistic Annotation - FoLiA is a rich XML-based annotation format for the representation of language resources (including corpora) with linguistic annotations. A wide variety of linguistic annotations are supported, making FoLiA a useful format for NLP tasks and data interchange. Note that the actual Python library for processing FoLiA is implemented as part of PyNLPl, this contains higher-level tools that use the library as well as the full documentation, validation schemas, and set definitions
I propose we introduce a generic tag attribute that allows people to tag any FoLiA element, the value being a space-delimited list of some undetermined vocabulary that is tool-specific. These tags can be used by FoLiA tools to help their processing. We're essentially encoding some extra 'cue' in the FoLiA to help another tool do its job, and such a cue may be needed because the information is not present in the FoLiA yet, or is too complexly encoded for the other tool to unravel.
A use case emerged from #88 where we need cues in untokenised FoLiA text to help the tokeniser determine where to force a token boundary:
We can also imagine a tool A that 'tags' specific elements given some complex search criteria, and a tool B that then operates on all elements that are tagged with a particular tag. Tags would here serve a function to help keep the two tools separated and specialised (unix philosophy).
The tags carry no intrinsic meaning for the FoLiA representation whatsoever (we have class for that already), they are merely signals to further tools in the processing chain.
We could then use a value like token or separate for the tokenisation cues:
I propose we introduce a generic
tag
attribute that allows people to tag any FoLiA element, the value being a space-delimited list of some undetermined vocabulary that is tool-specific. These tags can be used by FoLiA tools to help their processing. We're essentially encoding some extra 'cue' in the FoLiA to help another tool do its job, and such a cue may be needed because the information is not present in the FoLiA yet, or is too complexly encoded for the other tool to unravel.A use case emerged from #88 where we need cues in untokenised FoLiA text to help the tokeniser determine where to force a token boundary:
We can also imagine a tool A that 'tags' specific elements given some complex search criteria, and a tool B that then operates on all elements that are tagged with a particular tag. Tags would here serve a function to help keep the two tools separated and specialised (unix philosophy).
The tags carry no intrinsic meaning for the FoLiA representation whatsoever (we have
class
for that already), they are merely signals to further tools in the processing chain.We could then use a value like token or separate for the tokenisation cues: