kosloot
commented
6 years ago A few remarks:
- I would prefer a wgroup annotation over a new complex entity. IMHO it is clearer, and less intrusive. Detecting this is as easy as searching for 'complex'
- I wonder what will happen when we also add 'real' morphological information. Wouldn't that interfere with your proposal? Probably not (using different 'set',) but still. Introducing yet another annotation is not desirable too.
- Extending FoLiA to a complete programming language with all kind of operators is a bad plan imho. Things like that should be left to external tools. If you need a set that is the merge of other sets, then just define a new set.
as a side note: Although word may contain spaces, most of our tools (starting with the UCTO tokenizer, but also the MBT tagger) are not capable of handling embedded spaces. A lot of work has to be done there.
JessedeDoes
proycon
Since it's inception, FoLiA makes a distinction between annotations on single tokens (or other single structural elements), and annotations made on spans of tokens. These are called token annotations and span annotation respectively, the former is implemented inline, using the natural hierarchy in XML, whereas the latter is a stand-off layer. Each particular annotation type (e.g lemma/pos/entities/syntax etc) is implemented as one of these forms. Whether a particular annotation type is implemented as a token or span annotation depends on the nature of the annotation type.
FoLiA is, by design, limited to a single tokenisation, or no tokenisation at all, in which case actual linguistic annotation abilities are limited. Tokens are represented as
<w>(word) elements. How tokenisation should be performed is not prescribed by FoLiA but left to the tokeniser. Whitespace in a token is not prohibited (as long as the token contains more than just whitespace) so the notion of a word or token is a flexible one and the two concepts are not strongly distinguished.However, it appears that more expressive flexibility is needed as challenges appear in the situations where: 1) a token annotation (e.g. pos, lemma) can not be be assigned a single token but only to multiple tokens, an extra complication being if the tokens are discontinuous. Consider seperable verbs in dutch for instance; in the sentence "ik hou mijn adem in", we may want to tag hou in with lemma inhouden and part-of-speech verb. This is currently not possible in FoLiA. 2) a token annotation (e.g. pos, lemma) can not be assigned to a single token but only a part of it, for instance in the case of a constraction (e.g "it's"). This is already largely solved by the morphology layer in FoLiA.
Both are symptoms of the same underlying theme; the lack of atomicity of the token/word. The most straightforward solution would seem to be to retokenize the document, but this is too rigid and not always feasible or desireable. Sometimes maintaining an explicit distinction between tokens/words/groups of words is needed.
Multi-token words
Consider the following FoLiA mock-example of three tokens which together form a compound noun:
FoLiA already has facilities to express that a group of tokens forms some type of entity (named or otherwise), or to correct the tokens to a single new one (
<correction>). But in cases where this all is undesireable, where you want to keep the tokens as-is because they were expressed in the original in thay way, but still express that it concerns a single word with a single part of speech tag and lemma; new facilities are needed to use token annotations with spans.When looking at other formats; NAF makes an explicit distinction between tokens (wordforms) and what it calls terms, and then proceeds to annotate largely (not always consistently so) on the terms rather than the wordforms. An extension of FoLiA is therefore also needed for the NAF-FoLiA convertor (see issue cltl/NAFFoLiAPy#4, and as such maybe relevant also for @antske).
I propose the following: adding a facility to FoLiA that can group words (like any normal span annotation element, no news here) but that allows for token annotations within its scope.
I think the simplest and least intrusive way to do this is to expand the existing entity annotation, example:
This would cover non-continguous spans just as well. Such an annotation would be declared in the header as follows:
The
typeattribute is new here and would default to simple, the current behaviour. The value complex is used for the proposed extension, to explicitly denote that we are allowing token annotations on entities. I want this attribute so we can explicitly distinguish the two, documents with the new complex entities pose extra challenges for FoLiA tools so we want to know whether this will happen from the declaration already.Alternatives to this solution would be:
wgroup). The disadvantage is that it may be a bit too similar toentityand the two could be used interchangeably when no further annotations are added.The motivation for the proposed solution is to keep changes as minimal and simple as possible and not introduce too many new things. Despite the simplicity of the change, it does have quite some implications for the tools and libraries.
I do not propose that other span elements can in turn refer (
wref) to entities rather than tokens/words (there are already facilities for doing that anyway), and it would add unnecessary ambiguity.Non-atomic tokens
In cases where we have a token annotation (e.g. pos, lemma) that can not be assigned to a single token but only a part of it, we can use the already existing morphology layer:
Consider the example of the English contraction it's:
Here I want to stress that this is not the only possible representation for this contraction, as we can just as well express it with two tokens as shown in the next example. It's not FoLiA's job to favour one over the other, but that is a decision of the creator/researcher/tokeniser, FoLiA just has to provide the facilities that make both models possible:
The morphology notation in FoLiA is very powerful and nestable. Consider the arabic token فيبيتك. This consists of three words meaning "in your house", translitterated in the below example for ease of reading:
Morphemes (and phonemes) can explicitly be referred to (like of words/tokens) from any span annotation (
wref).My question (mainly for @kdepuydt, @JessedeDoes) if is this solution is sufficient (it can capture contractions, clitics, etc.. ) and linguistically accurate enough (e.g. grouping it all under morphology)? If there are counter-examples, I'd be very interested.
Compound classes
One point that arises from current annotations in the CRM and Gysseling corpora (historical dutch), is the use of what I call compound PoS-tags and lemmas. Take the arabic example above, the token itself does not have a PoS tag, but one may want to force a tag anyway and assign something like prep+n+pron. Recall that FoLiA itself does not define the tagset, so this would be valid. However, the semantics of it being some kind of compound class would not be formalised in any way. The question arises whether we need facilities for explicitly representing compound classes? Perhaps we should allow FoLiA set definitions to define operators such as
+, allowing for more expressivity in classes. This as opposed to really defining operators in FoLiA itself, because that begs the question which operators are needed and that is more a property of the vocabulary in question. In categorial grammars for instance, one would want to define/and\. In other vocabularies, perhaps more set-theoretic operators such as∪and∩make sense. If operators are introduced, then of course bracketing and operator precendence becomes a factor to take into account a well. The class would cease to be a simple reference and allow for a mini-language in it's own right, although for many tools this is of no consequence.I'm not yet including a specific proposal for this, but would very much like to hear your thoughts on this direction.