eduarddrenth
commented
7 years ago Thanks for this request, which improves linguistic support in TEI. Some challenges remain, especially when querying, presenting and researching TEI material in diffently annotated corpora.
1) @pos can have any content (which of course supports all tagsets) 2) @msd groups together several features (person, number, ...) and as well does not restrict content
Another remark is that in the documentation text I see no examples on @pos or @msd. Perhaps it is good to include a rationale behind the generalistic approach and to show in examples how to deal with the consequences for example when developing tools.
susannehaaf
peterstadler
bansp
rvdb
Quick links:
Introduction and summary
We propose an attribute class that gathers token-level attributes facilitating simple linguistic annotation. We propose that
<w>and<pc>should be its members by default. At the same time, grouping these attributes in a separate class (rather than defining them inside an element) makes it possible for other token-level elements to be added to that class, in TEI customizations. Our proposal addresses both the future uses (extending the TEI offer towards corpus linguists) as well as the existing resources (enriching the existing, for example literary or historical corpora, with basic linguistic annotation).The proposed class, att.linguistic, contains first of all two linguistic attributes originally defined inside
<w>, namely@lemmaand@lemmaRef. In addition to these, we suggest to add the following:@pos(for part-of-speech)@msd(for morphosyntactic description)@reg(for the regularized form in e.g. historical corpora)@jointo signal the absence of whitespace next to the element -- this is meant for all non-mixed-content contexts where the whitespace information is not conveyed by the actual presence of whitespace charactersWe stress that the goal of this feature request is not to facilitate in-depth linguistic annotation, but rather to equip "off-the-shelf" TEI in the very basic tools that linguists can use, and that non-linguists can safely add to their existing resources in order to enhance them. This is why we address the TEI namespace directly. Below, we first present the motivation for this feature request, offer a detailed description of the suggested attributes, and then address some potential counterproposals and provide further examples.
This is a lengthy ticket, but it concerns complex matter, and we want to minimize the danger of multi-thread discussion developing in the comments section, concerning points that we can address together and in a structured manner. The ticket is accompanied by a pull request (see the link above) which among others suggests documentation for the new class. This is in order to be maximally clear about what we suggest and also in order to minimize the workload expected of the Technical Council.
Context and motivation
The topic of the perceived inadequacy of the vanilla TEI recommendations (including the "Corpus" extension available from Roma) for building technologically effective, rather than merely theoretically "nice" linguistic resources has stirred some of the TEI community for years, essentially ever since the massively successful fork of the early TEI called CES (Corpus Encoding Standard). (X)CES and its variants have dominated the scene of light-weight language resources for years, gathering a large group of followers and producing an immense (for that time) amount of linguistic encoding.
Today, the TEI has a chance to claim a large part of the language resource market and to present an attractive offer not only to linguists as such, but also to philologists and text technologists who appreciate the added value that comes from enriching the existing structural markup with linguistic information. Especially in resources that already use the
<w>element, enrichment with part-of-speech and morphosyntactic information in addition to lemma identification drastically expands search options, options for sorting the results, or for investigating author-specific traits. Being able to provide local information on regularized forms affects search and display, and makes the system far more user-friendly and useful.Efforts targeting the enhancement of TEI markup for lexical resources as well as for constructing structurally robust multi-layer linguistic annotation are the topics of other projects that the LingSIG is involved in. In this very ticket, we concentrate on minimal extensions to the attribute repertoire of
<w>, aiming at a compromise between the needs of modern-day language technology and the tendency to keep extensions to the mainstream TEI in check.Description of the proposed additions
@posand@msd"POS" stands for "part of speech" and the attribute contains a symbol that classifies the content of the given
<w>in opposition to other elements in the POS space, according to predefined criteria (typically a mixture of lexical, syntactic, and semantic ones). Typical, textbook-level parts of speech include the familiar "noun" and "verb", but for linguistic-technological purposes the divisions are more fine-grained. The number of symbols in a tagset depends on what criteria are taken into account and on how much morphosyntactic information is mixed into the POS tagset (in practice, mixing "all" morphosyntactic information into the POS tagset is only possible in languages with relatively simple morphosyntax). A very simple repertoire of POS values can be found in the so-called Universal POS Tagset, where no morphosyntactic information is present. On the other end of the scale, one could place e.g. the CLAWS-8 tagset, which mixes parts of speech with morphosyntax, and thus distinguishes, e.g., between "base form of a lexical verb" and "past tense of a lexical verb" and "past participle of lexical verb" (all three of which would be just a "verb" in the 'Universal Tagset'). We mention this information here in order to explain why, for certain languages, for certain uses, and for certain tools, POS information and morphosyntactic information can be merged.Most parts of speech have sets of morphological or morphosyntactic categories associated with them (prepositions in English do not, verbs do). Information on the particular values of those categories constitutes morphosyntactic description and
@msdis the place for it. Recall that some extended POS tagsets incorporate morphosyntactic description directly, but that is not an option for all languages. Tens of corpora (e.g. those in MULTEXT and MULTEXT-East projects) have followed the CES guidelines which distinguish between POS and MSD. See the original CES documentation (only mildly helpful) and a fragment of T. Erjavec's tutorial on morphosyntactic tagging for the Multext-East project (using CES at that time, note that 'ctag' was CES's name for 'pos'):Above,
<lex>lists output of the morphological analyser, whereas<disamb>encloses a disambiguated sequence. This is still mild, now compare a case from the National Corpus of Polish (fileann_morphosyntax.xml; warning: the file is HUGE), a set of potential interpretations of the adjectival form 'kategoryczne' ("base" is the lemma):The first option denotes a bundle of features "plural+nominative+animate masculine+positive" (see the tagset documentation for other categories and values).
The 'universal' (in a pragmatic sense) repertoire of morphosyntactic features can be seen in the guidelines offered by the Universal Dependencies project.
@regIn historical corpora, regularization is a very frequent matter which may be applied not only to words but also to punctuation characters.
Our argument for introducing
@regas an attribute of<w>rests on the following assumptions and observations:@reggreatly simplifies the encoding of some historical corpora -- it is not a theoretical wonder and cure-all, it is just an attested convenienceThe
@regattribute can be used to mark up regularization in corpora with non-standardized spelling (historical corpora, literary corpora of historical texts, etc.). The following examples illustrate this:Source: Aviso. Relation oder Zeitung. Wolfenbüttel, 1609. In: Deutsches Textarchiv.
Source: Gottfried, Newe Welt Vnd Americanische Historien. Frankfurt/M., 1631. In: Deutsches Textarchiv.
The TEI has at its disposal the powerful mechanism of
//choice/reg|origfor such cases. However, with the usage of//choice/reg|origfor the annotation of regularized word forms, text annotation in specialized corpora may need to be extended significantly since there may already exist further (elaborate) annotation on or below the token level, which gets automatically multiplied with the use of<choice>. If the<choice>machinery is applied, these existing annotations will end up being represented twice: once in<reg>and once in<orig>. Thus texts might be significantly extended due to such unnecessary repetition, which may also negatively affect the sustainability of the resource. The following examples illustrate this problem:Example: Tagging of highlighting within a word (here: typeface switch from Fraktur to Antiqua):
Source: Marperger, Der allzeit-fertige Handels-Correspondent. Hamburg, 1717, facs. 655. In: Deutsches Textarchiv.
Example: Word syllabification and hyphenation at the end of a page
In cases of word syllabification and hyphenation at the end of a page, comparably large amounts of text material have to be included in the
<w>element (like running titles, catchwords, pagebreaks). The use of the<choice>mechanism would mean extending this by repetition. The alternative is a single attribute doing the necessary work -- compare what would have to become of the example below if the//choice/reg|origmachinery were to be employed, against using just the following substitution in the second line:<w reg="desselbigen">deſ-<lb/>.Source: Marperger, Der allzeit-fertige Handels-Correspondent. Hamburg, 1717. In: Deutsches Textarchiv. part 1: facs. 671, part 2: facs. 672.
Example: Word syllabification and hyphenation combined with marginal notes
Furthermore, consider cases of word syllabification and hyphenation at the end of a line which is followed by a marginal note. Here, the lexical material of the marginal note would be subject to linguistic analysis as well, thus resulting in several nested
//choice/reg|origelements, increasing the redundancy and the curation effort.Source: Arndt: Von wahrem Christenthumb. Vol. 1. Magdeburg, 1610. In: Deutsches Textarchiv
When using
<choice>, one would always have to include the additional text parts twice in the transcription (once in<orig>and once in<reg>) or to shift the complexity elsewhere (without diminishing it -- escaping complexity in this case may mean standoff markup or fragmented markup, neither option being particularly attractive). We argue that, while not as powerful as<choice>, in the cases presented here and similar,@regconstitutes a very reasonable encoding compromise that will allow numerous encoding projects to embrace TEI nearly out-of-the-box.Several issues need to be made clear when applying regularization, and below, we mention the most important of them. These issues should be highlighted and exemplified in the documentation, and we will be happy to submit the relevant extensions to the Guidelines and/or specs if the present request is accepted.
@regmay need to store multi-word sequences (example: German historical form ſovieler regularized into "so vieler"); in this,@regis fully analogous to the existing attribute@lemma, using teidata.text as its datatype; the analogy with@lemmaextends to most other issues concerning@regas an attribute.@prevand@nextattributes to perform the linking role (together with a convention saying that, e.g., the first of such<w>s is going to carry the@regattribute. If this cannot be guaranteed, it means that the<choice>system needs to be used instead of@reg.<choice>/@reg) may be difficult or impossible to realise -- when, for example, a single-token original is regularized to more than one token inside the@regattribute, and it's the regularized data stream that should be annotated for POS. We do not treat such potential cases as invalidating the entire approach; they merely show that the convenience@regattribute is not usable in just such cases, and that the encoder has to resort to the<choice>machinery and possibly deal with the overwhelming complexity of a construction rooted in<choice>(as illustrated in the examples in this section).@regapproach together with fragmentation handling (@prevand@next) that may be a better tool to handle some of such cases, because using<choice>would involve not only additional lexical information but would also require a potentially arbitrary word-order decision in the<reg>branch.@joinConsider a version of an example used in Chapter 17: "What did you make up?" (unlike the Guidelines, we retain the punctuation).
Now contrast this with the version actually provided there (we add the
<pc>for completeness, but the argument would hold for<w>as well, in compounds written together, but commonly analysed as composed of separate tokens).The first, inline example provides more information than the second -- it uses additional typographical markup, namely whitespace. The second, sequential example only lists the tokens in the order found in the sentence, but loses the information on the lack (or the presence) of whitespace. In order to preserve this kind of information, we adopt the
@joinattribute from ISO MAF (Morpho-syntactic Annotation Framework, ISO 24611:2012), with the values: 'left', 'right', and 'both'. With this attribute, our sequential example would be rendered as follows:An issue may arise concerning the redundancy of marking the absence of whitespace on two elements. From the top-down, global perspective, it is indeed redundant. From the bottom-up, "streamable" perspective, it is not redundant, and it is the latter perspective that we assume as fundamental: we want to make the TEI more attractive for the linguistic enrichment of massive amounts of text that are submitted to rapid processing in a streaming fashion. Nothing precludes project-specific approaches to this issue. For example, the National Corpus of Polish only marked the absence of the preceding whitespace, on all segments. We would like to provide support for the general, redundant, "streamable" case, of which project-specific decisions can be proper subsets.
Existing alternatives to the proposed attributes
It might be suggested that the existing attributes
@ana,@correspand@typecan be used for the purpose of providing lightweight linguistic markup. This section argues against that suggestion on several grounds.A fixed, homogeneous set of linguistic features
Especially in the existing literary resources, which may become subject to enrichment with linguistic markup, the three attributes above, in any combination, may have already been put to work for the purposes of literary analysis, prosopography, etc. In such cases, the devil-advocate suggestion would be "use those of them that happen to be available and squeeze linguistic analysis into them". Such an approach, however, would immediately lead to heterogeneity exactly where we need order and homogeneity: linguistic annotation would generally start with information on lemma, POS, often also morphosyntactic description and (especially for texts with non-standardized orthography) the regularized form. The set of terms used to name these features is quite well established among linguists. Giving up this homogeneity already at the start in favour of ad hoc arrangements would have severe effects on the interoperability of the resulting corpora.
Our goal here is to provide a structured and well-documented proposal for information containers which represent basic and frequent features of linguistic analysis, and in doing so, to increase the attractiveness of the TEI for those interested in quickly and efficiently creating resources with lightweight linguistic markup, rather than enhancing the feeling that "anything goes".
Reserving some of the generic attributes for linguistic purposes would mean (a) removing them from the general pool of attributes available for non-linguistic uses and (b) cutting off some of the legacy literary resources that already use those attributes.
Avoiding pointers
@anaand@correspare pointer-based, and a large part of our motivation is to create a processing-friendly and self-contained system, with no need of pulling information from any outside sources.Consider an example from section 17.4 (v. 3.1.0, generated on 2016-12-15):
We set aside the absolutely non-linguistic practice of random inclusion of the following whitespace inside
<w>, which betrays the made-up status of this example, and focus on the use of@anahere. The pointers in the examples are resolved in an<interpGrp>placed most probably within the TEI Header. For POS tagging this means, on average, minimally about 50 URIs to resolve (50 for the German basic STTS, over 160 for the English CLAWS-8). In practice, however, for efficient further processing, analysis tools would probably just have to be adjusted to not resolve the links but rather pre-parse them by cutting off the leading '#' and treat the rest as atomic values (comprising all the information needed for most scenarios). Thus, the original TEI design would be circumvented for pragmatic reasons.Next, the question would be where to place the results of further analysis, such as morphosyntactic information. POS and MSD could be packed into a single
@anaattribute, probably as a sequence of two URIs (and we put aside the fact that attaching semantics to a position in a chain of URIs inside a single attribute value smells of malpractice): the Polish example of multiple values of @msd is by no means the most extreme one -- the number of combinations is literally in the order of hundreds. Do we expect that all these URIs are resolvable somewhere in the corpus header? And if we do not assume that, but rather treat the leading '#' as a necessity imposed by the datatype of the attribute, it means that in practice we're saying, "well, if you want to encode your corpus in the TEI, we have this kludge here prepared for you, whereby you just need to pretend that these URIs aren't URIs...". Note also that we can only offer this kludge to freshly built corpora that haven't used@anafor any other purpose -- but why build a fresh corpus upon kludges?Summing up, aside from resigning from the proposed homogeneity of feature names and feature values that we focused on in the previous section,
@anacould be made to work for the indicated purpose in a project that would (i) clearly narrow its semantics (as we can see in ch. 17,@anais used for many purposes, including various kinds of literary analysis) and that would (ii) adjust its analysis tools to not resolve the links but rather to pre-parse them by cutting off the leading '#' and treat the rest as atomic values. But this shows exactly how and where our suggestion is superior: first of all, the above scenario works only for corpora that have been designed for (sparse) linguistic markup from the beginning: the attribute@anais not used for literary analysis there, and processing/analysis tools must be sensitive to the internal make-up of its value. This alone excludes legacy literary corpora and off-the-shelf linguistic tools.Suggested deployment strategy
Minimally, we expect that the only impact visible in the schema will be four new attributes usable for the
<w>and<pc>elements. However, because these attributes are now (together with@lemmaand@lemmaRef) gathered into a separate class, they will be available to ODD designers for inclusion into other elements. So, for example, people who plan to base their tokenization on the<seg>element, will be able to add it toatt.linguistic. We suggest to add<seg>to this class in the official customization for "Linguistic Corpora" (available through Roma).Adoption of this ticket, in whole or in part, should be reflected in the text of the Guidelines, and the present pull request suggests minimal changes to this effect. If the ticket is accepted, we will be happy to supply a proposal for thorough modifications in the "Simple Analytic Mechanisms", preferably in a separate pull request, in case these changes become a subject of extended debate.
The ISO MAF specification envisions two more values for
@join, namely "no" and "overlap". In order to preserve full compatibility with ISO MAF, we suggest to keep all the values and that is how the attribute is defined the spec.Illustration
The fragment below comes from "His Majesties declaration: to all his loving subjects, of the causes which moved him to dissolve the last Parliament. Published by His Majesties speciall command", part of TCP Phase I, 1640-1660 texts, published on GitHub by Martin Mueller in the English-Civil-War repository, with some enhancement in the form of the
@joinattribute:The next fragment comes from "A most notable and worthy example of an vngratious sonne, who in the pride of his hart denied his owne father and how God for his offence turned his meate into loathsome toades. To the tune of Lord Darley.", part of the EarlyPrints repository.
The following example comes from 18th century German prose, taken from Blumenbach, Johann Friedrich: Zwo Abhandlungen über die Nutritionskraft. St. Petersburg, 1789; see also the respective page in: Deutsches Textarchiv. (Note that the
@joinattribute was added to the original data here.)The following is an example from 17th century German prose, taken from Friderici, Daniel: Musica Figuralis, Oder Newe Klärliche Richtige/ vnd vorstentliche vnterweisung/ Der SingeKunst. Rostock, 1619; see also the corresponding page in Deutsches Textarchiv.