Discuss how to make InkVisitor into a full-text manager and (actual) annotation tool

[davidzbiral]

Now, InkV does not store and manage full-texts, and the statement-level Text field is edited manually. But we absolutely need this feature.

• NLP people read us as an annotation tool - so the best way is to become one.
• The DH community always wants this feature.
• Sometimes, there is lack of clarity over which part of the Text field the statement actually represents, because editors are saving some context (to make the text intelligible) - sometimes marking that context away by using pointed brackets {}. We need to avoid this lack of clarity in alignment full-text / CASTEMO Statement.
• (Not so important: It would be nice to be able to produce full-text export enriched some knowledge from CASTEMO knowledge graphs.)

"There will be a thorough development in this, because it is almost certain that InkV will need to become a full-text manager too, and so it will eventually really become what people read it like - an annotation tool, without losing any strength, but rather acquiring further strenghts - storing the corpus directly in it, and connecting the Text field of Statement with the actual full-text through markup at the background."

New additions:

• I don't understand my previous hesitations around Ts and Sub-Ts. Of course, it must be appended to an R, not a T. And then, it can be used in several Ts, some of them using the same R (e.g. T "Register of Toulouse" and T "Endura in the registers from 1270-1330").
• Pseudo-XML markup marking the UUIDs of territories (hierarchical) and statements must be added. (Pseudo- because it needs to allow overlapping markup, i.e. it is not W3C-valid XML. This is a known weakness of XML.)
• Using this (and not GitHub/DISSINET/corpus) in an NLP pipeline must be solved, incl. where to store (GitHub?) the derivatives (tokenized, segmented, part-of-speech tagged, dependency-parsed...).
• We will start with plain text, but should be prepared for PDF and image (pixel coordinates!). Ideally, this should incorporate an existing open-source annotation tool at least for image and PDF annotation.
• We need selection of text to mark what the statement is modelling.
• A statement is often divided into two by an inserted clause - i.e. the markup must use the UUID but also be able to use it twice or somehow tweak the markup to have UUID-part1 and UUID-part2 ... UUID-part N.
• Relation to text field must be solved.
• We must meet.

[tomaham]

@davidzbiral We have discussed this partially during the prep for hck2 - there was this idea of solution that the full text would be part of the territory-statement composition, but you/we had rejected it if I recall rightly, because of connected complexities.

I would hold our horses and really discuss it first. So can we change the title of this issue to "Discuss..." from "Make...." ?

[davidzbiral]

To be sure, I only rejected it as work realistic before Hackathon 2. I absolutely think we should go towards having full texts in InkV in a more long-term run. In any case, don't worry - what will be the focal points of 1.4 and whether this is a part of them will be properly discussed. Name changed.

[davidzbiral]

Moving some info from one (previously too focused) milestone here so that it does not get lost:

Highly relevant for the DH community, digital history, as well as for qualitative sociology (CAQDAS users). Could go even beyond the support of segmented TXT files - in terms of ignoring the coding (because TEI/XML contains annotation that is, rather, an alternative to the knowledge graph constructed in InkV).

Important for:

• Bringing closer the DISSINET research environments (here, metadata stored in the db, and the full-texts).
• Solving the relation between the text field in a Statement and the larger text.
• Easier alignment between CASTEMO coding and full-text (e.g. for machine learning).
• Indispensible if we are to offer an actual alternative to CAQDAS.
• Satisfying the need of the community - InkV is text-oriented (source-oriented), but offers no full-text management options.

Concerning the relevance of this area, e.g., review from the DH community on a CASTEMO tutorial proposal says: "It's not clear what kinds of text can be used as input to Castemo (could it for example handle a standard TEI document, or output from treetagger?), or whether it assumes that the text will be entered by hand."