Repetition, retracing, reformulation correction

[JeltevanBoheemen]

In the current parses, the following CHAT annotated constructs are ignored:

• repetition [/]
• retracing [//]
• reformulation [///]

The following steps should be implemented and executed:

• [x] identify parse files with these constructs. This is already done by @JanOdijk, paths are provided in error check Excel file.
• [x] extract the part of the original utterance that is ignored
• [x] parse the above part separately, using the same Alpino version as the original parse
• [x] take everything under the top/top node, and prepend it to the original top/top node
• [ ] fix node attributes for new tree:
  • @begin and @end
  • [ ] count new number of leaf nodes
  • [ ] add the count to original nodes
  • @id
  • presumed implementation is a depth-first sequence 0...n
  • [ ] check this with existing files
  • [ ] fix reparses
  • [x] add metadata concerning false starts. @JanOdijk developed a script for this purpose.

[JanOdijk]

The metadata can be added by using the function cleantext from the module cleanCHILDEStokens

def cleantext(utt: str, repkeep: bool, tokenoutput: bool = False) -> Tuple[CleanedText, Metadata]:

where utt is a string with CHAT-annotations (to be taken from the existing metadata associated with the parse tree), the output is a tuple consisting of a cleaned text (all CHAT-annotations applied) and a list of Metadata .

For the cases at hand, it will turn the CHAT codes [/] and [//] into appropriate metadata.

There is no need to run this specifically for this set, because metadata for all AnnCor utterances must be generated by using this function