Implement an incremental construction of FoLiA (output) files

[kosloot]

For some purposes it is handy to make it possible to incrementally construct a FoLiA document while keeping only fragments in memory. There are probably many way to accomplish this, but I would like to sketch a simple approach, which might be useful for ucto or frog.

This approach doesn't rely on SAX or XmlReader based features yet. That might be needed in the long run, but for now I propose an even simpler idea.

I will describe the API, which should almost be selfexplaining...

• constructor: foliaStream( const string& id. ostream& os ) creates an empty FoLiA document with id=id. The document contains a basic document, including an empty \<text> node, which is considered the root. We will manipulate this doc, and gradually write it to a stream.

The xmltag could be a parameter too? for other roots.

Also in the future we could initialize from a FoLiA file, using XmlReader to initialize only a base document.

• members:

  • document() returns the FoLiA proto-document
  • root() returns the root of the FoLiA proto-document. (so the Text, for now)
  • output_header() Output the header of the folia document to the stream, up to the opening tag of the root node. This should FREEZE all global operations on the document (adding annotations and sets and such) as they can never be output again. Calling more then once is prohibited.
  • add_node( FoLiAElement * ) add a FoLiA subtree under the root. This tree should be created as parts of document() (to get indexes and sets right) It can be called multiple times.
  • flush( ) Writes all availanle FoLiA nodes under root() to the file and REMOVES them from the document. Keeping the footprint of the document low. When output_header() is not called yet, do that first.
  • output_footer() Output the tail of the Document, from till the end. Can only be called only once. flushes the foliaStream() first. So it even writes the header, if not already done.

• destructor: call output_footer() when necessary, the closes the stream and removed the document.

Typical usage: In UCTO:

• create a foliaStream
• declare all needed annotations/setnames in the document.
• output_header() loop:
  • read a 'line'
  • tokenize
  • create FoLiA node(s) relative to the document()
  • add()
  • flush()
  • loop again output_footer()

In FROG

• create a foliaStream
• declare all needed annotations/setnames in the document.
• output_header() loop:
  • read a 'line' / sentence /paragraph?
  • tokenize the line to a FoLiAFragment. (might need some ucto hacking)
  • perform MBLEM/MBMA/NER etc on the fragment. (might need work)
  • add()
  • flush()
  • loop again output_footer()

[kosloot]

This is now handled by the new folia_processor. Which is used by Ucto and Frog now.