[foliasplit] Split a FoLiA document into multiple

proycon / foliatools

A number of command-line tools for working with FoLiA (Format for Linguistic Annotation). Includes validators, converters, visualisers, and more.

GNU General Public License v3.0

10 stars 4 forks source link

[foliasplit] Split a FoLiA document into multiple #20

Closed proycon closed 3 years ago

proycon commented 3 years ago

New: Implement a tool that splits a FoLiA document into multiple, on the basis of:

specific IDs of structural elements to take as the new roots
after processing x instances of a particular element (for splitting e.g. every 1000 sentences)
sections that have associated submetadata, the submetadata will then become the metadata of the new document

kosloot commented 3 years ago

Are you thinking of real separate documents? Or splitting in a master document and a bunch of external ones? I think both variants could be useful.

proycon commented 3 years ago

This is splitting actual documents yes, not using the external functionality.

proycon commented 3 years ago

Ok, I will take of the "external" mechanism too, you're right that it will be useful and this is the proper place for it. Better solve it well once and for all.

proycon commented 3 years ago

May have to do additional testing to see if some performance bottlenecks can be identified and solved.