=pod

=encoding utf8

=head1 NAME

korapxml2krill - Merge KorAP-XML data and create Krill documents

=head1 SYNOPSIS

$ korapxml2krill [archive|extract] --input <directory|archive> [options]

=head1 DESCRIPTION

L is a library to convert KorAP-XML documents to files compatible with the L<Krill|https://github.com/KorAP/Krill> indexer. The C command line tool is a simple wrapper of this library.

=head1 INSTALLATION

The preferred way to install L is to use L<cpanm|App::cpanminus>.

$ cpanm https://github.com/KorAP/KorAP-XML-Krill.git

In case everything went well, the C tool will be available on your command line immediately. Minimum requirement for L is Perl 5.32. Optionally installing L speeds up archive building. Optional support for L to calculate available cores is available. In addition to work with zip archives, the C tool needs to be present.

=head1 ARGUMENTS

$ korapxml2krill -z --input --output

Without arguments, C converts a directory of a single KorAP-XML document. It expects the input to point to the text level folder.

=over 2

=item B

$ korapxml2krill archive -z --input <directory|archive> --output <directory|tar>

Converts an archive of KorAP-XML documents. It expects a directory (pointing to the corpus level folder) or one or more zip files as input.

=item B

$ korapxml2krill extract --input --output --sigle

Extracts KorAP-XML documents from a zip file.

=item B

$ korapxml2krill serial -i -i -o -cfg

Convert archives sequentially. The inputs are not merged but treated as they are (so they may be premerged or globs). the C<--out> directory is treated as the base directory where subdirectories are created based on the archive name. In case the C<--to-tar> flag is given, the output will be a tar file.

=item B

$ korapxml2krill slimlog >

Filters out all useless aka succesfull information from logs, to simplify log checks. Expects no further options.

=back

=head1 OPTIONS

=over 2

=item B<--input|-i> <directory|zip file>

Directory or zip file(s) of documents to convert.

Without arguments, C expects a folder of a single KorAP-XML document, while C expects a KorAP-XML corpus folder or a zip file to batch process multiple files. C expects zip files only.

C supports multiple input zip files with the constraint that the first archive listed contains all primary data files and all meta data files.

-i file/news.zip -i file/news.malt.zip -i "#file/news.tt.zip"

Input may also be defined using BSD glob wildcards.

-i 'file/news*.zip'

The extended input array will be sorted in length order, so the shortest path needs to contain all primary data files and all meta data files.

(The directory structure follows the base directory format that may include a C<.> root folder. In this case further archives lacking a C<.> root folder need to be passed with a hash sign in front of the archive's name. This may require to quote the parameter.)

To support zip files, a version of C needs to be installed that is compatible with the archive file.

B<The root folder switch using the hash sign is experimental and may vanish in future versions.>

=item B<--input-base|-ib>

The base directory for inputs.

=item B<--output|-o> <directory|file>

Output folder for archive processing or document name for single output (optional), writes to C by default (in case C is not mandatory due to further options).

=item B<--overwrite|-w>

Overwrite files that already exist.

=item B<--token|-t> #

Define the default tokenization by specifying the name of the foundry and optionally the name of the layer-file. Defaults to C<OpenNLP#tokens>. This will directly take the file instead of running the layer implementation!

=item B<--base-sentences|-bs> #

Define the layer for base sentences. If given, this will be used instead of using C<Base#Sentences>. Currently C<DeReKo#Structure> and C<DGD#Structure> are the only additional layers supported.

Defaults to unset.

=item B<--base-paragraphs|-bp> #

Define the layer for base paragraphs. If given, this will be used instead of using C<Base#Paragraphs>. Currently C<DeReKo#Structure> and C<DGD#Structure> are the only additional layer supported.

Defaults to unset.

=item B<--base-pagebreaks|-bpb> #

Define the layer for base pagebreaks. Currently C<DeReKo#Structure> is the only layer supported.

Defaults to unset.

=item B<--skip|-s> [#]

Skip specific annotations by specifying the foundry (and optionally the layer with a C<#>-prefix), e.g. C or C<Mate#Morpho>. Alternatively you can skip C<#ALL>. Can be set multiple times.

=item B<--anno|-a> #

Convert specific annotations by specifying the foundry (and optionally the layer with a C<#>-prefix), e.g. C or C<Mate#Morpho>. Can be set multiple times.

=item B<--non-word-tokens|-nwt>

Tokenize non-word tokens like word tokens (defined as matching C</[\d\w]/>). Useful to treat punctuations as tokens.

Defaults to unset.

=item B<--non-verbal-tokens|-nvt>

Tokenize non-verbal tokens marked as in the primary data as the unicode symbol 'Black Vertical Rectangle' aka \x25ae.

Defaults to unset.

=item B<--jobs|-j>

Define the number of spawned forks for concurrent jobs of archive processing. Defaults to C<0> (everything runs in a single process).

If C is not set to true, this will also apply to extraction.

Pass C<-1>, and the value will be set automatically to 5 times the number of available cores, in case L is available and can read CPU count (see C<--job-count>). Be aware, that the report of available cores may not work in certain conditions. Benchmarking the processing speed based on the number of jobs may be valuable.

This is I.

=item B<--job-count|-jc>

Print job and core information that would be used if C<-1> was passed to C<--jobs>.

=item B<--koral|-k>

Version of the output format. Supported versions are: C<0> for legacy serialization, C<0.03> for serialization with metadata fields as key-values on the root object, C<0.4> for serialization with metadata fields as a list of C"@type":"koral:field" objects.

Currently defaults to C<0.03>.

=item B<--sequential-extraction|-se>

Flag to indicate, if the C value also applies to extraction. Some systems may have problems with extracting multiple archives to the same folder at the same time. Can be flagged using C<--no-sequential-extraction> as well. Defaults to C.

=item B<--meta|-m>

Define the metadata parser to use. Defaults to C. Metadata parsers can be defined in the C namespace. This is I.

=item B<--gzip|-z>

Compress the output. Expects a defined C file in single processing.

=item B<--cache|-c>

File to mmap a cache (using L). Defaults to C in the calling directory.

=item B<--cache-size|-cs>

Size of the cache. Defaults to C<50m>.

=item B<--cache-init|-ci>

Initialize cache file. Can be flagged using C<--no-cache-init> as well. Defaults to C.

=item B<--cache-delete|-cd>

Delete cache file after processing. Can be flagged using C<--no-cache-delete> as well. Defaults to C.

=item B<--config|-cfg>

Configure the parameters of your call in a file of key-value pairs with whitespace separator

overwrite 1 token DeReKo#Structure ...

Supported parameters are: C, C, C, C, C, C, C, C, C, C, C, C, C, C, C, C, C, C, C (semicolon separated), C (semicolon separated), C (semicolon separated).

Configuration parameters will always be overwritten by passed parameters.

=item B<--temporary-extract|-te>

Only valid for the C and C commands.

This will first extract all files into a directory and then will archive. If the directory is given as C<:temp:>, a temporary directory is used. This is especially useful to avoid massive unzipping and potential network latency.

=item B<--to-tar>

Only valid for the C command.

Writes the output into a tar archive.

=item B<--sigle|-sg>

Extract the given texts. Can be set multiple times. I<Currently only supported on C.> Sigles have the structure C/C/C. In case the C path is omitted, the whole document will be extracted. On the document level, the postfix wildcard C<*> is supported.

=item B<--lang>

Preferred language for metadata fields. In case multiple titles are given (on any level) with different C attributes, the language given is preferred. Because titles may have different sources and different priorities, non-specific language titles may still be preferred in case the title source has a higher priority.

=item B<--log|-l>

The L log level, defaults to C.

=item B<--quiet>

Silence all information (non-log) outputs.

=item B<--help|-h>

Print help information.

=item B<--version|-v>

Print version information.

=back

=head1 PERFORMANCE

There are some ways to improve performance for large tasks:

=item First unpack

Using the archive or serial command on one or multiple zip files can be very slow, as it needs to unpack small portions every time. It's better to use C<--temporary-extract> to unpack the whole archive first into a temprary directory and then read the extracted files. This is especially important for remote archives

=item Limit annotations

Per default, all supported annotation layers are sought. This can be limited by adding C<--skip '#ALL'> and only listing the expected annotations with C<--anno>.

=item Checking the parallel job count

By providing the number of parallel jobs using C<--jobs>, the execution can be tailored to specific hardware environments.

=head1 ANNOTATION SUPPORT

L has built-in importer for some annotation foundries and layers developed in the KorAP project that are part of the KorAP preprocessing pipeline. The base foundry with paragraphs, sentences, and the text element are mandatory for L<Krill|https://github.com/KorAP/Krill>.

Base

Paragraphs

#Sentences

Connexor

Morpho

#Phrase
#Sentences
#Syntax

CoreNLP

Constituency

#Morpho
#NamedEntities
#Sentences

CorpusExplorer

Morpho

CMC

Morpho

DeReKo

Structure

DGD

Morpho

#Structure

DRuKoLa

Morpho

Glemm

Morpho

Gingko

Morpho

HNC

Morpho

LWC

Dependency

Malt

Dependency

MarMoT

Morpho

Mate

Dependency

#Morpho

MDParser

Dependency

NKJP

Morpho

#NamedEntities

OpenNLP

Morpho

#Sentences

RWK

Morpho

#Structure

Sgbr

Lemma

#Morpho

Spacy

Morpho

Talismane

Dependency

#Morpho

TreeTagger

Morpho

#Sentences

UDPipe

Dependency

#Morpho

XIP

Constituency

#Morpho
#Sentences

More importers are in preparation. New annotation importers can be defined in the C namespace. See the built-in annotation importers as examples.

=head1 METADATA SUPPORT

L has built-in importer for some meta data variants that are part of the KorAP preprocessing pipeline.

=over 2

=item B

Meta data for all I5 files

=item B

Meta data from the Schreibgebrauch project

=item B

Meta data from the Gingko project in addition to I5

=item B

Meta data for the ICC in addition to I5

=item B

Meta data for the NKJP corpora

=back

New meta data importers can be defined in the C namespace. See the built-in meta data importers as examples.

The I5 metadata definition is based on TEI-P5 and supports C<ExenoDataE> with C<EmetaE> elements like

...

that are directly translated to Krill objects. The supported values are:

=over 2

=item C

=over 4

=item C

String meta data value

=item C

String meta data value that can be given multiple times

=item C

String meta data value that is tokenized and can be searched as token sequences

=item C

Date meta data value (as "yyyy/mm/dd" with optional granularity)

=item C

Numerical meta data value

=item C

Non-indexed meta data value (only retrievable)

=item C

Non-indexed attached URI, takes the desc as the title for links

=back

=item C

The key of the meta object that may be prefixed by C or C, in case the C<ExenoDataE> information is located on these levels. The text level introduces no prefixes.

=item C (optional)

A prefixed namespace of the key

=item C (optional)

A description of the key

=item text content

The value of the meta object

=back

=head1 About KorAP-XML

KorAP-XML (Bański et al. 2012) is an implementation of the KorAP data model (Bański et al. 2013), where text data are stored physically separated from their interpretations (i.e. annotations). A text document in KorAP-XML therefore consists of several files containing primary data, metadata and annotations.

The structure of a single KorAP-XML document can be as follows:

data.xml
header.xml
- base
  - tokens.xml
  - ...
- struct
  - structure.xml
  - ...
- corenlp
  - morpho.xml
  - constituency.xml
  - ...
- tree_tagger
  - morpho.xml
  - ...
- ...

The C contains the primary data, the C contains the metadata, and the annotation layers are stored in subfolders like C, C or C (so-called "foundries"; Bański et al. 2013).

Metadata is available in the TEI-P5 variant I5 (Lüngen and Sperberg-McQueen 2012). See the documentation in L for translatable fields.

Annotations correspond to a variant of the TEI-P5 feature structures (TEI Consortium; Lee et al. 2004). Annotation feature structures refer to character sequences of the primary text inside the C element of the C. A single annotation containing the lemma of a token can have the following structure:

zum

The C and C attributes are refering to the character span in the primary text. Depending on the kind of annotation (e.g. token-based, span-based, relation-based), the structure may vary. See L<KorAP::XML::Annotation::*> for various annotation preprocessors.

Multiple KorAP-XML documents are organized on three levels following the "IDS Textmodell" (Lüngen and Sperberg-McQueen 2012): corpus E document E text. On each level metadata information can be stored, that C will merge to a single metadata object per text. A corpus is therefore structured as follows:

- header.xml + - header.xml + - data.xml - header.xml - ... - ...

A single text can be identified by the concatenation of the corpus identifier, the document identifier and the text identifier. This identifier is called the text sigle (e.g. a text with the identifier C<18486> in the document C<060> in the corpus C has the text sigle C<WPD17/060/18486>, see C<--sigle>).

These corpora are often stored in zip files, with which C can deal with. Corpora may also be split in multiple zip archives (e.g. one zip file per foundry), which is also supported (see C<--input>).

Examples for KorAP-XML files are included in L in form of a test suite. The resulting JSON format merges all annotation layers based on a single token stream.

=head2 References

Piotr Bański, Cyril Belica, Helge Krause, Marc Kupietz, Carsten Schnober, Oliver Schonefeld, and Andreas Witt (2011): KorAP data model: first approximation, December.

Piotr Bański, Peter M. Fischer, Elena Frick, Erik Ketzan, Marc Kupietz, Carsten Schnober, Oliver Schonefeld and Andreas Witt (2012): "The New IDS Corpus Analysis Platform: Challenges and Prospects", Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC 2012). L<PDF|http://www.lrec-conf.org/proceedings/lrec2012/pdf/789_Paper.pdf>

Piotr Bański, Elena Frick, Michael Hanl, Marc Kupietz, Carsten Schnober and Andreas Witt (2013): "Robust corpus architecture: a new look at virtual collections and data access", Corpus Linguistics 2013. Abstract Book. Lancaster: UCREL, pp. 23-25. L<PDF|https://ids-pub.bsz-bw.de/frontdoor/deliver/index/docId/4485/file/Ba%c5%84ski_Frick_Hanl_Robust_corpus_architecture_2013.pdf>

Kiyong Lee, Lou Burnard, Laurent Romary, Eric de la Clergerie, Thierry Declerck, Syd Bauman, Harry Bunt, Lionel Clément, Tomaz Erjavec, Azim Roussanaly and Claude Roux (2004): "Towards an international standard on featurestructure representation", Proceedings of the fourth International Conference on Language Resources and Evaluation (LREC 2004), pp. 373-376. L<PDF|http://www.lrec-conf.org/proceedings/lrec2004/pdf/687.pdf>

Harald Lüngen and C. M. Sperberg-McQueen (2012): "A TEI P5 Document Grammar for the IDS Text Model", Journal of the Text Encoding Initiative, Issue 3 | November 2012. L<PDF|https://journals.openedition.org/jtei/pdf/508>

TEI Consortium, eds: "Feature Structures", Guidelines for Electronic Text Encoding and Interchange. L<html|https://www.tei-c.org/release/doc/tei-p5-doc/en/html/FS.html>

=head1 AVAILABILITY

https://github.com/KorAP/KorAP-XML-Krill

=head1 COPYRIGHT AND LICENSE

Author: L<Nils Diewald|https://www.nils-diewald.de/>

Contributor: Eliza Margaretha, Marc Kupietz

L is developed as part of the L<KorAP|https://korap.ids-mannheim.de/> Corpus Analysis Platform at the L<Leibniz Institute for the German Language (IDS)|https://www.ids-mannheim.de/>, member of the L<Leibniz-Gemeinschaft|http://www.leibniz-gemeinschaft.de/>.

This program is free software published under the L<BSD-2 License|https://opensource.org/licenses/BSD-2-Clause>.

=cut

KorAP / KorAP-XML-Krill

readme

Paragraphs

Morpho

Constituency

Morpho

Morpho

Structure

Morpho

Morpho

Morpho

Morpho

Morpho

Dependency

Dependency

Morpho

Dependency

Dependency

Morpho

Morpho

Morpho

Lemma

Morpho

Dependency

Morpho

Dependency

Constituency