ufal / ParCzech

ParCzech is a project on compiling Czech parliamentary data into annotated corpora.

https://ufal.mff.cuni.cz/parczech

0 stars 1 forks source link

Fix tokenizer #14

Closed matyaskopp closed 4 years ago

matyaskopp commented 4 years ago

[x] Don't split text to sentence if <note> is included (Note can be inside sentence)
[x] Don't add space before ␣</s> but after sentence end

enchantement

[x] allow list of node types that can be tokenized and is included in sentence:

...... 
tisk <ref ana="#parla.print" n="7/1001 source="https://...T=1001&O=7">1001</ref>/8 
......

<s>
...... 
<w>tisk</w> <ref ana="#parla.print" n="7/1001 source="https://...T=1001&O=7"><w>1001</w></ref><pc>/</pc><w>8</w>
......
</s>