empty alignment group with TEI.2 tag

[monzug]

with the xml tag from perseus (e.g. www.perseus.tufts.edu/hopper/xmlchunk?doc=Perseus%3Atext%3A2008.01.0594%3Achapter%3D8 ) I do not get any error in tokenization but the alignment groups is empty. if I replace the TEI.2 with the one we have in alpheios (TEI xmlns="http://www.tei-c.org/ns/1.0" xmlns:py="http://codespeak.net/lxml/objectify/pytype" py:pytype="TREE") it works.

[irina060981]

It returns an empty result because Tokenization service returns an empty array with tokens. I think, I should add an error for that.

[monzug]

Can we support TEI.2?

[irina060981]

Can we support TEI.2?

@balmas , I believe it is an issue for a remote tokenizer?

[balmas]

yes. transferring.

[monzug]

an other sample of empty text is from following xml file (https://texts.alpheios.net/text/urn:cts:latinLit:phi0472.phi001.perseus-lat2/passage/1.1-1.10): see in edit mode

<?xml version="1.0" encoding="utf-8"?>

Pulsis urbe regibus prima pro libertate arma corripuit. Nam Porsenna rex Etruscorum ingentibus copiis aderat et Tarquinios manu reducebat.


[monzug]

to test also with two target texts (to upload file alignment-lat-lat-eng.tsv)

[irina060981]

I checked this case with TEI.2, the problem is that the following tag is not enough <TEI.2> should be <TEI.2 xmlns="http://www.tei-c.org/ns/1.0">

With this definition tokenization works properly. Would send this question to Bridgit too

[monzug]

yes, that's how I changed it to successfully align

On Fri, Feb 12, 2021 at 12:55 PM Sklyarova Irina notifications@github.com wrote:

I checked this case with TEI.2, the problem is that the following tag is not enough

should be With this definition tokenization works properly. Would send this question to Bridgit too — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or unsubscribe .

[irina060981]

I have updated tokenizer service to support TEI tags without namespace. How it checks:

1. First it searches for dts:fragment tag (it is for DTS texts)
2. If fails - it searches for the tag with namespace xmlns = "http://www.tei-c.org/ns/1.0"

This steps were before my update I have added

3. If fails - it searches for tag and gets text from it (and all inner nodes)

So now all these templates are rendered

<TEI>
<text>
this is body text
</text>
</TEI>

<TEI xmlns="http://www.tei-c.org/ns/1.0">  
<text>
this is body text
</text>
</TEI>

<TEI.2>
<text>
this is body text
</text>
</TEI.2>

<TEI.2 xmlns="http://www.tei-c.org/ns/1.0">
<text>
this is body text
</text>
</TEI.2>

[irina060981]

I found one edge case - if you try to tokenize xml with tei namespace that is not equal to "http://www.tei-c.org/ns/1.0", than it fails

I didn't find the way to make namespace less concrete , but I checked and didn't find other correct namespace for TEI.

If we want to support xml with other namespace - I should spend more time and some examples.

[monzug]

They all work now. This is about TEI.2 tag. if there are issues with other tags, we will enter a new github.

[irina060981]

ok, thank you, @monzug , I think that is really difficult to find all variants at once. But I will be able to do fixes based on examples