multiword tokenization of stanza

[KoichiYasuoka]

Tamil tokenizer of stanza needs mwt model. For example, the word குதிரையும் is divided into two words:

>>> import stanza
>>> nlp=stanza.Pipeline(lang="ta",processors="tokenize,mwt")
>>> doc=nlp("காசுக்கு ஒரு குதிரையும் வேண்டும், காற்றைப் போலப் பறக்கவும் வேண்டும்.")
>>> print(doc)
[
  [
    {
      "id": 1,
      "text": "காசுக்கு",
      "misc": "start_char=0|end_char=8"
    },
    {
      "id": 2,
      "text": "ஒரு",
      "misc": "start_char=9|end_char=12"
    },
    {
      "id": [
        3,
        4
      ],
      "text": "குதிரையும்",
      "misc": "start_char=13|end_char=23"
    },
    {
      "id": 3,
      "text": "குதிரையைய்"
    },
    {
      "id": 4,
      "text": "உம்"
    },
    {
      "id": 5,
      "text": "வேண்டும்",
      "misc": "start_char=24|end_char=32"
    },
    {
      "id": 6,
      "text": ",",
      "misc": "start_char=32|end_char=33"
    },
    {
      "id": 7,
      "text": "காற்றைப்",
      "misc": "start_char=34|end_char=42"
    },
    {
      "id": 8,
      "text": "போலப்",
      "misc": "start_char=43|end_char=48"
    },
    {
      "id": 9,
      "text": "பறக்கவும்",
      "misc": "start_char=49|end_char=58"
    },
    {
      "id": 10,
      "text": "வேண்டும்",
      "misc": "start_char=59|end_char=67"
    },
    {
      "id": 11,
      "text": ".",
      "misc": "start_char=67|end_char=68"
    }
  ]
]

But thamizhi-udp does not use mwt model, thus the word disappears:

>>> import stanza
>>> nlp=stanza.Pipeline(lang="ta",processors="tokenize",tokenize_model_path="./models/ta_ttb_tokenizer.pt")
>>> doc=nlp("காசுக்கு ஒரு குதிரையும் வேண்டும், காற்றைப் போலப் பறக்கவும் வேண்டும்.")
>>> print(doc)
[
  [
    {
      "id": 1,
      "text": "காசுக்கு",
      "misc": "start_char=0|end_char=8"
    },
    {
      "id": 2,
      "text": "ஒரு",
      "misc": "start_char=9|end_char=12"
    },
    {
      "id": 4,
      "text": "வேண்டும்",
      "misc": "start_char=24|end_char=32"
    },
    {
      "id": 5,
      "text": ",",
      "misc": "start_char=32|end_char=33"
    },
    {
      "id": 6,
      "text": "காற்றைப்",
      "misc": "start_char=34|end_char=42"
    },
    {
      "id": 7,
      "text": "போலப்",
      "misc": "start_char=43|end_char=48"
    },
    {
      "id": 8,
      "text": "பறக்கவும்",
      "misc": "start_char=49|end_char=58"
    },
    {
      "id": 9,
      "text": "வேண்டும்",
      "misc": "start_char=59|end_char=67"
    },
    {
      "id": 10,
      "text": ".",
      "misc": "start_char=67|end_char=68"
    }
  ]
]

[sarves]

Hi @KoichiYasuoka

Yes yes, true. Thank you for pointing out this. Will incorporate it.

Thank you Sarves

[KoichiYasuoka]

I've just introduced thamizhi-udp in my diary to my Japanese colleagues. Stanza's original mwt model works with your pos model pretty well, and they can be connected with your parse.sh.