albbas
commented
8 years ago Comment 11214
Date: 2016-03-02 13:52:29 +0100 From: Kevin Brubeck Unhammer <<unhammer+apertium>>
For Bug #2160, we want certain kinds of MWE ambiguity to be at least partly (the linguistic part) represented on the FST level. We will likely also need new language-general mechanisms regarding how to do formatting and tokenising.
What we want:
-
Treat "good morning" as ambiguous between a single interjection "
", and an adj " " followed by noun " " as in "this is a good morning". -
Similarly treat "3." as ambiguous between single ordinal, and numeral followed by punctuation; "<3.>" vs "<3>" "<.>".
-
Not have to treat "we are" as ambiguous between single and multiple cohorts – it's always exactly two; and similarly not look at "we 3." ambiguous between "<we 3.>" and "
" "<3.>", but ambiguous between " " "<3.>" and " " "<3>" "<.>".
I've looked at using pmatch for this by simply repeating the lexicon, e.g.
Define morphology @bin"analyser-disamb-gt-desc.hfst" ; Define blank Whitespace | Punct ; Define morphoword morphology LC([blank | #]) RC([blank | # ]); Define morphowords morphology % :%# morphology LC([blank | #]) RC([blank | # ]); Define token [ morphoword | morphowords ] EndTag(token); regex token ;
Consider "dalle go" – this might be unambiguous, but treat it as ambiguous for this example. The above pmscript will give
$ echo dalle go | hfst-tokenise --cg -kawu: ana.bin
"
but of course that fails when there's a word before:
echo 'go dan dihte' | hfst-tokenise --cg -kawu: ana.bin
"
since now it looks at the MWE ambiguity between "go" and "dalle go" – and pmatch always wants the longest possible match.
This issue seems solvable by intersecting the forms of the morphology with the repeated morphology, although I couldn't get this to work within the pmscript (one would think [morpho2words & [morphology].u] would work, but no such luck). It seems to work with plain HFST commands though:
Makefile:
space.hfst: echo '% :%#' | hfst-regexp2fst -o $@
repeated.hfst: analyser-disamb-gt-desc.hfst space.hfst hfst-concatenate -1 $< -2 space.hfst -o analyser-space.hfst hfst-concatenate -F -1 analyser-space.hfst -2 $< -o analyser-space-analyser.hfst hfst-invert analyser-space-analyser.hfst -o generator-space-generator.hfst hfst-project -p upper $< -o forms.hfst hfst-compose-intersect -1 generator-space-generator.hfst -2 forms.hfst -o generator-repeated.hfst hfst-invert generator-repeated.hfst -o $@ echo dan dihte | hfst-lookup $@ echo go dan dihte | hfst-lookup $@
pmscript:
Define morphology @bin"analyser-disamb-gt-desc.hfst" ; Define repeated @bin"repeated.hfst" ; Define blank Whitespace | Punct ; Define morphoword morphology LC([blank | #]) RC([blank | # ]); Define morphowords repeated LC([blank | #]) RC([blank | # ]); Define token [ morphoword | morphowords ] EndTag(token); regex token ;
This looks closer to what we want:
$ echo dalle go | hfst-tokenize --cg -kawu: rep.pmhfst
"
| $ echo go dalle go | hfst-tokenize --cg -kawu: rep.pmhfst
" |
|---|
"
The repeated.hfst took quite a lot of time+ram+disk space, but perhaps that can be optimised – the final file seems to be just 28M.
Are the problems with this method not shown above?
If we add an error-tagged entry in the noun compounding lexicon that inserts a space before the compound border, will it work fine for catching compound-space errors (while still showing the ambiguity with non-errors that are simply two nouns)?
This issue was created automatically with bugzilla2github
Bugzilla Bug 2162
Date: 2016-03-02T13:52:29+01:00 From: Kevin Brubeck Unhammer <<unhammer+apertium>> To: Sjur Nørstebø Moshagen <>
CC: lene.antonsen, linda.wiechetek, sjur.n.moshagen, thomas.omma, trond.trosterud
Blocker for: #2160 Last updated: 2017-09-08T14:40:47+02:00