Support single word rule composition

[patapizza]

Today Duckling doesn't allow to create multiple patterns to match against a single word. As a result, we can't compose rules/dimensions and have to duplicate the regexes.

This is the case for DE (e.g. here and here), NL (e.g. here), EL (e.g. here), and RU (e.g. here).

[chessai]

Yes, this is a somewhat annoying thing about hacking on duckling.

[emlautarom1]

I've been thinking about how we could approach this issue. My idea is to add a rule to each language that splits single words so we can reuse the rules that already exist.

---

Assume that we have Language X and it has rules for hello, world and intersection, so it correctly parses hello (with hello rule), world (with world rule), and hello world, world hello, hello hello, world world (with intersection rule).

What I would like to do is add a single-word decomposition rule that will handle helloworld, hellohello, worldhello and worldworld. The idea is that this rule will only split the word into subwords which then will be picked up by the intersection rule.

For example, rule single-word decomposition converts helloworld into hello world. Then, rule intersection picks hello world and works as expected.

The single-word decompositon rule would need to know in advance all possible subwords so then it can separate them. A rough implementation would be something like:

ruleSingleWordDecomposition :: Rule
ruleSingleWordDecomposition = Rule
  { name = "single-word decomposition"
  , pattern =
    [ regex "(\\S+)" -- we match 'single' words
    ]
  , prod = \tokens -> case tokens of
      (Token RegexMatch (GroupMatch (w:_)):_) -> do
        let rgx = "(hello|world)" :: String -- we need to compute in advance a big regex of possible "subwords"
        let ws = Text.pack <$> getAllTextMatches (show w =~ rgx)
        Just $ Token RegexMatch $ GroupMatch ws -- How do we return the "subwords" so other rules can pick them up?
      _ -> Nothing
  }

Note: we need to split the "single word" in Haskell (let ws = Text.pack <$> getAllTextMatches (show w =~ rgx)) since we can't do it with pure regexes AFAIK

In this case, our single-word decomposition rule knows that "single words" can be composed only from the "subwords" hello and world.

The main issue that I found with this is how to use the output of this rule as input for the remaining rules (they don't pick it up). So, even if we're able to build the required regex and split the "single word" into "subwords" I can't seem to redirect them to other rules.

I'm trying to avoid adding special cases or hacking into the internals of Duckling (Engine.hs and the like), so any hint would be appreciated.

[emlautarom1]

One quick note, the mentioned examples in the first comment aren't permalinks so I can't be sure to what they refer to. I understand that this issue is related to:

• https://github.com/facebook/duckling/issues/481
• https://github.com/facebook/duckling/issues/415

[emlautarom1]

@chessai Do you think that this could work?

[andhai]

Regarding @emlautarom1's interesting suggestion, I see two problems that should be considered before going this route.

First, decomposing German number words like "dreiundzwanzig" 'three.and.twenty' (23) can lead to unnwanted ambiguities because "drei und zwanzig" 'three and twenty' is also a grammatical phrase in German. Consider, for instance, the sentence "sie wählten zwischen dreiundzwanzig Kandidaten" 'they made a selection between twenty three candidates'. The problem is that the sentence "sie wählten zwischen drei und zwanzig Kandidaten" 'they made a selection between three candidates and twenty candidates'. Hence, the suggested decomposition makes the two sentences indistinguishable.

Similarly, if single-word decomposition is applied the compound "zweihundertjährige" 'two.hundred.year.old.PL' and the phrase "zwei hundertjährige" 'two hundred.year.old.PL' are converted to the same string. Thus, the difference between sentences like "sie fällten zweihundertjährige Eichen" 'they chopped oak trees that were two hundred years old' and "sie fällten zwei hundertjährige Eichen" 'they chopped two oak trees that were one hundred years old' is erased.

So the upshot is that we lose information by applying single-word decomposition.