search

MotherTongues / mothertongues-UI

1 stars 2 forks source link

fix(search): if substitutions are defined, we should tokenize input based on those as well #17

Closed roedoejet closed 5 months ago

roedoejet commented 5 months ago

Previously, we tokenized input based on the alphabet, to allow weighted multigraph substitutions, but if there are substitutions defined that are not in the alphabet, they would be missed. So, we should tokenize on the union of the alphabet and the substitution targets.

github-actions[bot] commented 5 months ago

PR Preview Action v1.4.7 :---: :rocket: Deployed preview to https://MotherTongues.github.io/mothertongues-UI/pr-preview/pr-17/ on branch gh-pages at 2024-04-24 21:33 UTC

dhdaines commented 5 months ago

Ah - is this what was causing the Michif substitutions not to work correctly?

roedoejet commented 5 months ago

Ah - is this what was causing the Michif substitutions not to work correctly?

Yes I believe so. At least with this change, it seems the entries Carmen mentioned in her email are being returned properly now.