SILE vs. TeX hyphenation patterns

[Omikhleia]

Things not addressed in #2102:

• Things where we have our own upgrades, so likely won't change?
  • Turkish (tr) : many changes by @alerque
  • Esperanto (eo): solution by @ctrlcctrlv differing from what TeX currently has
• Things that might have to wait for a better implementation of BCP47...
  • en-GB / en-US (but see also just below, last item)
  • zh-Latn-pinyin
  • mul-ethi (Ethiopic)
  • nb/nn variants (Norsk, we do have them though a refactor could be neat eventually)
  • several la variants (liturgic/classic Latin)
  • mn (Mongolian in Cyrl script or Cyrl-x-lmc for Xalx Mongolian variant)
  • sh (Cyrl vs. Latn, Serbo-Croatian, now deprecated?)
• Things I am not sure of...
  • our "el" vs. TeX's "grc" is problematic and the situation is messy = 0.16.x ?
    • It seems SILE's "el" patterns were added by Simon in https://github.com/sile-typesetter/sile/commit/c6e8d0b86edbb40b8e36a4b2d90773117a6474e8 ("Oops, forgot to include these") directly on master in Feb. 2014. The TeX "grc" patterns were updated in May 2016 ("added support for curly beta") and after better checking, I confirm that's our differences with it.
    • So my assumption is that Simon added the "grc" patterns of that time as "el" -- and this was likely wrong. It should have been kept as "grc", for Ancient Greek (→ added to my first comment above, we could safely add it too under that name Nah boustrophedon then pees on itself).
    • As for "el", we should likely alias it to "el-monoton": It seems to me that Modern Greek is monotonic in most contexts since the 1982 reform. (We should state it the documentation too...) = Strictly speaking, we should postpone it to 0.16.x as it could make rendered documents different?
    • However, the boustrophedon package would kill any standard "grc"...
• Things where we differ but may change for better
  • pt (Portuguese): We have @jodros 's updates, but an incoming PR might improve upon it: https://github.com/hyphenation/tex-hyphen/pull/62 Should we wait or go for an early adoption? = 0.16.x or earlier ?
  • es (Spanish): Lots of differences with our patterns = 0.16.x ?
  • th (Thai): Lots of differences with our patterns, and no idea where the latter come from. = (?)
  • bg (Bulgarian) = Lots of differences with our patterns, and no idea where the latter come from. = 0.16.x ? Note the TeX patterns were updated in 2017 so whatever we had might be outdated (?)
  • de (German) = We have something, TeX has several variants depending on orthography reforms, frankly I don't know.
  • en (English, see also above): Currently we are based on en-US, but we lack a whole "additional patterns" that were not present in the original TeX file and were added later (though at an earlier stage than our import). My understanding is that "old" TeX being memory constrained, these were absent from the original implementation. But it's pretty unclear why we don't have the extra patterns... = 0.16.x ? as it could make rendered documents different...

Originally posted by @Omikhleia in https://github.com/sile-typesetter/sile/issues/2102#issuecomment-2336669874

[jodros]

pt (Portuguese): We have @jodros 's updates, but an incoming PR might improve upon it: https://github.com/hyphenation/tex-hyphen/pull/62 Should we wait or go for an early adoption? = 0.16.x or earlier ?

Glad to know about @leolca's work! I can't see reasons to wait.

[Omikhleia]

Glad to know about @leolca's work! I can't see reasons to wait.

Yep, it really made my day when I read their (proposed) TUGboart article. It's scholarly, plenty of attempts, plenty of tables with numbers... It"s very interesting (even the things that they tried, judged inefficient, etc.)