sile-typesetter / sile

The SILE Typesetter โ€” Simonโ€™s Improved Layout Engine
https://sile-typesetter.org
MIT License
1.64k stars 96 forks source link

Unicode suplimental blocks (UTF16) not supported #551

Open alerque opened 5 years ago

alerque commented 5 years ago

There doesn't seem to be any way to output UTF16 range characters like this negative circled letter a: ๐Ÿ….

The SU.utf8char() and doesn't handle math in that range, and I've tried it manually with Lua strings:

SILE.typesetter:typeset("\xF0\x9F\x85\x90")
SILE.typesetter:typeset("\u{D83C}\u{DD50}")
SILE.typesetter:typeset("\u{D83C}".."\u{DD50}")
SILE.typesetter:typeset("\u{0001F150}")
SILE.typesetter:typeset("\u{1F150}")

One or the other of bidi.lua or unicode.lua dies with either not understanding the code point or not being able to break words.

alerque commented 5 years ago

I tried to put together a MWE for this, but can't replicate my results from the other day.

\begin{document}
\script[src=packages/unichar]
\font[family=Symbola]
\begin{raggedright}
These are just a unicode strings:
โ‘ โ’ถ๐Ÿ…

These are with the unichar package:
\unichar{U+2460}\unichar{U+24B6}\unichar{U+1F150}

These are from Lua strings with Unicode escapes:
\script{SILE.typesetter:typeset("\u{2460}\u{24B6}\u{1F150}")}
\end{raggedright}
\end{document}

image

Apparently my (much larger) document I was playing with the other day has something else fouling up the works because this seems jut fine out of the box.

P.S. I do not and never have done drugs.

alerque commented 5 years ago

Maybe I'm not smoking something after all. There is something wrong here because what works in the MWE often doesn't work in the real world. It works right up until you reach out to the supplemental block stuff and then I start seeing messages such as:

/usr/share/sile/packages/bidi.lua:205: attempt to index a nil value (field '?')

I'll keep trying to isolate what this actually is.

alerque commented 5 years ago

Setting \bidi-off in my real document solves this. The only catch is trying to make an MWE with \bidi-on doesn't fail.