No ligatures with Arabic

jbezos commented 5 years ago

I'd like to see if the bidi algorithm in babel works with harftex, and which changes would be necessary.

So, I've downloaded the binaries for Windows from http://ctan.ijs.si/mirror/w32tex/current/harftex-w32.tar.xz, but the sample files doesn't show the text correctly - only isolated forms. I've added the following features, and there are a few ligatures, but not all: +fina;+init;+medi;+isol;+med2;+mset;+rtla;+curs;+falt. I've also installed the files available here, with no luck.

khaledhosny commented 5 years ago

Please attach the full log of one of the files you processed.

jbezos commented 5 years ago

Here is the log for is a shortened version of test.tex (only the first paragraph with Amiri).


*test.tex
(./test.tex (./harfload.sty
(c:/----/texmf-dist/tex/luatex/luaotfload/luaotfload.sty
(c:/----/texmf-dist/tex/latex/base/ltluatex.tex
File: ltluatex.tex 2018/10/21 v1.1i LuaTeX support for plain TeX (core) 
\e@alloc@attribute@count=\count277
\e@alloc@ccodetable@count=\count278
\e@alloc@luafunction@count=\count279
\e@alloc@whatsit@count=\count280
\e@alloc@bytecode@count=\count281
\e@alloc@luachunk@count=\count282
\catcodetable@initex=\catcodetable1
\catcodetable@string=\catcodetable2
\catcodetable@latex=\catcodetable3
\catcodetable@atletter=\catcodetable4
)
Lua module: luaotfload-main 2019-05-18 2.97 luaotfload entry point
Lua module: luaotfload-init 2019-05-18 2.97 luaotfload submodule / initializatio
n
Lua module: lualibs 2019-05-18 2.65 ConTeXt Lua standard libraries.
Lua module: lualibs-extended 2019-05-18 2.65 ConTeXt Lua libraries -- extended c
ollection.
Lua module: luaotfload-log 2019-05-18 2.97 luaotfload submodule / logging
Lua module: luaotfload-parsers 2019-05-18 2.97 luaotfload submodule / filelist
Lua module: luaotfload-configuration 2019-05-18 2.97 luaotfload submodule / conf
ig file reader
luaotfload | conf : Root cache directory is C:/----/texmf-var/lu
atex-cache/generic/names.
luaotfload | init : Loading fontloader “fontloader-2019-05-18.lua” from kpse
-resolved path “c:/----/texmf-dist/tex/luatex/luaotfload/fontl
oader-2019-05-18.lua”.
Lua-only attribute luaotfload@state = 1
Lua-only attribute luaotfload@noligature = 2
Lua-only attribute luaotfload@syllabe = 3
luaotfload | init : Context OpenType loader version “3.108”
Inserting `luaotfload.node_processor' at position 1 in `pre_linebreak_filter'.
Inserting `luaotfload.node_processor' at position 1 in `hpack_filter'.
Lua module: luaotfload-loaders 2019-05-18 2.97 luaotfload submodule / callback h
andling
Inserting `luaotfload.define_font' at position 1 in `define_font'.
Lua module: luaotfload-database 2019-05-18 2.97 luaotfload submodule / database
Lua module: luaotfload-colors 2019-05-18 2.97 luaotfload submodule / color
Lua-only attribute luaotfload_color_attribute = 4
Lua module: luaotfload-resolvers 2019-05-18 2.97 luaotfload submodule / resolver
s
luaotfload | conf : Root cache directory is C:/----/texmf-var/lu
atex-cache/generic/names.
Lua module: luaotfload-features 2019-05-18 2.97 luaotfload submodule / features
Lua module: luaotfload-letterspace 2019-05-18 2.97 luaotfload submodule / color
Lua module: luaotfload-auxiliary 2019-05-18 2.97 luaotfload submodule / auxiliar
y functions
Inserting `luaotfload.aux.set_sscale_dimens' at position 1 in `luaotfload.patch_
font'.
Inserting `luaotfload.aux.patch_cambria_domh' at position 2 in `luaotfload.patch
_font'.
Inserting `luaotfload.aux.fixup_fontdata' at position 1 in `luaotfload.patch_fon
t_unsafe'.
Inserting `luaotfload.aux.set_capheight' at position 3 in `luaotfload.patch_font
'.
Inserting `luaotfload.rewrite_fontname' at position 4 in `luaotfload.patch_font'
.
luaotfload | main : initialization completed in 0.125 seconds
)
Inserting `Harf get_glyph_string callback' at position 1 in `get_glyph_string'.
Inserting `Harf get_char_tounicode callback' at position 1 in `get_char_tounicod
e'.
Inserting `Harf wrapup_run callback' at position 1 in `wrapup_run'.
Inserting `Harf pre_output_filter callback' at position 1 in `pre_output_filter'
.
Removing  `luaotfload.node_processor' from `hpack_filter'.
Inserting `Harf hpack_filter callback' at position 1 in `hpack_filter'.
Inserting `luaotfload.node_processor' at position 2 in `hpack_filter'.
Removing  `luaotfload.node_processor' from `pre_linebreak_filter'.
Inserting `Harf pre_linebreak_filter callback' at position 1 in `pre_linebreak_f
ilter'.
Inserting `luaotfload.node_processor' at position 2 in `pre_linebreak_filter'.)
luaotfload | db : Font names database loaded from C:/----/texmf-
var/luatex-cache/generic/names/luaotfload-names.luc
Overfull \hbox (1.7245pt too wide) in paragraph at lines 13--26
\amiriten لفظا ومعنى بالنحت، والابدال، والقلب، 
حتى صارت ألفاظا مستقلة وتكوّنت الأفعال، وا
لأسماء، والحروف|

\hbox(10.10742+5.35156)x469.75499, glue set - 1.0, direction TRT
.\begindir TLT
.\begindir TRT
.\amiriten ل
.\amiriten ف
.\amiriten ظ
.etc.

Overfull \hbox (5.97255pt too wide) in paragraph at lines 27--29
[]\amiriten المحتوى الذي تصل إليه (في حالة عدم اس
تخدام مواقع مؤمنة عن طريق بروتوكول نقل الن
ص التشعبي|

\hbox(7.91992+5.35156)x469.75499, glue set - 1.0, direction TRT
.\localpar
..\localinterlinepenalty=0
..\localbrokenpenalty=0
..\localleftbox=null
..\localrightbox=null
.\begindir TLT
.\begindir TRT
.\hbox(0.0+0.0)x20.0, direction TRT
.\amiriten ا
.etc.

[1{c:/----/texmf-var/fonts/map/pdftex/updmap/pdftex.map}])

Here is how much of HarfTeX's memory you used:
 67 strings out of 496358
 100000,35830 words of node,token memory allocated
 241 words of node memory still in use:
   2 hlist, 1 rule, 3 kern, 2 glyph, 1 attribute, 23 glue_spec, 1 attribute_list
 nodes
   avail lists: 2:20,3:6,4:6,5:62,6:2,7:1071,9:23,10:5
 2200 multiletter control sequences out of 65536+600000
 52 fonts using 3383205 bytes
 61i,4n,30p,224b,34s stack positions out of 5000i,500n,10000p,200000b,100000s
<c:/----/texmf-dist/fonts/truetype/public/amiri/amiri-bold.ttf><
c:/----/texmf-dist/fonts/truetype/public/amiri/amiri-regular.ttf
><c:/----/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb>
Output written on test.pdf (1 page, 24272 bytes).

khaledhosny commented 5 years ago

I don’t see any thing suspecios. Can you post the exact test.tex file you are using?

jbezos commented 5 years ago

I deleted accidentally the first lines in the log. They might be relevant:

This is HarfTeX, Version 0.1.2 (TeX Live 2019/W32TeX)  (format=harftex 2019.5.5)  27 MAY 2019 16:27
 restricted system commands enabled.
**test.tex
(./test.tex (./harfload.sty

Here is the file:

\pdfvariable suppressoptionalinfo \numexpr1+32+64+128+512\relax

\tracingstats=1
\tracingonline=1

\input harfload.sty

\font\amiriten="[amiri-regular.ttf]" at 10pt
\font\amiritenb="[amiri-bold.ttf]" at 10pt

\begingroup\pardir TRT\textdir TRT
\amiriten
هذا كتاب صغير في بحث جديد، تنبّهنا له ونحن ننشر الطبعة الثانية من كتابنا الفلسفة
اللغوية لأنّ موضوعه تابع لموضوعنا. أو هي خطوة ثانية في تاريخ اللغة باعتبار
منشأها وتكونها ونموها. فالفلسفة اللغوية تبحث في كيف نطق الانسان الأول، وكيف
نشأت اللغة وتولّدت الألفاظ من حكاية الأصوات الخارجية، ك{\amiritenb قصف الرعد}، وهبوب الرياح،
والقطع والكسر، وحكاية التف والنفخ والصفير ونحوها. ومن المقاطع الطبيعية التي
ينطق بها الانسان غريزيا كالتأوه، والزفير. وكيف تنوّعت تلك الأصوات لفظا ومعنى
بالنحت، والابدال، والقلب، حتى صارت ألفاظا مستقلة وتكوّنت الأفعال، والأسماء،
والحروف وصارت اللغة على نحو ما هي عليه.  وأما تاريخ اللغة فيتناول النظر في
ألفاظها وتراكيبها، بعد تمام تكونها، فيبحث فيما طرأ عليهما من التغيير والتجدد أو
الدثور، فيبين الألفاظ والتراكيب التي دثرت من اللغة بالاستعمال، وما قام مقامها
من الألفاظ الجديدة، والتراكيب الجديدة، بما تولّد فيها، أو اقتبسته من سواها، مع
بيان الأحوال التي قضت بدثور القديم وتولد الجديد، وأمثلة مما دثر، أو أهمل، أو
تولّد، أو دخل.

المحتوى الذي تصل إليه (في حالة عدم استخدام مواقع مؤمنة عن طريق بروتوكول نقل
النص التشعبي الآمن {\textdir TLT HTTPS}).
\par\endgroup\vfill\eject
\bye

khaledhosny commented 5 years ago

Hmm, still nothing out of ordinary. What is the output of harftex --credits?

jbezos commented 5 years ago

It seems to be a problem with the binary 0.1.2. See https://tex.stackexchange.com/questions/493180/how-to-install-harftex-on-tex-live/493185?noredirect=1#comment1245498_493185 . So very likely this issue must be moved to harftex.

khaledhosny commented 5 years ago

harftex --credits would show the versions of used library, and I suspect that a different version of HarfBuzz is used in the w32tex binaries than I have in my own builds, and that might be causing some problem.

jbezos commented 5 years ago

Here is:

This is HarfTeX, Version 0.1.2 (TeX Live 2019/W32TeX)

HarfTeX merges and builds upon (parts of) the code from these projects:

luatex    : Hans Hagen, Hartmut Henkel, Taco Hoekwater, Luigi Scarso
tex       : Donald Knuth
etex      : Peter Breitenlohner, Phil Taylor and friends
omega     : John Plaice and Yannis Haralambous
aleph     : Giuseppe Bilotta
pdftex    : Han The Thanh and friends
kpathsea  : Karl Berry, Olaf Weber and others
lua       : Roberto Ierusalimschy, Waldemar Celes and Luiz Henrique de Figueiredo
metapost  : John Hobby, Taco Hoekwater, Luigi Scarso, Hans Hagen and friends
pplib     : Pawe\l Jackowski
fontforge : George Williams (partial)

Compiled with libpng 1.6.37; using 1.6.37
Compiled with lua version 5.3.5
Compiled with mplib version 2.00
Compiled with zlib 1.2.11; using 1.2.11
Compiled with harfbuzz 2.5.0; using 2.5.0

And the libraries:

09/07/2015  15:32            18.624 api-ms-win-core-file-l1-2-0.dll
09/07/2015  15:32            18.624 api-ms-win-core-file-l2-1-0.dll
09/07/2015  15:32            21.184 api-ms-win-core-localization-l1-2-0.dll
09/07/2015  15:32            19.136 api-ms-win-core-processthreads-l1-1-1.dll
09/07/2015  15:32            19.136 api-ms-win-core-synch-l1-2-0.dll
09/07/2015  15:32            18.624 api-ms-win-core-timezone-l1-1-0.dll
09/07/2015  15:32            22.720 api-ms-win-crt-convert-l1-1-0.dll
09/07/2015  15:32            19.136 api-ms-win-crt-environment-l1-1-0.dll
09/07/2015  15:32            20.672 api-ms-win-crt-filesystem-l1-1-0.dll
09/07/2015  15:32            19.648 api-ms-win-crt-heap-l1-1-0.dll
09/07/2015  15:32            19.136 api-ms-win-crt-locale-l1-1-0.dll
09/07/2015  15:32            29.376 api-ms-win-crt-math-l1-1-0.dll
09/07/2015  15:32            26.816 api-ms-win-crt-multibyte-l1-1-0.dll
09/07/2015  15:32            19.648 api-ms-win-crt-process-l1-1-0.dll
09/07/2015  15:32            23.232 api-ms-win-crt-runtime-l1-1-0.dll
09/07/2015  15:32            24.768 api-ms-win-crt-stdio-l1-1-0.dll
09/07/2015  15:32            24.768 api-ms-win-crt-string-l1-1-0.dll
09/07/2015  15:32            21.184 api-ms-win-crt-time-l1-1-0.dll
09/07/2015  15:32            19.136 api-ms-win-crt-utility-l1-1-0.dll
25/05/2019  07:25         5.790.208 harftex.dll
18/04/2019  06:03        27.534.336 icudt64.dll
12/04/2019  15:19           170.496 lua53.dll
09/06/2016  15:46           440.120 msvcp140.dll
18/02/2011  16:40           773.968 msvcr100.dll
09/07/2015  15:33           900.288 ucrtbase.dll
09/06/2016  15:46            83.784 vcruntime140.dll

khaledhosny commented 5 years ago

OK, so as I suspected the w23tex binary is using HarfBuzz 2.5.0 and I’m still using 2.4.0. I think I know what changed in HarfBuzz that might be causing trouble here, I’ll update to HarfBuzz 2.5.0 and check.

khaledhosny commented 5 years ago

Patch sent to TeX Live mailing list, there is small issue in the HarfBuzz build scripts in TeX Live that was masked in previous releases.

khaledhosny commented 5 years ago

Can you try with the latest w32tex binary?

jbezos commented 5 years ago

It works. Now I'm adapting babel to the harf renderer (and I was able to typeset correctly some documents, after a few minor changes).

khaledhosny commented 5 years ago

Great.

khaledhosny commented 5 years ago

At some point I’d like to handle bidi in the renderer in a way or another (potentially using ICU), and I understand that babl has some bidi support as well, so I need to see if putting it in the renderer and make it it on by default is a better approach or if it would cause any conflicts.

jbezos commented 5 years ago

Babel doesn't impose any bidi algorithm. It must be set explicitly as a package option (currently is bidi=basic for luatex). So, nothing prevents using other algorithms.

jbezos commented 5 years ago

Thinking about this, I was wondering if harftex provides (or will provide) some access to the ICU API with lua.

khaledhosny commented 5 years ago

Yes, it will probably be in the form of an icu module in Lua that exposes some of ICU APIs, though I haven’t done any work on this area yet.

jbezos commented 5 years ago

Definitely, harftex looks great. Thank you.

khaledhosny / harftex

No ligatures with Arabic #2