Error

[deepakjois]

I am running this on Mac OS X, but I assume the basic steps are the same.

Running lualatex with the amiri-sample.tex file inside the project folder, after copying the justenoughharfbuzz .so results in the following error.

I also tried going back a few revisions of the C file and recompiling and make sure it was not due to some new changes that were introduced. What commit in SILE are you using to compile the justenoughharfbuzz .so file?

lualatex examples/amiri-sample.tex
This is LuaTeX, Version beta-0.80.0 (TeX Live 2015) (rev 5238)
 restricted \write18 enabled.
(./examples/amiri-sample.tex
LaTeX2e <2015/01/01>
Babel <3.9l> and hyphenation patterns for 79 languages loaded.
(/usr/local/texlive/2015/texmf-dist/tex/latex/base/article.cls
Document Class: article 2014/09/29 v1.4h Standard LaTeX document class
(/usr/local/texlive/2015/texmf-dist/tex/latex/base/size10.clo)) (./harfbuzz.sty
(/usr/local/texlive/2015/texmf-dist/tex/lualatex/luacode/luacode.sty
(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/ifluatex.sty)
(/usr/local/texlive/2015/texmf-dist/tex/luatex/luatexbase/luatexbase.sty
(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/luatex.sty
(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/infwarerr.sty)
(/usr/local/texlive/2015/texmf-dist/tex/latex/etex-pkg/etex.sty)
(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/luatex-loader.sty
(/usr/local/texlive/2015/texmf-dist/scripts/oberdiek/oberdiek.luatex.lua)))
(/usr/local/texlive/2015/texmf-dist/tex/luatex/luatexbase/luatexbase-compat.sty
)
(/usr/local/texlive/2015/texmf-dist/tex/luatex/luatexbase/luatexbase-modutils.s
ty
(/usr/local/texlive/2015/texmf-dist/tex/luatex/luatexbase/luatexbase-loader.sty
(/usr/local/texlive/2015/texmf-dist/tex/luatex/luatexbase/luatexbase.loader.lua)
)
(/usr/local/texlive/2015/texmf-dist/tex/luatex/luatexbase/modutils.lua))
(/usr/local/texlive/2015/texmf-dist/tex/luatex/luatexbase/luatexbase-regs.sty)
(/usr/local/texlive/2015/texmf-dist/tex/luatex/luatexbase/luatexbase-attr.sty
(/usr/local/texlive/2015/texmf-dist/tex/luatex/luatexbase/attr.lua))
(/usr/local/texlive/2015/texmf-dist/tex/luatex/luatexbase/luatexbase-cctb.sty
(/usr/local/texlive/2015/texmf-dist/tex/luatex/luatexbase/cctb.lua))
(/usr/local/texlive/2015/texmf-dist/tex/luatex/luatexbase/luatexbase-mcb.sty
(/usr/local/texlive/2015/texmf-dist/tex/luatex/luatexbase/mcb.lua))))
lualatex(63925,0x7fff7578c000) malloc: *** error for object 0x10577e7f0: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
(./hf_fontload.lua)Abort trap: 6

[michal-h21]

The font config function was moved to standalone library in SILE recently. It was commit 4bcdefb6a6b5053134fb66049849f691503097ba, so any previous commit should work.

I should adapt the code for this change of course, maybe luaotfload font database could be used for font loading instead of SILE's library.

Anyway, your error log indicates that there is a problem in loading of justenoughharfbuzz.so, I can't replicate that on TL 2015 on Linux.

[deepakjois]

Thank you. Like you noticed on StackExchange, I can’t even get a simple Lua C module to work on my computer right now. Will update back here if I have anything new to report.

Good luck on this project. It will be really great if you are able to use Harfbuzz cleanly inside of LuaTeX. Right now, there is a huge difference in quality of Arabic (Urdu in my case specifically) rendering between LuaLaTeX and XeLaTeX (which uses Harfbuzz). It is a bit unfortunate, because XeTeX is not really being maintained. Moreover, I would really like to use the Lua integration features of LuaTeX and not deal with macro programming.

For e.g, here is a screenshot for LuaLaTeX:

vs. XeLaTeX:

[deepakjois]

This is a problem specific to the Mac and to the way the .so file is built. The justenoughharfbuzz.so file is linked to the liblua that comes with the Lua installation on my Mac. That is what is likely causing this issue.

Anyway, when I get the time I will try a different approach. It seems a more robust option to copy over the source of justenoughharfbuzz.so and compile it on our own linking to harfbuzz and the lua headers that come with LuaTeX. A simple build script will be needed.

I started to do this here: https://github.com/deepakjois/tex-sandbox/tree/master/luatex-lua-c-bridge

[michal-h21]

Have you succeed in your approach? Have you tried LuaTeX 0.85? It seems that it should fix Mac issues with missing symbols, am I right? Does your testing harfbuzz.so work at least with texlua?

[deepakjois]

@michal-h21 Yes, LuaTeX 0.85 does fix Mac issues. But it is incompatible with TeXLive 2015. So I cannot use it. However, I managed to compile a fresh copy of LuaTeX 0.80 (the version included with TeXLive 2015) with the right flags, so that is working.

However, the problem is also with the way justenoughharfbuzz.so is compiled on the Mac. It links with the lualib on my system when it is built (for some reason it is built with the -llua option passed to the linker). This is the root cause of the error above. The Lua page for building modules explicitly states that we should not compile modules that way. (see section titled ‘Do Not Link Modules to the Lua Core Libraries’).

So what I need to do is to copy justenoughharfbuzz.c and build it myself with the right flags. I have not got to that yet. Will post here when I do.

[deepakjois]

Ok, I got justenoughharfbuzz.c to build correctly on my Mac. However, the changes you made in Nov (commit acb5db1ebcf8bad05f4f9a34cf6ee3d996c02a52 onwards) did not work for me. When I run with those changes, I get the unshaped output. I had to revert to the to commit 90a78ebb39e7873baefcd7179cce85c31ed579c3 and the shaping worked correctly for me.

Here is the working source tree: https://github.com/deepakjois/luatex-harfbuzz-shaper

[davidcarlisle]

On 24 November 2015 at 03:49, Deepak Jois notifications@github.com wrote:

@michal-h21 https://github.com/michal-h21 Yes, LuaTeX 0.85 does fix Mac issues. But it is incompatible with TeXLive 2015. So I cannot use it.

You just need a suitable .ini file to make the formats that knows about the renamed primitives, you can find code for latex here

https://github.com/josephwright/luatex85

David

[deepakjois]

@davidcarlisle Thank for that link. I did see it on the LuaTeX mailing list yesterday. I want to try it, but some things were not entirely clear:

1. I am new to this, so I just wanted to make sure that the process described in the README is reversible so that I can revert back to the standard TeXLive binary in case I face any issues. Should I just backup the old files and when I want to get back to the standard TeXLive binary, I will need just copy back the old files and rerun fmtutil-sys command?
2. Where exactly in my local tex/latex tree should I place the files ltluatex.lua, luatex85.sty, dvilualatex.ini and lualatex.ini? On my mac I see that the ini files are under texmf-dist/tex/latex/latexconfig/.

[deepakjois]

@michal-h21 I was mistaken earlier. I got confused by the PDF that is already present in the source tree. Although everything works and the PDF files are generated. I am not seeing the output similar to the PDF files you have included. I am seeing unshaped nodes when I generate a PDF from the amiri-sample.tex file. So clearly something is wrong on the mac. I will need to investigate more.

[davidcarlisle]

On 24 November 2015 at 08:25, Deepak Jois notifications@github.com wrote:

we can't really offer support for this yet as it's not set up for end user use and teh details of exactly what changes are required for 0.85 are still being worked out, however.....

@davidcarlisle https://github.com/davidcarlisle Thank for that link. I did see it on the LuaTeX mailing list yesterday. I want to try it, but some things were not entirely clear:

1. I am new to this, so I just wanted to make sure that the process described in the README is reversible so that I can revert back to the standard TeXLive binary in case I face any issues. Should I just backup the old files and when I want to get back to the standard TeXLive binary, I will need just copy back the old files and rerun fmtutil-sys command?

yes (or you could put them in a new texmf tree and arrange that tex looks in the new one first, but that's more complicated so simply backing up teh originals and putting the test versions "in place" is easiest, for both these ini files and for the luatex binary itself.

1. Where exactly in my local tex/latex tree should I place the files ltluatex.lua, luatex85.sty, dvilualatex.ini and lualatex.ini? On my mac I see that the ini files are under texmf-dist/tex/latex/latexconfig/.

easiest if you put the ini and lua files overwriting the existing places (kpsewhich ltluatex.lua for example will show you which tex is using) the luatex85.sty file (which currently you will need in more or less any document) you could put anywhere in your local input tree, or just have it in your test directory.

David

[deepakjois]

@davidcarlisle Thank you very much. I understand your point about not offering support. I realise I am on my own if things go wrong :). That’s the reason I wanted to make sure I was taking the necessary precautions before using the code.

@michal-h21 Could you please tell me what this line of code does: https://github.com/michal-h21/luatex-harfbuzz-shaper/blob/master/hb_fontload.lua#L62

I cannot figure out where the property hbfont.face comes from. It isn’t set anywhere in the C code, as far as I can see. It seems that this property is crucial the further functioning of the harfbuzz shaper. If this property is nil (as is the case when I run it) the whole thing short-circuits and shaper function (_shape) is not called at all.

[deepakjois]

@michal-h21 Also in this line: https://github.com/michal-h21/luatex-harfbuzz-shaper/blob/master/hb_fontload.lua#L169

You seem to be passing seven arguments to the harfbuzz._shape function, but the C function is expecting 8 arguments. Even SILE calls the function using 8 arguments.

[michal-h21]

I probably should have done the recent changes in a testing branch. New version of justenoughharfbuzz doesn't include font searching function, so I began to incorporate code from luaortload. This will have advantage that even fonts in TeX tree will be supported. Moreover, fonts can be declared using luaotfload syntax. In the meantime, new version of luaotfload was released and I suspect that the libraries I used changed, I haven't checked it yet.

Anyway, at the moment it is safest to use just older version of luatex shaper and justenoughharfbuzz

On Tue, Nov 24, 2015 at 6:19 AM, Deepak Jois notifications@github.com wrote:

Ok, I got justenoughharfbuzz.c to build on my Mac. However, the changes you made in Nov (commit acb5db1 https://github.com/michal-h21/luatex-harfbuzz-shaper/commit/acb5db1ebcf8bad05f4f9a34cf6ee3d996c02a52 onwards) did not work for me. When I run with those changes, I get the unshaped output. I had to revert to the to commit 90a78eb https://github.com/michal-h21/luatex-harfbuzz-shaper/commit/90a78ebb39e7873baefcd7179cce85c31ed579c3 and the shaping worked correctly for me.

Here is the working source tree: https://github.com/deepakjois/luatex-harfbuzz-shaper

— Reply to this email directly or view it on GitHub https://github.com/michal-h21/luatex-harfbuzz-shaper/issues/1#issuecomment-159152051 .

[michal-h21]

The problem is that SILE's Harfbuzz interface is constantly changing. I have some health issues past few weeks, so I am not really productive in adapting to these changes.

it would be nice if justenoghharfbuzz was standalone library, available for install through Luarocks for instance. but this may happen once it is stable.

[deepakjois]

@michal-h21 No problem. I am now using the justenoughharfbuzz.c source from commit simoncozens/sile@9ca00679f4de29a292b408fb7ea5321391334993 (it is after this point that the source seems to have diverged and become incompatible with the shaper as it is written right now.)

Now it works as expected. I will close the issue now. You can view the changes I made here deepakjois/luatex-harfbuzz-shaper@16678afab5f2717d2a10194264eed2ab9b00590f