Support LuaLaTex

[liuxinyu95]

Summary

On 4/25/2023, we experienced a build break for the PDF book for 3 days. The root cause was because the delayed answers to the exercise exceeded the memory limitation (5M) of XeLaTex in Linux platform. I turned on the option to delay populating answers at the end of the book, LaTex had to cache the answers in memory unless meet a shipoutAnswer command. As I added more answers, the cached content finally could not be held in memory any more (with total 14 chapters, a preface, and 2 appendices). As the mitigation, I temporarily distributed the answers as the last section of each chapter (force shipoutAnswer in every chapter). This prevented memory bloating, but the reading experience is supplement to have all answers together after the main content. I am working on a long-term solution to this issue.

Timeline

• On 4/25: A new answer about DFS Maze solution with stack caused delayed answer exceeded XeLaTex limit, causing build break in github workflow.
• After hours, I received build break notification. From the build log, it was out of memory. I randomly reduced some label/refs, and questions in exercise/answers. But it didn't work;
• On 4/26: I though the out of memory issue might be caused by big tikz/pgf figures. Followed recommendation, I tried externalize tikz/pgf figures. It reduced some memory, but still exceeded finally. In prelude.stl:

\usetikzlibrary{external,patterns,matrix,positioning,shapes,arrows}
\tikzexternalize[prefix=img/]

In Makefile:

TEX_FLAGS = -shell-escape
%.pdf: %.tex; latexmk -cd -xelatex $(TEX_FLAGS) $<

• I tried to switch from XeLaTex to LuaLaTex as it can dynamically allocate memories. However, I realized it need more change as the xeCJK pkg can't work with LuaLaTex. I need a quick mitigation.
• On 4/27, I established a cloud desktop in Linux, with the latext TeXLive, split out fix-linux-bb branch, tried to reproduce the issue.
• I found the eps files can't be recognized well in the TeXLive version. I replaced most eps figures with PNG (exported from drawio) in chapter 14, and found that the last chapter could be build standalone successfully. The build break only happened to the entire book build.
• I applied 'binary search' to comment out newly added answers, but didn't find any infinite populated content.
• Finally, I came to the question: 'How does the delayed answer work?' After check the exercise pkg doc, I found this:

More precisely, the answers are stored in a vertical box. When \shipoutAnswer is encountered, this box is emptied and its contents is placed in the main vertical list.

• At this point, I realized as I added more exercise/answers it will eventually exceed memory in the future with this settings.
• I forced a shipoutAnswer in chapter 13, build pass.
• I temporarily distributed answers to each section.

Future Solution

Given it's my plan to add more and more exercise/answers. Current 'delayed answer' approach is not scale with any fixed memory settings. If we want to achieve the good reader experience of continued main content, the long-term solution is to switch to LuaLaTex. However, this need careful changes, because:

1. CTex depends on xeCJK, but the community decided not to port CJK to LuaLaTex, but adopted luatexja, see this.
2. I have different default font settings for CN/EN on each OS (e.g., STSong in Mac OS X, Noto CJK for Linux, and SimSun for Windows). They depend on setCJKXxxFont.

I am going to take stepped approach to iteratively make this change:

1. Create a LuaTex branch to separate this migration from XeTeX to LuaTeX (will clean up some legacy branches);
2. Create a MacOSX branch to go on adding answers to chapter 14, As the memory quota in Mac is still enough. With this branch, I can complete the book as soon as possible;
3. Later merge the master branch (algoxy) with the above two.

Besides, I'll also explore better solution to secure figure quality (may be SVG is better than PNG) with reasonable size overhead.

-- Xinyu

[liuxinyu95]

Created two new branches (luatex and macosx), and cleaned the legacy ones (zh-cn, jvm, and fix-linux-bb).

[liuxinyu95]

After adding another 4 answers, the XeLaTex also broke with OOM issue in Mac OS X.

[liuxinyu95]

Switched to LuaTex.

1. LuaTeX need -shell-escape to detect Mac OS X vs. Linux, but it needn't this option to detect Windows. I changed the way to make (default) font selection based in \IfFontExitsTF (provided in fontspec). Assume STSong is the default font in Mac OS X, and Noto CJK is needed in Linux.
2. If the Linux package manager don't have the full Noto CJK, we need manually download/setup them. We need 3 things: (a) Noto Serif, (b) Noto Sans, and (3) Noto Sans Mono. The can be download from github. Followed this, we download them, and copy the OTF files to ~/.local/share/fonts (the default user level font folder), then force refresh fc-cache -f -v;
3. The \NeedsTeXFormat{LaTeX2e} is needed, or the LuaTeX build fail.

[liuxinyu95]

I finally solved the build-broken issue by switching to LuaLaTeX. The last issue was with luatexja-fontspec pkg. It works well in Mac OS X, but cause the following error in Linux:

! Undefined control sequence. 
\try@load@fontshape ...sname \relax \ltj@@is@kenc 
                        {\f@encoding }\ifin@ \else...

I found PDF in CN/EN can be build successfully after exclude the luatexja-fontspec pkg. I also found the unplugged project has the same issue about OS detection/font settings.

As the next step. I'll (1) try the svg pkg to bring high resolution figure effect; (2) set Noto CJK as the preferred font for all OS platforms.

[liuxinyu95]

The CN font doesn't look as well as the one in unplugged project.

[liuxinyu95]

Track the CN font quality issue, and vector based figure quality improvement separately.