More LaTeX callbacks should be exclusive

zauguin commented 5 years ago

There are some callbacks, like mlist_to_hlist, which are currently classified as list callbacks. Therefore multiple callbacks can be added there, even if they by definition can't compose. E.g. the mlist_to_hlist callback outputs a hlist, so passing this to the next callback as mlist leads to trouble. To get better diagnostics, they should become exclusive.

(Part 1 of https://mailman.ntg.nl/pipermail/dev-luatex/2019-October/006310.html)

Also for some of these,

Minimal example showing the bug

\RequirePackage{latexbug}
\documentclass{article}
% Package A
\directlua{
  luatexbase.add_to_callback('mlist_to_hlist', function (h, d, p)
     % Doing something awesome with h
     return node.mlist_to_hlist(h, d, p)
  end, 'mlist modifier A')
}

% Package B
\directlua{
  luatexbase.add_to_callback('mlist_to_hlist', function (h, d, p)
     % Doing something else with h
     return node.mlist_to_hlist(h, d, p)
  end, 'mlist modifier B')
}
% This should fail because a second mlist_to_hlist doesn't make sense, but it gets passed a hlist instead

\begin{document}
$a$
\end{document}

Log file (required) and possibly PDF file

Log file

``` This is LuaTeX, Version 1.10.0 (TeX Live 2019) (format=lualatex 2019.5.5) 26 MAY 2019 14:05 restricted system commands enabled. **list.tex (./list.tex LaTeX2e <2018-12-01> Lua module: luaotfload-main 2019-05-18 2.9707 luaotfload entry point Lua module: luaotfload-init 2019-05-18 2.9707 luaotfload submodule / initializat ion Lua module: lualibs 2019-05-18 2.6503 ConTeXt Lua standard libraries. Lua module: lualibs-extended 2019-05-18 2.6503 ConTeXt Lua libraries -- extended collection. Lua module: luaotfload-log 2019-05-18 2.9707 luaotfload submodule / logging Lua module: luaotfload-parsers 2019-05-18 2.9707 luaotfload submodule / filelist Lua module: luaotfload-configuration 2019-05-18 2.9707 luaotfload submodule / co nfig file reader luaotfload | conf : Root cache directory is /home/marcel/.texlive2019/texmf-var/ luatex-cache/generic/names. luaotfload | init : Loading fontloader components individually. Lua-only attribute luaotfload@state = 2 Lua-only attribute luaotfload@noligature = 3 Lua-only attribute luaotfload@syllabe = 4 luaotfload | init : Context OpenType loader version “3.108” Inserting `luaotfload.node_processor' at position 1 in `pre_linebreak_filter'. Inserting `luaotfload.node_processor' at position 1 in `hpack_filter'. Lua module: luaotfload-loaders 2019-05-18 2.9707 luaotfload submodule / callback handling Inserting `luaotfload.define_font' at position 1 in `define_font'. Lua module: luaotfload-database 2019-05-18 2.9707 luaotfload submodule / databas e Lua module: luaotfload-colors 2019-05-18 2.9707 luaotfload submodule / color Lua-only attribute luaotfload_color_attribute = 5 Lua module: luaotfload-resolvers 2019-05-18 2.9707 luaotfload submodule / resolv ers luaotfload | conf : Root cache directory is /home/marcel/.texlive2019/texmf-var/ luatex-cache/generic/names. Lua module: luaotfload-features 2019-05-18 2.9707 luaotfload submodule / feature s Lua module: luaotfload-letterspace 2019-05-18 2.9707 luaotfload submodule / colo r Lua module: luaotfload-auxiliary 2019-05-18 2.9707 luaotfload submodule / auxili ary functions Inserting `luaotfload.aux.set_sscale_dimens' at position 1 in `luaotfload.patch_ font'. Inserting `luaotfload.aux.patch_cambria_domh' at position 2 in `luaotfload.patch _font'. Inserting `luaotfload.aux.fixup_fontdata' at position 1 in `luaotfload.patch_fon t_unsafe'. Inserting `luaotfload.aux.set_capheight' at position 3 in `luaotfload.patch_font '. Inserting `luaotfload.rewrite_fontname' at position 4 in `luaotfload.patch_font' . luaotfload | main : initialization completed in 0.154 seconds (/usr/local/texlive/2019/texmf-dist/tex/latex/base/article.cls Document Class: article 2018/09/03 v1.4i Standard LaTeX document class (/usr/local/texlive/2019/texmf-dist/tex/latex/base/size12.clo File: size12.clo 2018/09/03 v1.4i Standard LaTeX file (size option) luaotfload | db : Font names database loaded from /home/marcel/.texlive2019/texm f-var/luatex-cache/generic/names/luaotfload-names.luc) \c@part=\count80 \c@section=\count81 \c@subsection=\count82 \c@subsubsection=\count83 \c@paragraph=\count84 \c@subparagraph=\count85 \c@figure=\count86 \c@table=\count87 \abovecaptionskip=\skip41 \belowcaptionskip=\skip42 \bibindent=\dimen102 ) (/usr/local/texlive/2019/texmf-dist/tex/latex/xcolor/xcolor.sty Package: xcolor 2016/05/11 v2.12 LaTeX color extensions (UK) (/usr/local/texlive/2019/texmf-dist/tex/latex/graphics-cfg/color.cfg File: color.cfg 2016/01/02 v1.6 sample color configuration ) Package xcolor Info: Driver file: luatex.def on input line 225. (/usr/local/texlive/2019/texmf-dist/tex/latex/graphics-def/luatex.def File: luatex.def 2018/01/08 v1.0l Graphics/color driver for luatex ) Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1348. Package xcolor Info: Model `hsb' substituted by `rgb' on input line 1352. Package xcolor Info: Model `RGB' extended on input line 1364. Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1366. Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1367. Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1368. Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1369. Package xcolor Info: Model `Gray' substituted by `gray' on input line 1370. Package xcolor Info: Model `wave' substituted by `hsb' on input line 1371. ) (/usr/local/texlive/2019/texmf-dist/tex/lualatex/luacode/luacode.sty Package: luacode 2012/01/23 v1.2a lua-in-tex helpers (mpg) (/usr/local/texlive/2019/texmf-dist/tex/generic/oberdiek/ifluatex.sty Package: ifluatex 2016/05/16 v1.4 Provides the ifluatex switch (HO) Package ifluatex Info: LuaTeX detected. ) (/usr/local/texlive/2019/texmf-dist/tex/luatex/luatexbase/luatexbase.sty Package: luatexbase 2015/10/04 v1.3 luatexbase interface to LuaTeX (/usr/local/texlive/2019/texmf-dist/tex/luatex/ctablestack/ctablestack.sty Package: ctablestack 2015/10/01 v1.0 Catcode table stable support \@catcodetablestackcnt=\count88 ) \CatcodeTableOther=\catcodetable5 \CatcodeTableExpl=\catcodetable6 ) \luacode@lines=\toks14 \luacode@table@soft=\catcodetable7 ) \dropsidebearings=\count89 Inserting `Drop sidebearings after linebreaking' at position 1 in `post_linebrea k_filter'. (./list.aux) LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 62. LaTeX Font Info: ... okay on input line 62. LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 62. LaTeX Font Info: ... okay on input line 62. LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 62. LaTeX Font Info: ... okay on input line 62. LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 62. LaTeX Font Info: ... okay on input line 62. LaTeX Font Info: Checking defaults for TU/lmr/m/n on input line 62. LaTeX Font Info: ... okay on input line 62. LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 62. LaTeX Font Info: ... okay on input line 62. LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 62. LaTeX Font Info: ... okay on input line 62. (/usr/local/texlive/2019/texmf-dist/tex/context/base/mkii/supp-pdf.mkii [Loading MPS to PDF converter (version 2006.09.02).] \scratchcounter=\count90 \scratchdimen=\dimen103 \scratchbox=\box27 \nofMPsegments=\count91 \nofMParguments=\count92 \everyMPshowfont=\toks15 \MPscratchCnt=\count93 \MPscratchDim=\dimen104 \MPnumerator=\count94 \makeMPintoPDFobject=\count95 \everyMPtoPDFconversion=\toks16 ) LaTeX Font Info: External font `cmex10' loaded for size (Font) <12> on input line 73. LaTeX Font Info: External font `cmex10' loaded for size (Font) <8> on input line 73. LaTeX Font Info: External font `cmex10' loaded for size (Font) <6> on input line 73. [1 {/usr/local/texlive/2019/texmf-var/fonts/map/pdftex/updmap/pdftex.map}] [2] [3] (./list.aux)) Here is how much of LuaTeX's memory you used: 1344 strings out of 494291 100000,106986 words of node,token memory allocated 377 words of node memory still in use: 2 hlist, 1 vlist, 1 rule, 2 glue, 3 kern, 1 glyph, 4 attribute, 44 glue_spec, 4 attribute_list, 1 write nodes avail lists: 2:43,3:11,4:8,5:62,6:5,7:263,8:1,9:34,10:2,11:12 5598 multiletter control sequences out of 65536+600000 29 fonts using 1872463 bytes 25i,5n,35p,168b,136s stack positions out of 5000i,500n,10000p,200000b,100000s < /usr/local/texlive/2019/texmf-dist/fonts/opentype/public/lm/lmroman12-bold.otf>< /usr/local/texlive/2019/texmf-dist/fonts/opentype/public/lm/lmroman12-regular.ot f> Output written on list.pdf (3 pages, 11659 bytes). PDF statistics: 35 PDF objects out of 1000 (max. 8388607) 20 compressed objects within 1 object stream 0 named destinations out of 1000 (max. 131072) 1 words of extra memory for PDF output out of 10000 (max. 100000000) ```

zauguin commented 5 years ago

There are some callbacks where chaining just doesn't make sense:

mlist_to_hlist The result is a hlist and no longer a mlist
linebreak_filter The result is already broken into lines, it's no longer a hlist.

Then there are some callbacks which currently should become exclusive, but it might be better to add a new callback type for them to allow composition:

new_graf Currently it is simple, but has to return a bool which decides if the new paragraph is indented. I think most usecases do not really need to modify this though, so if we add a "I don't care" return value we could chain multiple, as long as only one of them returns some real value.
process_rule We could add a callback type where we try all callbacks until one processes the rule, but we could also make it exclusive and let the callback be responsible for implementing a more extensible system.

And there are some odd cases:

hyphenate/ligaturing/kerning ... On the one hand they should be exclusive because running these passes multiple times doesn't really make sense, but I also don't think that it really hurts if e.g. hyphenation does run multiple times. But these callbacks are kind of broken anyway (see eg. https://mailman.ntg.nl/pipermail/dev-luatex/2018-November//006139.html) so I don't think they are currently worth the effort.

davidcarlisle commented 5 years ago

The first two seem like clear bugs and we could fix in the upcoming release? There should be no compatibility issues as changing already gives an error.

On Wed, 2 Oct 2019 at 15:11, Marcel Krüger notifications@github.com wrote:

There are some callbacks where chaining just doesn't make sense:

mlist_to_hlist The result is a hlist and no longer a mlist

linebreak_filter The result is already broken into lines, it's no longer a hlist.

Then there are some callbacks which currently should become exclusive, but it might be better to add a new callback type for them to allow composition:

new_graf Currently it is simple, but has to return a bool which decides if the new paragraph is indented. I think most usecases do not really need to modify this though, so if we add a "I don't care" return value we could chain multiple, as long as only one of them returns some real value.

process_rule We could add a callback type where we try all callbacks until one processes the rule, but we could also make it exclusive and let the callback be responsible for implementing a more extensible system.

And there are some odd cases:

hyphenate/ligaturing/kerning ... On the one hand they should be exclusive because running these passes multiple times doesn't really make sense, but I also don't think that it really hurts if e.g. hyphenation does run multiple times. But these callbacks are kind of broken anyway (see eg. https://mailman.ntg.nl/pipermail/dev-luatex/2018-November//006139.html) so I don't think they are currently worth the effort.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/latex3/latex2e/issues/188?email_source=notifications&email_token=AAJVYAWFK6VEE2X3QJ7ZXQDQMSTZTA5CNFSM4I4WIW22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAE4JNI#issuecomment-537511093, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJVYASNS6JRFFAMYDHZJALQMSTZTANCNFSM4I4WIW2Q .

zauguin commented 5 years ago

@davidcarlisle Right, the question is if process_rule should also become exclusive directly or if want to keep it as it is until another solution is found.

zauguin commented 5 years ago

I pushed the change for mlist_to_hlist, linebreak_filter and process_rule to develop in 6f04d30344776d7b66892690b5be4a8abf42338f. Any opinions on how to handle new_graf and hyphenate/ligaturing/kerning?

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity.

zauguin commented 3 years ago

new_graf already became exclusive in the last release. The other ones need careful handling since some of them can break just by providing dummys, but otherwise they would be good fits for getting pre/post filter variants. A good approach might be to provide such pre/post filters, but add code to only register them if at least of of the callbacks is non-empty.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity.

davidcarlisle commented 3 years ago

@zauguin do we want to handle this now or push to the spring milestone?

josephwright commented 5 days ago

We have some other callback-related stuff on the slate for Spring 2025: can we wrap this one up too?

latex3 / latex2e

More LaTeX callbacks should be exclusive #188

Minimal example showing the bug

Log file (required) and possibly PDF file