Support ConTeXt standalone

[andreiborisov]

👋🏻

I have trouble to make the package work with ConTeXt standalone distribution starting 2.15.0.

I'm installing the package the same way, however, compiles arbrubtly stop since 2.15.0 without any further issues printed in the log.

Any suggestions on what it can be or instructions for me to diagnose the issue further?

The issue seems to persist up to 3.4.0.

[Witiko]

Hello, thank you for the report. What is your example document and what command are you using to compile it?

[andreiborisov]

Sorry for the wait. I've made an MWE repo so you can investigate locally: https://github.com/andreiborisov/witiko-markdown-issue-402

Please let me know if I can help in any other way 🙏🏻

[andreiborisov]

@Witiko, did you have the chance to look into it? It seems the issue is with a particular file:

runtime error : input file 'expl3-generic.tex' is not found, quitting

[Witiko]

I haven't had time to look into it yet but from your description, it seems that ConTeXt standalone does not contain the expl3 programming language, which is used extensively throughout since around version 2.15.0. Therefore, you may need to either place the .tex files of expl3 into your working directory or into your ConTeXt standalone installation.

[andreiborisov]

Oh, thank you, that makes sense! 🙏🏻 I'll look into it and report my findings.

[Witiko]

Thanks. I'd like to install ConTeXt standalone and test this out before the end of the month but I will appreciate your report.

[Witiko]

In addition to expl3, we require a number of libraries that are only available in TeX Live / LuaTeX and not in ConTeXt standalone / LuaMetaTeX, so full support of ConTeXt Standalone and LMTX is a larger undertaking but your effort should help us implement the support sooner. At the moment, only ConTeXt with LuaTeX and in TeX Live is officially supported.

[andreiborisov]

Tried installing expl3 package and getting a new error now:

context-1  | Package: expl3 2024-03-14 L3 programming layer (code)
context-1  |
context-1  | lua error       > lua error on line 154 in file expl3-code.tex:
context-1  |
context-1  | token call, execute: /data/context/tex/texmf-modules/latex/l3kernel/expl3.lua:421: table index is nil
context-1  | stack traceback:
context-1  |    /data/context/tex/texmf-modules/latex/l3kernel/expl3.lua:421: in main chunk
context-1  |    [C]: in upvalue 'requiem'
context-1  |    ...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:180: in function <...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:165>
context-1  |    (...tail calls...)
context-1  |    [ctxlua]:1: in main chunk
context-1  | <empty file>
context-1  | mtx-context     | fatal error: return code: 1
context-1 exited with code 1

Can you draft a list of other missing packages so I can check if installing them is enough?

[Witiko]

Tried installing expl3 package and getting a new error now: [...]

Interesting. The message indicates that the error is with your installation of expl3 (expl3.lua) rather than with other packages. You may want to ask the LaTeX team in the latex3/latex3 repo to see what the error message that you see is about. I can attest to expl3 working fine with ConTeXt MkIV in TeX Live.

Can you draft a list of other missing packages so I can check if installing them is enough?

Please, see Section 1.1 (Requirements) of the technical documentation of the Markdown package. Namely, in addition to expl3, we require the lua-uni-algos Lua package and the lt3luabridge plain TeX package. We also use the tinyyaml Lua package, but we carry a copy of it in the file markdown-tinyyaml.lua, which is distributed together with the Markdown package.

[Witiko]

Tried installing expl3 package and getting a new error now: [...]

I am seeing the same error in the Witiko/lt3luabridge repo. As discussed in https://github.com/Witiko/lt3luabridge/pull/25, this error is due to expl3 and LuaMetaTeX. Namely, since TeX Live 2024, expl3 no longer works with LuaMetaTeX. I opened an issue for this in https://github.com/latex3/latex3/issues/1518. Are you also using LuaMetaTeX?

Please note that even after the issue with expl3 and LuaMetaTeX has been fixed, the Markdown package has relied on the Selene Unicode package since Markdown 3.0.0. Selene Unicde is compiled into LuaTeX but unavailable in LuaMetaTeX. Until we can get rid of the dependency on Selene Unicode, LuaMetaTeX is not supported out-of-box at the moment.

[Witiko]

Furthermore, since https://github.com/Witiko/markdown/pull/416 and https://github.com/Witiko/markdown/pull/432, we rely on the file UnicodeData.txt being installed. This file is always available in TeX Live but not necessarily in ConTeXt standalone.

[Witiko]

@andreiborisov As you can see, the Markdown package for TeX would benefit from an active developer who uses ConTeXt and who would push for better compatibility with ConTeXt standalone. I don't use ConTeXt standalone myself, so it's less of a priority for me.

[andreiborisov]

Are you also using LuaMetaTeX?

Yes (it's the default engine for ConTeXt now). I'll try to roll back to ConTeXt Mark IV for my purposes, but LuaMetaTeX support would still be great to have in the long run: https://github.com/Witiko/markdown/issues/436

Furthermore, since #416 and #432, we rely on the file UnicodeData.txt being installed. This file is always available in TeX Live but not necessarily in ConTeXt standalone.

Is it better to bundle it with the Markdown package or list it as a dependency? If it's just a single file that is not part of any separately available package, perhaps it would make sense to bundle it up since it would be difficult for ConTeXt users to install it.

As you can see, the Markdown package for TeX would benefit from an active developer who uses ConTeXt and who would push for better compatibility with ConTeXt standalone. I don't use ConTeXt standalone myself, so it's less of a priority for me.

I'm glad to help where I can, but I'm afraid I might not be proficient enough in raw TeX to be awfully useful 🥲

At the very least, I want to make it work with ConTeXt Mark IV standalone. Would it be useful if I make a PR with a manual on installing the package in ConTeXt Mark IV standalone? Or is it better to provide a Dockerfile?

[Witiko]

Is it better to bundle it with the Markdown package or list it as a dependency? If it's just a single file that is not part of any separately available package, perhaps it would make sense to bundle it up since it would be difficult for ConTeXt users to install it.

We can definitely include them in the *.zip files that are produced by the CI and attached to each release and document them in Section 1.2 (Installation) of the user manual. However, we should likely also list them as a dependency in Section 1.1.1 (Lua Requirements) of the technical documentation.

In the long run, it might be best to only use the file UnicodeData.txt during the build. Namely, we could precompile a PEG pattern for Unicode punctuation, so that we don't need UnicodeData.txt at runtime. This would remove the dependency and make the runtime faster at the cost of making the build process more complex.

I'm glad to help where I can, but I'm afraid I might not be proficient enough in raw TeX to be awfully useful 🥲

That's not necessarily an issue, most of the work necessary is in Lua and bugging me about the open issues. :wink:

Would it be useful if I make a PR with a manual on installing the package in ConTeXt Mark IV standalone? Or is it better to provide a Dockerfile?

Extending Section 1.2.1 (Local Installation) of the user manual with extra steps necessary with ConTeXt standalone seems useful. Having a Dockerfile that would allow us to build a Docker image with the Markdown package on top of current ConTeXt standalone and run tests on it in the CI seems even more useful.

[andreiborisov]

So, I've installed everything mentioned and getting a new error now:

...tex/texmf-modules/luatex/lua-uni-algos/lua-uni-parse.lua:50: Please call kpse.set_program_name() before using the library
stack traceback:
    [C]: in function 'kpse.find_file'
    ...tex/texmf-modules/luatex/lua-uni-algos/lua-uni-parse.lua:50: in function 'lua-uni-parse.parse_file'
    .../tex/texmf-modules/luatex/lua-uni-algos/lua-uni-case.lua:28: in main chunk
    [C]: in upvalue 'requiem'
    ...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:180: in function <...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:165>
    (...tail calls...)
    ...tex/texmf-modules/luatex/lua-uni-algos/lua-uni-algos.lua:17: in main chunk
    [C]: in upvalue 'requiem'
    ...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:180: in function <...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:165>
    (...tail calls...)
    ...a/context/tex/texmf-modules/luatex/markdown/markdown.lua:82: in main chunk
    [C]: in upvalue 'requiem'
    ...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:180: in function <...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:165>
    (...tail calls...)
    [ctxlua]:1: in main chunk

[Witiko]

ConTeXt has its own stub library for KPathSea, which is insufficient for the Markdown Package for TeX. The real KPathSea library needs to be loaded and initialized before loading the Markdown package, see the example document for ConTeXt:

https://github.com/Witiko/markdown/blob/e7c6a5b99ec4e2e135061e63694c5ef2e92ed444/examples/context-mkiv.tex#L6-L11

The reason why we don't do this automatically inside the module is because the KPathSea library cannot be initialized multiple times, so there is a potential for issues if multiple modules tried to initialize KPathSea. Furthermore, it seems impossible to determine whether the current kpse global is the stub or the real KPathSea library and whether KPathSea has already been initialized. Therefore, users currently need to load and initialize KPathSea manually.

[Witiko]

We are almost out of high-priority tickets, so I am hoping to start work on reducing external dependencies and making the Markdown package smaller and more self-sufficient in the following months. This should make the use with ConTeXt standalone much easier and your research in this ticket will be the basis for the work.