Usage with ConTeXt LMTX standalone

[andreiborisov]

Hi! First of all, thank you for such an awesome package!

I'm trying to use it with ConTeXt.

After installing ConTeXt LMTX distribution, I've added all relevant files from the release 2.9.0 to the appropriate folders

tex/texmf-modules/tex/context/third/markdown/t-markdown.tex
tex/texmf-modules/tex/generic/markdown/markdown.tex
tex/texmf-modules/tex/luatex/markdown/markdown.lua

and ran mtxrun --generate as per ConTeXt module installation instructions.

Unfortunately, when I try to use the package I'm getting an error:

markdown        > Buffering markdown input into the temporary input file "helloworld.markdown.in" and scanning for the closing token sequence "\stopmarkdown".
markdown        > The ending token sequence was found.
markdown        > Including markdown document "./helloworld.markdown.in".
lua error       > lua error on line 7 in file /Users/andreiborisov/dev/context-example/helloworld.tex:

token call, execute: ...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:180: module 'unicode' not found:
    no field package.preload['unicode']
    no file '/usr/local/share/lua/5.4/unicode.lua'
    no file '/usr/local/share/lua/5.4/unicode/init.lua'
    no file '/usr/local/lib/lua/5.4/unicode.lua'
    no file '/usr/local/lib/lua/5.4/unicode/init.lua'
    no file './unicode.lua'
    no file './unicode/init.lua'
stack traceback:
    [C]: in upvalue 'requiem'
    ...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:180: in function <...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:165>
    (...tail calls...)
    ...ntext/tex/texmf-modules/tex/luatex/markdown/markdown.lua:72: in main chunk
    [C]: in upvalue 'requiem'
    ...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:180: in function <...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:165>
    (...tail calls...)
    [ctxlua]:1: in main chunk
 1     \usemodule[t][markdown]
 2     
 3     \starttext
 4     
 5     \startmarkdown
 6     Hello *world*!
 7 >>  \stopmarkdown
 8     
 9     \stoptext
10     
mtx-context     | fatal error: return code: 256

MWE

```tex \usemodule[t][markdown] \starttext \startmarkdown Hello *world*! \stopmarkdown \stoptext ```

Full log

```log /bin/bash /Users/andreiborisov/Library/Application Support/Texpad/BuildScripts/ConTeXt_Suite.tpbuild resolvers | formats | executing runner 'run luametatex format': /usr/local/context/tex/texmf-osx-64/bin/luametatex --jobname="helloworld" --fmt=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.fmt --lua=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.lui --c:autogenerate --c:currentrun=1 --c:fulljobname="/Users/andreiborisov/dev/context-example/helloworld.tex" --c:input="/Users/andreiborisov/dev/context-example/helloworld.tex" --c:kindofrun=1 --c:maxnofruns=9 --c:synctex="1" --c:texmfbinpath="/usr/local/context/tex/texmf-osx-64/bin" fonts > beware: no fonts are loaded yet, using 'lm mono' in box system > system > ConTeXt ver: 2021.05.31 14:13 LMTX fmt: 2021.6.1 int: english/english system > system > 'cont-new.mkxl' loaded open source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > beware: some patches loaded from cont-new.mkiv close source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > files > jobname 'helloworld', input '/Users/andreiborisov/dev/context-example/helloworld', result 'helloworld' fonts > latin modern fonts are not preloaded languages > language 'en' is active system > synctex functionality is enabled, expect 5-10 pct runtime overhead! open source > level 1, order 2, name '/Users/andreiborisov/dev/context-example/helloworld.tex' modules > 'markdown' is loaded open source > level 2, order 3, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' loading > ConTeXt User Module / markdown open source > level 3, order 4, name 'markdown.tex' (l.596) markdown.tex info: Using mode 2: Direct Lua access. close source > level 3, order 4, name 'markdown.tex' modules > 'database' is loaded open source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' resolvers > lua > loading file '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.lua' succeeded close source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' system > module wrapping error in 'markdown' close source > level 2, order 5, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' fonts > preloading latin modern fonts (second stage) fonts > otf loading > loading '/usr/local/context/tex/texmf/fonts/opentype/public/lm/lmroman12-regular.otf', hash 'lmroman12-regular' fonts > otf loading > loading done fonts > otf loading > saving '/usr/local/context/tex/texmf/fonts/opentype/public/lm/lmroman12-regular.otf' in cache system > lua > compiling '/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/fonts/otl/lmroman12-regular.tma' into '/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/fonts/otl/lmroman12-regular.tmd' system > lua > dumping '/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/fonts/otl/lmroman12-regular.tma' into '/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/fonts/otl/lmroman12-regular.tmd' stripped fonts > otf loading > loading, optimizing, packing and caching time 0.068 fonts > otf loading > loading '/usr/local/context/tex/texmf/fonts/opentype/public/lm-math/latinmodern-math.otf', hash 'latinmodern-math' fonts > otf loading > loading done fonts > otf loading > saving '/usr/local/context/tex/texmf/fonts/opentype/public/lm-math/latinmodern-math.otf' in cache system > lua > compiling '/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/fonts/otl/latinmodern-math.tma' into '/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/fonts/otl/latinmodern-math.tmd' system > lua > dumping '/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/fonts/otl/latinmodern-math.tma' into '/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/fonts/otl/latinmodern-math.tmd' stripped fonts > otf loading > loading, optimizing, packing and caching time 1.224 fonts > 'fallback modern-designsize rm 12pt' is loaded markdown > Buffering markdown input into the temporary input file "helloworld.markdown.in" and scanning for the closing token sequence "\stopmarkdown". markdown > The ending token sequence was found. markdown > Including markdown document "./helloworld.markdown.in". lua error > lua error on line 7 in file /Users/andreiborisov/dev/context-example/helloworld.tex: token call, execute: ...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:180: module 'unicode' not found: no field package.preload['unicode'] no file '/usr/local/share/lua/5.4/unicode.lua' no file '/usr/local/share/lua/5.4/unicode/init.lua' no file '/usr/local/lib/lua/5.4/unicode.lua' no file '/usr/local/lib/lua/5.4/unicode/init.lua' no file './unicode.lua' no file './unicode/init.lua' stack traceback: [C]: in upvalue 'requiem' ...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:180: in function <...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:165> (...tail calls...) ...ntext/tex/texmf-modules/tex/luatex/markdown/markdown.lua:72: in main chunk [C]: in upvalue 'requiem' ...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:180: in function <...xt/tex/texmf-context/tex/context/base/mkiv/l-sandbox.lua:165> (...tail calls...) [ctxlua]:1: in main chunk 1 \usemodule[t][markdown] 2 3 \starttext 4 5 \startmarkdown 6 Hello *world*! 7 >> \stopmarkdown 8 9 \stoptext 10 mtx-context | fatal error: return code: 256 ```

Any help would be appreciated🙏

[Witiko]

@andreiborisov Hello. There is currently no official support for LuaMetaTeX, although it would be nice to have.

The logs indicate that LuaMetaTeX lacks the Selene Unicode library (unicode) that is required by the markdown package. However, this does not sound right, since the LuaMetaTeX manual claims to support unicode.

You should be able to compile by using a shell escape bridge, which is used to access Lua in legacy engines such as pdfTeX and XeTeX. You can request the shell escape bridge by redefining the \markdownMode command to the value 0:

\usemodule[t][markdown]
\def\markdownMode{0}

\starttext

\startmarkdown
Hello *world*!
\stopmarkdown

\stoptext

This requires that your installation contains LuaTeX. Please, let me know how it goes!

[andreiborisov]

Thanks! I can confirm that the issue is LuaMetaTeX, since when running with

context --luatex helloworld.text

instead of

context helloworld.text

everything compiles smoothly.

Unfortunately,

\def\markdownMode{0}

doesn't work either producing:

markdown        > Buffering markdown input into the temporary input file "helloworld.markdown.in" and scanning for the closing token sequence "\stopmarkdown".
markdown        > The ending token sequence was found.
markdown        > Including markdown document "./helloworld.markdown.in".
markdown        > Writing a helper Lua script to the file "helloworld.markdown.lua".
markdown        > Executing a helper Lua script from the file "helloworld.markdown.lua" and storing the result in the file "helloworld.markdown.out".
texlua "./helloworld.markdown.lua" > "./helloworld.markdown.out"

runtime error : input file 'helloworld.markdown.out' is not found, quitting
mtx-context     | fatal error: return code: 256

helloworld.markdown.lua (prettified):

local ran_ok, error = pcall(function()
    local kpse = require("kpse")
    kpse.set_program_name("luatex")
    local lfs = require("lfs")
    local cacheDir = "./_markdown_helloworld"
    if not lfs.isdir(cacheDir) then assert(lfs.mkdir(cacheDir)) end
    local md = require("markdown")
    local convert = md.new({
        cacheDir = "./_markdown_helloworld",
        frozenCacheFileName = "./_markdown_helloworld/frozenCache.tex",
        frozenCacheCounter = 0
    })
    local file = assert(io.open("./helloworld.markdown.in", "r"))
    local input = assert(file:read("*a"))
    assert(file:close())
    print(convert(input:gsub("\r\n?", "\n") .. "\n"))
end)
if not ran_ok then
    local file = io.open("./helloworld.markdown.err", "w")
    if file then
        file:write(error .. "\n")
        file:close()
    end
    print(
        '\\markdownError{An error was encountered while executing Lua code}{For further clues, examine the file "./helloworld.markdown.err"}')
end

Full log

``` resolvers | formats | executing runner 'run luametatex format': /usr/local/context/tex/texmf-osx-64/bin/luametatex --jobname="helloworld" --fmt=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.fmt --lua=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.lui --c:currentrun=1 --c:fulljobname="./helloworld.tex" --c:input="./helloworld.tex" --c:kindofrun=1 --c:maxnofruns=9 --c:texmfbinpath="/usr/local/context/tex/texmf-osx-64/bin" fonts > beware: no fonts are loaded yet, using 'lm mono' in box system > system > ConTeXt ver: 2021.05.31 14:13 LMTX fmt: 2021.6.1 int: english/english system > system > 'cont-new.mkxl' loaded open source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > beware: some patches loaded from cont-new.mkiv close source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > files > jobname 'helloworld', input './helloworld', result 'helloworld' fonts > latin modern fonts are not preloaded languages > language 'en' is active open source > level 1, order 2, name './helloworld.tex' modules > 'markdown' is loaded open source > level 2, order 3, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' loading > ConTeXt User Module / markdown open source > level 3, order 4, name 'markdown.tex' (l.526) markdown.tex info: Using mode 0: Shell escape via write18. close source > level 3, order 4, name 'markdown.tex' modules > 'database' is loaded open source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' resolvers > lua > loading file '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.lua' succeeded close source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' system > module wrapping error in 'markdown' close source > level 2, order 5, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' fonts > preloading latin modern fonts (second stage) fonts > 'fallback modern-designsize rm 12pt' is loaded markdown > Buffering markdown input into the temporary input file "helloworld.markdown.in" and scanning for the closing token sequence "\stopmarkdown". markdown > The ending token sequence was found. markdown > Including markdown document "./helloworld.markdown.in". markdown > Writing a helper Lua script to the file "helloworld.markdown.lua". markdown > Executing a helper Lua script from the file "helloworld.markdown.lua" and storing the result in the file "helloworld.markdown.out". texlua "./helloworld.markdown.lua" > "./helloworld.markdown.out" runtime error : input file 'helloworld.markdown.out' is not found, quitting mtx-context | fatal error: return code: 256 ```

[Witiko]

@andreiborisov

[...] running with context --luatex helloworld.text [...] everything compiles smoothly.

Thanks for clearing that up.

Unfortunately, \def\markdownMode{0} doesn't work either producing: [...]

Interesting. The output shows that the texlua "./helloworld.markdown.lua" > "./helloworld.markdown.out" command has been executed and yet ConTeXt claims that helloworld.markdown.out does not exist. Does it exist? What happens when you run texlua "./helloworld.markdown.lua" > "./helloworld.markdown.out" manually?

[andreiborisov]

The output shows that the texlua "./helloworld.markdown.lua" > "./helloworld.markdown.out" command has been executed and yet ConTeXt claims that helloworld.markdown.out does not exist. Does it exist? What happens when you run texlua "./helloworld.markdown.lua" > "./helloworld.markdown.out" manually?

It doesn't exist, and when I run it manually it produces

helloworld.markdown.out:

\markdownError{An error was encountered while executing Lua code}{For further clues, examine the file "./helloworld.markdown.err"}

and helloworld.markdown.err:

./helloworld.markdown.lua:1: module 'markdown' not found:
    no field package.preload['markdown']
    [kpse lua searcher] file not found: 'markdown'
    [kpse C searcher] file not found: 'markdown'

🤔🤔

EDIT:

texlua isn't available in ConTeXt standalone distribution it seems, only luatex, which is causing the issue.

[Witiko]

texlua isn't available in ConTeXt distribution it seems, which is causing the issue.

Even more interesting! So where is your texlua coming from if not from the ConTeXt standalone distribution? From the error message, it seems that it has no clue about the existence of the markdown.lua package.

What is also interesting is that we don't see this \markdownError when compiling the ConTeXt document:

texlua "./helloworld.markdown.lua" > "./helloworld.markdown.out"

runtime error : input file 'helloworld.markdown.out' is not found, quitting
mtx-context     | fatal error: return code: 256

This might indicate that texlua "./helloworld.markdown.lua" > "./helloworld.markdown.out" has not actually been executed despite the log message. This may be because the shell access through the \write18 command has been disabled in LMTX.

If you try \def\markdownMode{1}, does it make any difference? This will still use shell to execute texlua, but now it uses Lua's os.execute Lua command to do that. Do you perhaps receive the \markdownError text on the output when you compile the MWE?

[Witiko]

(We may or may not stumble upon an easy solution, but this information is extremely helpful for me as a ConTeXt non-user in fixing this issue further down the road.)

[andreiborisov]

Even more interesting! So where is your texlua coming from if not from the ConTeXt standalone distribution? From the error message, it seems that it has no clue about the existence of the markdown.lua package.

Yes, it was from TeXLive distro🤦‍♂️, but the plot thickens even further: at first, I thought, it makes sense, that it doesn't know anything about context package since it comes from TeXLive, but after symlinking luatex as texlua inside context distro's bin directory, and using it instead, nothing changed. It still behaves exactly the same.

This might indicate that texlua "./helloworld.markdown.lua" > "./helloworld.markdown.out" has not actually been executed despite the log message, since the shell access through the \write18 command has been disabled in LMTX, which may well have happened, since LMTX is not too conservative with changes from what I've heard.

If you try \def\markdownMode{1}, does it make any difference? Do you perhaps receive the \markdownError text on the output when you compile the MWE?

That seems to be the case! As with \def\markdownMode{1} I get:

markdown        > Buffering markdown input into the temporary input file "helloworld.markdown.in" and scanning for the closing token sequence "\stopmarkdown".
markdown        > The ending token sequence was found.
markdown        > Including markdown document "./helloworld.markdown.in".
markdown        > Writing a helper Lua script to the file "helloworld.markdown.lua".
markdown        > Executing a helper Lua script from the file "helloworld.markdown.lua" and storing the result in the file "helloworld.markdown.out".
warning: /usr/local/context/tex/texmf/web2c/texmf.cnf:49: (kpathsea) No cnf value on line: OSFONTDIR =.
open source     > level 2, order 6, name 'helloworld.markdown.out'
tex error       > tex error on line 1 in file helloworld.markdown.out: (l.1) markdown.tex error: An error was encountered while executing Lua code

<macro> \markdownError #1#2->\errhelp {#2.}\errmessage {(l.\the \inputlineno ) markdown.tex error: #1}

<line 4.1> \markdownError{An error was encountered while executing Lua code}{For further clues, examine the file "./helloworld.markdown.err"}

1 >>  \markdownError{An error was encountered while executing Lua code}{For further clues, examine the file "./helloworld.markdown.err"}
2
For further clues, examine the file "./helloworld.markdown.err".
mtx-context     | fatal error: return code: 256

(and the same error in helloworld.markdown.err).

Full log

```log resolvers | formats | executing runner 'run luametatex format': /usr/local/context/tex/texmf-osx-64/bin/luametatex --jobname="helloworld" --fmt=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.fmt --lua=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.lui --c:currentrun=1 --c:fulljobname="./helloworld.tex" --c:input="./helloworld.tex" --c:kindofrun=1 --c:maxnofruns=9 --c:texmfbinpath="/usr/local/context/tex/texmf-osx-64/bin" fonts > beware: no fonts are loaded yet, using 'lm mono' in box system > system > ConTeXt ver: 2021.05.31 14:13 LMTX fmt: 2021.6.1 int: english/english system > system > 'cont-new.mkxl' loaded open source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > beware: some patches loaded from cont-new.mkiv close source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > files > jobname 'helloworld', input './helloworld', result 'helloworld' fonts > latin modern fonts are not preloaded languages > language 'en' is active open source > level 1, order 2, name './helloworld.tex' modules > 'markdown' is loaded open source > level 2, order 3, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' loading > ConTeXt User Module / markdown open source > level 3, order 4, name 'markdown.tex' (l.528) markdown.tex info: Using mode 1: Shell escape via os.execute. close source > level 3, order 4, name 'markdown.tex' modules > 'database' is loaded open source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' resolvers > lua > loading file '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.lua' succeeded close source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' system > module wrapping error in 'markdown' close source > level 2, order 5, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' fonts > preloading latin modern fonts (second stage) fonts > 'fallback modern-designsize rm 12pt' is loaded markdown > Buffering markdown input into the temporary input file "helloworld.markdown.in" and scanning for the closing token sequence "\stopmarkdown". markdown > The ending token sequence was found. markdown > Including markdown document "./helloworld.markdown.in". markdown > Writing a helper Lua script to the file "helloworld.markdown.lua". markdown > Executing a helper Lua script from the file "helloworld.markdown.lua" and storing the result in the file "helloworld.markdown.out". warning: /usr/local/context/tex/texmf/web2c/texmf.cnf:49: (kpathsea) No cnf value on line: OSFONTDIR =. open source > level 2, order 6, name 'helloworld.markdown.out' tex error > tex error on line 1 in file helloworld.markdown.out: (l.1) markdown.tex error: An error was encountered while executing Lua code \markdownError #1#2->\errhelp {#2.}\errmessage {(l.\the \inputlineno ) markdown.tex error: #1} \markdownError{An error was encountered while executing Lua code}{For further clues, examine the file "./helloworld.markdown.err"} 1 >> \markdownError{An error was encountered while executing Lua code}{For further clues, examine the file "./helloworld.markdown.err"} 2 For further clues, examine the file "./helloworld.markdown.err". mtx-context | fatal error: return code: 256 ```

[andreiborisov]

However, I'm not sure how to enable shell access in the ConTeXt LMTX, here are the command-line options, maybe you can spot something relevant...

mtx-context     | ConTeXt Process Management 1.04
mtx-context     |
mtx-context     | basic options:
mtx-context     |
mtx-context     | --run                  process (one or more) files (default action)
mtx-context     | --make                 create context formats
mtx-context     |
mtx-context     | --ctx=name             use ctx file (process management specification)
mtx-context     | --noctx                ignore ctx directives and flags
mtx-context     | --interface            use specified user interface (default: en)
mtx-context     |
mtx-context     | --autopdf              close pdf file in viewer and start pdf viewer afterwards
mtx-context     | --purge                purge files either or not after a run (--pattern=...)
mtx-context     | --purgeall             purge all files either or not after a run (--pattern=...)
mtx-context     |
mtx-context     | --usemodule=list       load the given module or style, normally part of the distribution
mtx-context     | --environment=list     load the given environment file first (document styles)
mtx-context     | --mode=list            enable given the modes (conditional processing in styles)
mtx-context     | --path=list            also consult the given paths when files are looked for
mtx-context     | --arguments=list       set variables that can be consulted during a run (key/value pairs)
mtx-context     | --randomseed=number    set the randomseed
mtx-context     | --result=name          rename the resulting output to the given name
mtx-context     | --trackers=list        set tracker variables (show list with --showtrackers)
mtx-context     | --directives=list      set directive variables (show list with --showdirectives)
mtx-context     | --silent=list          disable logcatgories (show list with --showlogcategories)
mtx-context     | --strip                strip Lua code (only meant for production where no errors are expected)
mtx-context     | --errors=list          show errors at the end of a run, quit when in list (also when --silent)
mtx-context     | --htmlerrorpage        generate html error page instead (optional: =scite)
mtx-context     | --noconsole            disable logging to the console (logfile only)
mtx-context     | --nodummy              don't create a dummy file (can confuse pdf viewers that keep file in view)
mtx-context     | --purgeresult          purge result file before run
mtx-context     |
mtx-context     | --forcexml             force xml stub
mtx-context     | --forcecld             force cld (context lua document) stub
mtx-context     | --forcelua             force lua stub (like texlua)
mtx-context     | --forcemp              force mp stub
mtx-context     |
mtx-context     | --arrange              run extra imposition pass, given that the style sets up imposition
mtx-context     | --noarrange            ignore imposition specifications in the style
mtx-context     |
mtx-context     | --pages                report pagenumbers of names pages and list references (--page=... or --page=...)
mtx-context     |
mtx-context     | --jit                  use luajittex with jit turned off (only use the faster virtual machine)
mtx-context     | --jiton                use luajittex with jit turned on (in most cases not faster, even slower)
mtx-context     |
mtx-context     | --once                 only run once (no multipass data file is produced)
mtx-context     | --runs                 process at most this many times
mtx-context     | --forcedruns           process this many times (permits for optimization trial runs)
mtx-context     |
mtx-context     | --batchmode            run without stopping and do not show messages on the console
mtx-context     | --nonstopmode          run without stopping
mtx-context     |
mtx-context     | --nosynctex            never initializes synctex (for production runs)
mtx-context     | --synctex              run with synctex enabled (better use \setupsynctex[state=start]
mtx-context     |
mtx-context     | --nodates              omit runtime dates in pdf file (optional value: a number (this 1970 offset time) or string "YYYY-MM-DD HH:MM")
mtx-context     | --nocompression        forcefully turns off compression in the backend
mtx-context     | --trailerid            alternative trailer id (or constant one)
mtx-context     |
mtx-context     | --generate             generate file database etc. (as luatools does)
mtx-context     | --paranoid             do not descend to .. and ../..
mtx-context     | --version              report installed context version
mtx-context     |
mtx-context     | --global               assume given file present elsewhere
mtx-context     | --nofile               use dummy file as jobname
mtx-context     |
mtx-context     |
mtx-context     | More information about ConTeXt and the tools that come with it can be found at:
mtx-context     |
mtx-context     | maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
mtx-context     | webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
mtx-context     | wiki     : http://contextgarden.net

and

mtxrun          | ConTeXt TDS Runner Tool 1.32
mtxrun          |
mtxrun          | --script               run an mtx script (lua prefered method) (--noquotes), no script gives list
mtxrun          | --evaluate             run code passed on the commandline (between quotes) (=loop) (exit|quit aborts)
mtxrun          | --execute              run a script or program (texmfstart method) (--noquotes)
mtxrun          | --resolve              resolve prefixed arguments
mtxrun          | --ctxlua               run internally (using preloaded libs)
mtxrun          | --internal             run script using built in libraries (same as --ctxlua)
mtxrun          | --locate               locate given filename in database (default) or system (--first --all --detail)
mtxrun          |
mtxrun          | --tree=pathtotree      use given texmf tree (default file: setuptex.tmf)
mtxrun          | --path=runpath         go to given path before execution
mtxrun          | --ifchanged=filename   only execute when given file has changed (md checksum)
mtxrun          | --iftouched=old,new    only execute when given file has changed (time stamp)
mtxrun          |
mtxrun          | --makestubs            create stubs for (context related) scripts
mtxrun          | --removestubs          remove stubs (context related) scripts
mtxrun          | --stubpath=binpath     paths where stubs wil be written
mtxrun          | --windows              create windows (mswin) stubs
mtxrun          | --unix                 create unix (linux) stubs
mtxrun          | --addbinarypath        prepend the (found) binarypath to runners
mtxrun          |
mtxrun          | --verbose              give a bit more info
mtxrun          | --trackers=list        enable given trackers
mtxrun          | --progname=str         format or backend
mtxrun          | --systeminfo=str       show current operating system, processor, etc
mtxrun          |
mtxrun          | --edit                 launch editor with found file
mtxrun          | --launch               launch files like manuals, assumes os support (--all,--list)
mtxrun          |
mtxrun          | --timedrun             run a script and time its run
mtxrun          | --autogenerate         regenerate databases if needed (handy when used to run context in an editor)
mtxrun          |
mtxrun          | --usekpse              use kpse as fallback (when no mkiv and cache installed, often slower)
mtxrun          | --forcekpse            force using kpse (handy when no mkiv and cache installed but less functionality)
mtxrun          |
mtxrun          | --prefixes             show supported prefixes
mtxrun          |
mtxrun          | --generate             generate file database
mtxrun          |
mtxrun          | --variables            show configuration variables
mtxrun          | --configurations       show configuration order
mtxrun          |
mtxrun          | --directives           show (known) directives
mtxrun          | --trackers             show (known) trackers
mtxrun          | --experiments          show (known) experiments
mtxrun          |
mtxrun          | --expand-braces        expand complex variable
mtxrun          | --resolve-path         expand variable (completely resolve paths)
mtxrun          | --expand-path          expand variable (resolve paths)
mtxrun          | --expand-var           expand variable (resolve references)
mtxrun          | --show-path            show path expansion of ...
mtxrun          | --var-value            report value of variable
mtxrun          | --find-file            report file location
mtxrun          | --find-path            report path of file
mtxrun          |
mtxrun          | --pattern=string       filter variables
mtxrun          |
mtxrun          |
mtxrun          | More information about ConTeXt and the tools that come with it can be found at:
mtxrun          |
mtxrun          | maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
mtxrun          | webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
mtxrun          | wiki     : http://contextgarden.net

[Witiko]

That seems to be the case! As with \def\markdownMode{1} I get:

This indicates that shell access is enabled, but we have to go through the os.execute Lua command rather than through the \write18 TeX command, which LuaMetaTeX seems to have dropped (good riddance). This solves one of our mysteries (why \def\markdownMode{0} did not even call texlua) and is a passable workaround. The only mystery that remains to be solved before you can typeset your document is why texlua does not see the Markdown package, i.e. why require('markdown') produces the error in https://github.com/Witiko/markdown/issues/80#issuecomment-852348680:

./helloworld.markdown.lua:1: module 'markdown' not found:
    no field package.preload['markdown']
    [kpse lua searcher] file not found: 'markdown'
    [kpse C searcher] file not found: 'markdown'

The other mystery is why LuaMetaTeX can't import the unicode library: This prevents the use of LuaMetaTeX (\def\markdownMode{2}) instead of LuaTeX (\def\markdownMode{1} or execution of ConTeXt with the --luatex flag). Solving this mystery would mean that we don't have to use texlua at all.

[andreiborisov]

The only mystery that remains to be solved before you can typeset your document is texlua not seeing the Markdown package [...]

My uneducated guess is that context executable itself provides some context (no pun intended) on how to find the installed packages in the distribution. And since it isn't present with a direct call to luatex/texlua, it fails to locate the markdown package.

I've copied the markdown.lua to the working directory, and it finally compiles🥳

[andreiborisov]

I wonder if the same trick would work with \def\markdownMode{2} and unicode.lua...

[Witiko]

I wonder if the same trick would work with \def\markdownMode{2} and unicode.lua...

The Selene Unicode library is a C library, so I am afraid that would require a bit more work. It is weird that LuaMetaTeX does not include it statically, since here is what the manual says:

For strings in the UTF8 encoding, i.e., strings containing characters above code point 127, the corresponding functions from the slnunicode library can be used, e.g., unicode.utf8.len, unicode.utf8.lower etc. The exceptions are unicode.utf8.find, that always returns byte positions in a string, and unicode.utf8.match and unicode.utf8.gmatch. While the latter two functions in general are UNICODE-aware, they fall-back to non-UNICODE-aware behavior when using the empty capture () but other captures work as expected. For the interpretation of character classes in unicode.utf8 functions refer to the library sources at http://luaforge.net/projects/sln.

It could be that Selene Unicode is available under a slightly different name in LuaMetaTeX.

[andreiborisov]

@Witiko I found the culprit:

The slunicode library is also gone. There are some helpers in the string library that can be used instead and one can write additional LUA code if needed.

Here it is in the manual.

[Witiko]

Excellent, thank you. This should be quite a straightforwards fix of removing require("unicode") from markdown.lua and then replacing every usage of unicode.* with the corresponding helpers. I don't have a ConTeXt LMTX installation on my box, which will cause some delay. If you'd like to tackle this yourself, this should be a nice project for a weekend evening.

[andreiborisov]

I'm not overly familiar with Lua, but I'll try to fix it and prepare a PR if successful👍

[Witiko]

I had some spare time, so I built [a Docker image with ConTeXt LMTX]() and played around:

$ git clone 'https://gitlab.com/islandoftex/images/context.git'
$ cd context
$ docker build --file Dockerfile.lmtx -t context:lmtx .
$ wget 'https://github.com/Witiko/markdown/releases/download/2.9.0/markdown.zip'
$ unzip markdown.zip
$ docker run -it -v markdown:/markdown -w /markdown --rm context:lmtx
root@e3ac27597702:/markdown#

The following patch for the markdown.lua file solves our issue:

72d71
< local unicode = require("unicode")
2276c2275
<   return unicode.utf8.char(tonumber(s))
---
>   return utf8.char(tonumber(s))
2279c2278
<   return unicode.utf8.char(tonumber("0x"..s))
---
>   return utf8.char(tonumber("0x"..s))
2286c2285
<   return unicode.utf8.char(n)
---
>   return utf8.char(n)
3283c3282
<     return unicode.utf8.lower(
---
>     return string.lower(

The only method with no satistfactory replacement was unicode.utf8.lower, which I have replaced with ASCII lower-casing. This will cause Markdown to consider [Žvýkačka] and [žvýkačka] to be different different link, image, and footnote tags, which I don't think is much of an issue, because this behavior is undefined and differs between implementations of Markdown. In addition, it's arguably not even right to use lower-casing for this. If anything, we should use case folding.

I would have added these changes to the Markdown package on the spot, but utf8.* is only supported in Lua ≥ 5.3, which is the default version of Lua in LuaTeX only since TeX Live 2019 or so. At the moment, we are compatible with TeX Live ≥ 2013 and there is no good reason to change that, so we'll probably want to support both Selene Unicode and utf8 through a drop-in replacement (shim) of Selene Unicode. This way, we can support both LuaTeX in TeX Live ≥ 2013 and LuaMetaTeX:

72c72
< local unicode = require("unicode")
---
> local ran_ok, unicode = pcall(require, "unicode")
> if not ran_ok then
>   unicode = {["utf8"]={char=utf8.char}}
> end
3283c3286
<     return unicode.utf8.lower(
---
>     return string.lower(

This change is now waiting to be added in #81.

[Witiko]

After I have solved this issue, another issue surfaced: not only does LuaMetaTeX not include Selene Unicode, it also does not include KPathSea. Here is what the LMTX manual has to say on the matter:

The KPSE library is no longer built-in, but one can use an external KPSE library, assuming that it is present on the system, because the engine has a so called optional library interface to it.

I am curious which library is responsible for finding the installed package and font files in ConTeXt if KPathSea is no longer included. We are using KPathSea to find the markdown-languages.json files in the TeX directory structure and merge their contents for the contentBlocks syntax extension. Unless you use the contentBlocks syntax extension, this should not bite you.

I would like to know how one would go about using an external KPSE library, since when LMTX makes it to TeX Live, there will be a KPathSea library that we can use even though LMTX does not have a static binding to it. This seems to be the last piece of the puzzle in regaining compatibility with ConTeXt LMTX.

[andreiborisov]

Wow, thank you for such a fast fix and the detailed explanation😍

I would like to know how one would go about using "an external KPSE library" as the manual says, since when LMTX makes it to TeX Live, there will be a KPathSea library that we can use even though LMTX does not have a static binding to it.

I found something in ConTeXt source code that seems to be related: https://github.com/contextgarden/context-mirror/blob/beta/tex/context/base/mkxl/libs-imp-kpse.lmt

I'm not sure if the library itself is included in the ConTeXt distribution though.

[Witiko]

Fantastic, thank you for your detective work! Let's come back to this when LMTX has been released in TeX Live and there is an actual KPathSea library to test this with. 🙂 For now, I have shimmed KPathSea in 575a1e0, so that we just look for files in the current working directory when KPathSea is unavailable.

[Witiko]

For some arcane reason, ConTeXt MkIV is returning exit code 1 in the unit tests since 575a1e0:

$ make -C tests
make[1]: Entering directory '/__w/markdown/markdown/tests'
find testfiles/ -type f -name '*.test' | parallel --halt 2 ./test.sh
Testfile testfiles/PHP_Markdown/nesting.test
  Format templates/context/
    Template templates/context/input.tex.m4
      Command texexec  --passon="--shell-escape --8bit --interaction=nonstopmode" test.tex
      Command context                                              --nonstopmode  test.tex
        Command terminated with exit code 1.
    Template templates/context/verbatim.tex.m4
      Command texexec  --passon="--shell-escape --8bit --interaction=nonstopmode" test.tex
      Command context                                              --nonstopmode  test.tex
        Command terminated with exit code 1.
  Format templates/latex/
    Template templates/latex/input.tex.m4
      Command pdflatex --shell-escape                  --interaction=nonstopmode  test.tex
      Command xelatex  --shell-escape --8bit           --interaction=nonstopmode  test.tex
      Command lualatex                                 --interaction=nonstopmode  test.tex
    Template templates/latex/verbatim.tex.m4
      Command pdflatex --shell-escape                  --interaction=nonstopmode  test.tex
      Command xelatex  --shell-escape --8bit           --interaction=nonstopmode  test.tex
      Command lualatex                                 --interaction=nonstopmode  test.tex
  Format templates/plain/
    Template templates/plain/input.tex.m4
      Command pdftex   --shell-escape                  --interaction=nonstopmode  test.tex
      Command xetex    --shell-escape --8bit           --interaction=nonstopmode  test.tex
      Command luatex                                   --interaction=nonstopmode  test.tex
    Template templates/plain/verbatim.tex.m4
      Command pdftex   --shell-escape                  --interaction=nonstopmode  test.tex
      Command xetex    --shell-escape --8bit           --interaction=nonstopmode  test.tex
      Command luatex                                   --interaction=nonstopmode  test.tex
Testfile testfiles/CommonMark_0.26/fenced-code.test
  ...

I can't reproduce this on my TeX Live 2021 installation and neither do I see anything particularly suspicious in the logs. It's not too worrying, because all the outputs are produced correctly, otherwise the tests would fail, but I would be happier if I understood. 😅

[andreiborisov]

After compiling and trying out the latest commit 33a185c132b8ba5d06e6ce5354881c241ca96c61, the ConTeXt LuaMetaTeX still doesn't work😔:

markdown        > Buffering markdown input into the temporary input file "helloworld.markdown.in" and scanning for the closing token sequence "\stopmarkdown".
markdown        > The ending token sequence was found.
markdown        > Including markdown document "./helloworld.markdown.in".
open source     > level 2, order 6, name './_markdown_helloworld/46bd0aa8671d4f971ec108d3f8bef40d.md.tex'
tex error       > tex error on line 1 in file ./_markdown_helloworld/46bd0aa8671d4f971ec108d3f8bef40d.md.tex:

lua error:

    registered function call [1097]: ...ext/tex/texmf-context/tex/context/base/mkxl/font-con.lmt:1048: attempt to index a nil value (global 'file')
stack traceback:
    ...ext/tex/texmf-context/tex/context/base/mkxl/font-con.lmt:1048: in metamethod 'index'
    ...ext/tex/texmf-context/tex/context/base/mkxl/font-def.lmt:181: in local 'r'
    ...ext/tex/texmf-context/tex/context/base/mkxl/font-ctx.lmt:1708: in field 'resolve'
    ...ext/tex/texmf-context/tex/context/base/mkxl/font-def.lmt:474: in field 'read'
    ...ext/tex/texmf-context/tex/context/base/mkxl/font-ctx.lmt:1335: in function <...ext/tex/texmf-context/tex/context/base/mkxl/font-ctx.lmt:1192>
    (...tail calls...)

1 >>  Hello \markdownRendererEmphasis{world}!\relax
close source    > level 2, order 6, name './_markdown_helloworld/46bd0aa8671d4f971ec108d3f8bef40d.md.tex'
close source    > level 1, order 6, name './helloworld.tex'
backend         > xmp > using file '/usr/local/context/tex/texmf-context/tex/context/base/mkiv/lpdf-pdx.xml'
system          | total runtime: 0.443 seconds of 0.484 seconds

Full log

```log resolvers | formats | executing runner 'run luametatex format': /usr/local/context/tex/texmf-osx-64/bin/luametatex --jobname="helloworld" --fmt=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.fmt --lua=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.lui --c:currentrun=1 --c:fulljobname="./helloworld.tex" --c:input="./helloworld.tex" --c:kindofrun=1 --c:maxnofruns=9 --c:texmfbinpath="/usr/local/context/tex/texmf-osx-64/bin" fonts > beware: no fonts are loaded yet, using 'lm mono' in box system > system > ConTeXt ver: 2021.05.31 14:13 LMTX fmt: 2021.6.1 int: english/english system > system > 'cont-new.mkxl' loaded open source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > beware: some patches loaded from cont-new.mkiv close source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > files > jobname 'helloworld', input './helloworld', result 'helloworld' fonts > latin modern fonts are not preloaded languages > language 'en' is active open source > level 1, order 2, name './helloworld.tex' modules > 'markdown' is loaded open source > level 2, order 3, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' loading > ConTeXt User Module / markdown open source > level 3, order 4, name 'markdown/markdown.tex' markdown > Using mode 2: Direct Lua access. close source > level 3, order 4, name 'markdown/markdown.tex' modules > 'database' is loaded open source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' resolvers > lua > loading file '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.lua' succeeded close source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' close source > level 2, order 5, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' fonts > preloading latin modern fonts (second stage) fonts > 'fallback modern-designsize rm 12pt' is loaded markdown > Buffering markdown input into the temporary input file "helloworld.markdown.in" and scanning for the closing token sequence "\stopmarkdown". markdown > The ending token sequence was found. markdown > Including markdown document "./helloworld.markdown.in". open source > level 2, order 6, name './_markdown_helloworld/46bd0aa8671d4f971ec108d3f8bef40d.md.tex' tex error > tex error on line 1 in file ./_markdown_helloworld/46bd0aa8671d4f971ec108d3f8bef40d.md.tex: lua error: registered function call [1097]: ...ext/tex/texmf-context/tex/context/base/mkxl/font-con.lmt:1048: attempt to index a nil value (global 'file') stack traceback: ...ext/tex/texmf-context/tex/context/base/mkxl/font-con.lmt:1048: in metamethod 'index' ...ext/tex/texmf-context/tex/context/base/mkxl/font-def.lmt:181: in local 'r' ...ext/tex/texmf-context/tex/context/base/mkxl/font-ctx.lmt:1708: in field 'resolve' ...ext/tex/texmf-context/tex/context/base/mkxl/font-def.lmt:474: in field 'read' ...ext/tex/texmf-context/tex/context/base/mkxl/font-ctx.lmt:1335: in function <...ext/tex/texmf-context/tex/context/base/mkxl/font-ctx.lmt:1192> (...tail calls...) 1 >> Hello \markdownRendererEmphasis{world}!\relax close source > level 2, order 6, name './_markdown_helloworld/46bd0aa8671d4f971ec108d3f8bef40d.md.tex' close source > level 1, order 6, name './helloworld.tex' backend > xmp > using file '/usr/local/context/tex/texmf-context/tex/context/base/mkiv/lpdf-pdx.xml' system | total runtime: 0.443 seconds of 0.484 seconds ```

[Witiko]

I don't see how the error is related to the Markdown package from the stack trace. The ./_markdown_helloworld/46bd0aa8671d4f971ec108d3f8bef40d.md.tex file has been produced by the Lua parser and successfully opened by ConTeXt. The file contains the text Hello \markdownRendererEmphasis{world}!\relax, so I am thinking that the default definition of \markdownRendererEmphasis could be the culprit. By default, it expands to \markdownRendererEmphasisPrototype, which is defined as follows:

\def\markdownRendererEmphasisPrototype#1{{\em#1}}

This seems to be an issue with fonts on your end. If you try to compile the following document, you should be getting the same issue:

\starttext
Hello {\em world}!
\stoptext

[andreiborisov]

This seems to be an issue with fonts on your end. If you try to compile the following document, you should be getting the same issue.

This both work:

\starttext

Hello {\em world}!

\stoptext

\usemodule[t][markdown]

\starttext

Hello \markdownRendererEmphasis{world}!\relax

\stoptext

but this is not:

\usemodule[t][markdown]

\starttext

\startmarkdown

Hello *world*!

\stopmarkdown

\stoptext

🤯

[Witiko]

Hmm, what if you try the following:

\usemodule[t][markdown]

\starttext

\begingroup
\catcode`\%=12
\input ./_markdown_helloworld/46bd0aa8671d4f971ec108d3f8bef40d.md.tex
\endgroup

\stoptext

There should not be not much more that is going on behind the scenes.

[andreiborisov]

Hmm, what if you try the following

This works just fine...

[Witiko]

But this no longer works fine, I assume?

\usemodule[t][markdown]

\starttext

\markdownInput{helloworld.markdown.in}

\stoptext

Let's expand the command, this should produce the same result as the above code:

\usemodule[t][markdown]

\starttext

\directlua{
  local print = tex.print
  \markdownPrepare
  local file = assert(io.open("./_markdown_helloworld/46bd0aa8671d4f971ec108d3f8bef40d.md.tex", "r"))
  local input = assert(file:read("*a"))
  assert(file:close())
  print(convert(input:gsub("\r\n?", "\n") .. "\n"))
}

\stoptext

If the above does not work, does the following work:

\usemodule[t][markdown]

\starttext

\directlua{
  local print = tex.print
  \markdownPrepare
}

\stoptext

It the last code still does not work, then some part of \markdownPrepare could be the culprit.

[andreiborisov]

But this no longer works fine, I assume?

Yep, it produces the same exact error as earlier.

Let's expand the command, this should produce the same result as the above code:

This is more interesting:

markdown        > Using mode 2: Direct Lua access.
close source    > level 3, order 4, name 'markdown/markdown.tex'
modules         > 'database' is loaded
open source     > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv'
resolvers       > lua > loading file '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.lua' succeeded
close source    > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv'
close source    > level 2, order 5, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex'
fonts           > preloading latin modern fonts (second stage)
fonts           > 'fallback modern-designsize rm 12pt' is loaded
suggestion      > use \withring instead of \r
lua error       > lua error on line 12 in file ./helloworld.tex:

token call, syntax: [ctxlua]:1: invalid escape sequence near '"\w'
 2
 3     \starttext
 4
 5     \directlua{
 6       local print = tex.print
 7       \markdownPrepare
 8       local file = assert(io.open("./_markdown_helloworld/54c7679395b5b68b2726f58d74fb517a.md.tex", "r"))
 9       local input = assert(file:read("*a"))
10       assert(file:close())
11       print(convert(input:gsub("\r\n?", "\n") .. "\n"))
12 >>  }
13
14     \stoptext
15
mtx-context     | fatal error: return code: 256

Full log

```log resolvers | formats | executing runner 'run luametatex format': /usr/local/context/tex/texmf-osx-64/bin/luametatex --jobname="helloworld" --fmt=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.fmt --lua=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.lui --c:currentrun=1 --c:fulljobname="./helloworld.tex" --c:input="./helloworld.tex" --c:kindofrun=1 --c:maxnofruns=9 --c:texmfbinpath="/usr/local/context/tex/texmf-osx-64/bin" fonts > beware: no fonts are loaded yet, using 'lm mono' in box system > system > ConTeXt ver: 2021.05.31 14:13 LMTX fmt: 2021.6.2 int: english/english system > system > 'cont-new.mkxl' loaded open source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > beware: some patches loaded from cont-new.mkiv close source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > files > jobname 'helloworld', input './helloworld', result 'helloworld' fonts > latin modern fonts are not preloaded languages > language 'en' is active open source > level 1, order 2, name './helloworld.tex' modules > 'markdown' is loaded open source > level 2, order 3, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' loading > ConTeXt User Module / markdown open source > level 3, order 4, name 'markdown/markdown.tex' markdown > Using mode 2: Direct Lua access. close source > level 3, order 4, name 'markdown/markdown.tex' modules > 'database' is loaded open source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' resolvers > lua > loading file '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.lua' succeeded close source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' close source > level 2, order 5, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' fonts > preloading latin modern fonts (second stage) fonts > 'fallback modern-designsize rm 12pt' is loaded suggestion > use \withring instead of \r lua error > lua error on line 12 in file ./helloworld.tex: token call, syntax: [ctxlua]:1: invalid escape sequence near '"\w' 2 3 \starttext 4 5 \directlua{ 6 local print = tex.print 7 \markdownPrepare 8 local file = assert(io.open("./_markdown_helloworld/54c7679395b5b68b2726f58d74fb517a.md.tex", "r")) 9 local input = assert(file:read("*a")) 10 assert(file:close()) 11 print(convert(input:gsub("\r\n?", "\n") .. "\n")) 12 >> } 13 14 \stoptext 15 mtx-context | fatal error: return code: 256 ```

If the above does not work, does the following work

This is even more interesting:

mkiv lua stats  > used config file: selfautoparent:/texmf/web2c/texmfcnf.lua
mkiv lua stats  > used cache path: /usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e
mkiv lua stats  > resource resolver: loadtime 0.004 seconds, 0 scans with scantime 0.000 seconds, 0 shared scans, 16 found files, scanned paths: <none>
mkiv lua stats  > stored bytecode data: 492 modules (0.235 sec), 99 tables (0.012 sec), 591 chunks (0.247 sec)
mkiv lua stats  > traced context: maxstack: 1565, freed: 0, unreachable: 1565
mkiv lua stats  > cleaned up reserved nodes: 66 nodes, 9 lists of 434
mkiv lua stats  > node memory usage: 6 attribute, 3 attribute_list, 2 glue, 58 glue_spec, 2 kern, 2 penalty, 2 temp
mkiv lua stats  > node list callback tasks: 9 unique task lists, 1 instances (re)created, 2 calls
mkiv lua stats  > used backend: pdf (backend for directly generating pdf output)
mkiv lua stats  > jobdata time: 0.001 seconds saving, 0.000 seconds loading
mkiv lua stats  > callbacks: internal: 0, file: 0, direct: 0, late: 0, function 0, total: 0 (0 per page)
mkiv lua stats  > loaded tex modules: 2 requested, all found (*-database t-markdown)
mkiv lua stats  > loaded patterns: en::1, load time: 0.000
mkiv lua stats  > loaded fonts: 3 files: latinmodern-math.otf, lmmono10-regular.otf, lmroman12-regular.otf
mkiv lua stats  > font engine: otf 3.116, afm 1.513, tfm 1.000, 8 instances, 3 shared in backend, 3 common vectors, 0 common hashes, load time 1.379 seconds
mkiv lua stats  > result saved in file: helloworld.pdf, compresslevel 1, objectcompresslevel 3
mkiv lua stats  > used platform: osx-64, type: unix, binary subtree: texmf-osx-64
mkiv lua stats  > used engine: luametatex version: 2.0911, functionality level: 20210527, format id: 575, compiler: clang
mkiv lua stats  > tex properties: 740032 hash slots used of 2097152, 46308 control sequences, approximate memory usage: 30 MB
mkiv lua stats  > lua properties: engine: lua 5.4, used memory: 53 MB, ctx: 51 MB, max: 51 MB, symbol mask: utf (τεχ)
mkiv lua stats  > runtime: 0.713 seconds
tex error       > tex error on line 0 in file :

lua error:

    run callback [33]: ...ext/tex/texmf-context/tex/context/base/mkxl/trac-deb.lmt:267: attempt to index a nil value (global 'file')
stack traceback:
    ...ext/tex/texmf-context/tex/context/base/mkxl/trac-deb.lmt:267: in upvalue 'function_0x7fd7954269b0'
    [string "local function_0x7fd791c9e8a0 = utilities.seq..."]:7: in local 'runner'
    ...ext/tex/texmf-context/tex/context/base/mkxl/luat-run.lmt:129: in function <...ext/tex/texmf-context/tex/context/base/mkxl/luat-run.lmt:126>

<empty file>

Full log

``` resolvers | formats | executing runner 'run luametatex format': /usr/local/context/tex/texmf-osx-64/bin/luametatex --jobname="helloworld" --fmt=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.fmt --lua=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.lui --c:currentrun=1 --c:fulljobname="./helloworld.tex" --c:input="./helloworld.tex" --c:kindofrun=1 --c:maxnofruns=9 --c:texmfbinpath="/usr/local/context/tex/texmf-osx-64/bin" fonts > beware: no fonts are loaded yet, using 'lm mono' in box system > system > ConTeXt ver: 2021.05.31 14:13 LMTX fmt: 2021.6.2 int: english/english system > system > 'cont-new.mkxl' loaded open source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > beware: some patches loaded from cont-new.mkiv close source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > files > jobname 'helloworld', input './helloworld', result 'helloworld' fonts > latin modern fonts are not preloaded languages > language 'en' is active open source > level 1, order 2, name './helloworld.tex' modules > 'markdown' is loaded open source > level 2, order 3, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' loading > ConTeXt User Module / markdown open source > level 3, order 4, name 'markdown/markdown.tex' markdown > Using mode 2: Direct Lua access. close source > level 3, order 4, name 'markdown/markdown.tex' modules > 'database' is loaded open source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' resolvers > lua > loading file '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.lua' succeeded close source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' close source > level 2, order 5, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' fonts > preloading latin modern fonts (second stage) fonts > 'fallback modern-designsize rm 12pt' is loaded close source > level 1, order 5, name './helloworld.tex' backend > xmp > using file '/usr/local/context/tex/texmf-context/tex/context/base/mkiv/lpdf-pdx.xml' mkiv lua stats > used config file: selfautoparent:/texmf/web2c/texmfcnf.lua mkiv lua stats > used cache path: /usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e mkiv lua stats > resource resolver: loadtime 0.004 seconds, 0 scans with scantime 0.000 seconds, 0 shared scans, 16 found files, scanned paths: mkiv lua stats > stored bytecode data: 492 modules (0.235 sec), 99 tables (0.012 sec), 591 chunks (0.247 sec) mkiv lua stats > traced context: maxstack: 1565, freed: 0, unreachable: 1565 mkiv lua stats > cleaned up reserved nodes: 66 nodes, 9 lists of 434 mkiv lua stats > node memory usage: 6 attribute, 3 attribute_list, 2 glue, 58 glue_spec, 2 kern, 2 penalty, 2 temp mkiv lua stats > node list callback tasks: 9 unique task lists, 1 instances (re)created, 2 calls mkiv lua stats > used backend: pdf (backend for directly generating pdf output) mkiv lua stats > jobdata time: 0.001 seconds saving, 0.000 seconds loading mkiv lua stats > callbacks: internal: 0, file: 0, direct: 0, late: 0, function 0, total: 0 (0 per page) mkiv lua stats > loaded tex modules: 2 requested, all found (*-database t-markdown) mkiv lua stats > loaded patterns: en::1, load time: 0.000 mkiv lua stats > loaded fonts: 3 files: latinmodern-math.otf, lmmono10-regular.otf, lmroman12-regular.otf mkiv lua stats > font engine: otf 3.116, afm 1.513, tfm 1.000, 8 instances, 3 shared in backend, 3 common vectors, 0 common hashes, load time 1.379 seconds mkiv lua stats > result saved in file: helloworld.pdf, compresslevel 1, objectcompresslevel 3 mkiv lua stats > used platform: osx-64, type: unix, binary subtree: texmf-osx-64 mkiv lua stats > used engine: luametatex version: 2.0911, functionality level: 20210527, format id: 575, compiler: clang mkiv lua stats > tex properties: 740032 hash slots used of 2097152, 46308 control sequences, approximate memory usage: 30 MB mkiv lua stats > lua properties: engine: lua 5.4, used memory: 53 MB, ctx: 51 MB, max: 51 MB, symbol mask: utf (τεχ) mkiv lua stats > runtime: 0.713 seconds tex error > tex error on line 0 in file : lua error: run callback [33]: ...ext/tex/texmf-context/tex/context/base/mkxl/trac-deb.lmt:267: attempt to index a nil value (global 'file') stack traceback: ...ext/tex/texmf-context/tex/context/base/mkxl/trac-deb.lmt:267: in upvalue 'function_0x7fd7954269b0' [string "local function_0x7fd791c9e8a0 = utilities.seq..."]:7: in local 'runner' ...ext/tex/texmf-context/tex/context/base/mkxl/luat-run.lmt:129: in function <...ext/tex/texmf-context/tex/context/base/mkxl/luat-run.lmt:126> resolvers | formats | executing runner 'run luametatex format': /usr/local/context/tex/texmf-osx-64/bin/luametatex --jobname="helloworld" --fmt=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.fmt --lua=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.lui --c:currentrun=2 --c:fulljobname="./helloworld.tex" --c:input="./helloworld.tex" --c:kindofrun=2 --c:maxnofruns=9 --c:texmfbinpath="/usr/local/context/tex/texmf-osx-64/bin" fonts > beware: no fonts are loaded yet, using 'lm mono' in box system > system > ConTeXt ver: 2021.05.31 14:13 LMTX fmt: 2021.6.2 int: english/english system > system > 'cont-new.mkxl' loaded open source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > beware: some patches loaded from cont-new.mkiv close source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > files > jobname 'helloworld', input './helloworld', result 'helloworld' fonts > latin modern fonts are not preloaded languages > language 'en' is active open source > level 1, order 2, name './helloworld.tex' modules > 'markdown' is loaded open source > level 2, order 3, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' loading > ConTeXt User Module / markdown open source > level 3, order 4, name 'markdown/markdown.tex' markdown > Using mode 2: Direct Lua access. close source > level 3, order 4, name 'markdown/markdown.tex' modules > 'database' is loaded open source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' resolvers > lua > loading file '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.lua' succeeded close source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' close source > level 2, order 5, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' fonts > preloading latin modern fonts (second stage) fonts > 'fallback modern-designsize rm 12pt' is loaded close source > level 1, order 5, name './helloworld.tex' backend > xmp > using file '/usr/local/context/tex/texmf-context/tex/context/base/mkiv/lpdf-pdx.xml' mkiv lua stats > used config file: selfautoparent:/texmf/web2c/texmfcnf.lua mkiv lua stats > used cache path: /usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e mkiv lua stats > resource resolver: loadtime 0.005 seconds, 0 scans with scantime 0.000 seconds, 0 shared scans, 16 found files, scanned paths: mkiv lua stats > stored bytecode data: 492 modules (0.225 sec), 99 tables (0.011 sec), 591 chunks (0.236 sec) mkiv lua stats > traced context: maxstack: 1565, freed: 0, unreachable: 1565 mkiv lua stats > cleaned up reserved nodes: 66 nodes, 9 lists of 434 mkiv lua stats > node memory usage: 6 attribute, 3 attribute_list, 2 glue, 58 glue_spec, 2 kern, 2 penalty, 2 temp mkiv lua stats > node list callback tasks: 9 unique task lists, 1 instances (re)created, 2 calls mkiv lua stats > used backend: pdf (backend for directly generating pdf output) mkiv lua stats > jobdata time: 0.001 seconds saving, 0.000 seconds loading mkiv lua stats > callbacks: internal: 0, file: 0, direct: 0, late: 0, function 0, total: 0 (0 per page) mkiv lua stats > randomizer: initialized with value 0.36860716768077 mkiv lua stats > loaded tex modules: 2 requested, all found (*-database t-markdown) mkiv lua stats > loaded patterns: en::1, load time: 0.000 mkiv lua stats > loaded fonts: 3 files: latinmodern-math.otf, lmmono10-regular.otf, lmroman12-regular.otf mkiv lua stats > font engine: otf 3.116, afm 1.513, tfm 1.000, 8 instances, 3 shared in backend, 3 common vectors, 0 common hashes, load time 0.134 seconds mkiv lua stats > result saved in file: helloworld.pdf, compresslevel 1, objectcompresslevel 3 mkiv lua stats > used platform: osx-64, type: unix, binary subtree: texmf-osx-64 mkiv lua stats > used engine: luametatex version: 2.0911, functionality level: 20210527, format id: 575, compiler: clang mkiv lua stats > tex properties: 740032 hash slots used of 2097152, 46307 control sequences, approximate memory usage: 30 MB mkiv lua stats > lua properties: engine: lua 5.4, used memory: 53 MB, ctx: 51 MB, max: 51 MB, symbol mask: utf (τεχ) mkiv lua stats > runtime: 0.442 seconds tex error > tex error on line 0 in file : lua error: run callback [33]: ...ext/tex/texmf-context/tex/context/base/mkxl/trac-deb.lmt:267: attempt to index a nil value (global 'file') stack traceback: ...ext/tex/texmf-context/tex/context/base/mkxl/trac-deb.lmt:267: in upvalue 'function_0x7ffb957e42b0' [string "local function_0x7ffb92f41af0 = utilities.seq..."]:7: in local 'runner' ...ext/tex/texmf-context/tex/context/base/mkxl/luat-run.lmt:129: in function <...ext/tex/texmf-context/tex/context/base/mkxl/luat-run.lmt:126> resolvers | formats | executing runner 'run luametatex format': /usr/local/context/tex/texmf-osx-64/bin/luametatex --jobname="helloworld" --fmt=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.fmt --lua=/usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.lui --c:currentrun=3 --c:fulljobname="./helloworld.tex" --c:input="./helloworld.tex" --c:kindofrun=2 --c:maxnofruns=9 --c:texmfbinpath="/usr/local/context/tex/texmf-osx-64/bin" fonts > beware: no fonts are loaded yet, using 'lm mono' in box system > system > ConTeXt ver: 2021.05.31 14:13 LMTX fmt: 2021.6.2 int: english/english system > system > 'cont-new.mkxl' loaded open source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > beware: some patches loaded from cont-new.mkiv close source > level 1, order 1, name '/usr/local/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl' system > files > jobname 'helloworld', input './helloworld', result 'helloworld' fonts > latin modern fonts are not preloaded languages > language 'en' is active open source > level 1, order 2, name './helloworld.tex' modules > 'markdown' is loaded open source > level 2, order 3, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' loading > ConTeXt User Module / markdown open source > level 3, order 4, name 'markdown/markdown.tex' markdown > Using mode 2: Direct Lua access. close source > level 3, order 4, name 'markdown/markdown.tex' modules > 'database' is loaded open source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' resolvers > lua > loading file '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.lua' succeeded close source > level 3, order 5, name '/usr/local/context/tex/texmf-context/tex/context/modules/mkiv/m-database.mkiv' close source > level 2, order 5, name '/usr/local/context/tex/texmf-modules/tex/context/third/markdown/t-markdown.tex' fonts > preloading latin modern fonts (second stage) fonts > 'fallback modern-designsize rm 12pt' is loaded close source > level 1, order 5, name './helloworld.tex' backend > xmp > using file '/usr/local/context/tex/texmf-context/tex/context/base/mkiv/lpdf-pdx.xml' mkiv lua stats > used config file: selfautoparent:/texmf/web2c/texmfcnf.lua mkiv lua stats > used cache path: /usr/local/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e mkiv lua stats > resource resolver: loadtime 0.004 seconds, 0 scans with scantime 0.000 seconds, 0 shared scans, 16 found files, scanned paths: mkiv lua stats > stored bytecode data: 492 modules (0.225 sec), 99 tables (0.011 sec), 591 chunks (0.236 sec) mkiv lua stats > traced context: maxstack: 1565, freed: 0, unreachable: 1565 mkiv lua stats > cleaned up reserved nodes: 66 nodes, 9 lists of 434 mkiv lua stats > node memory usage: 6 attribute, 3 attribute_list, 2 glue, 58 glue_spec, 2 kern, 2 penalty, 2 temp mkiv lua stats > node list callback tasks: 9 unique task lists, 1 instances (re)created, 2 calls mkiv lua stats > used backend: pdf (backend for directly generating pdf output) mkiv lua stats > jobdata time: 0.001 seconds saving, 0.000 seconds loading mkiv lua stats > callbacks: internal: 0, file: 0, direct: 0, late: 0, function 0, total: 0 (0 per page) mkiv lua stats > randomizer: resumed with value 0.36860716768077 mkiv lua stats > loaded tex modules: 2 requested, all found (*-database t-markdown) mkiv lua stats > loaded patterns: en::1, load time: 0.000 mkiv lua stats > loaded fonts: 3 files: latinmodern-math.otf, lmmono10-regular.otf, lmroman12-regular.otf mkiv lua stats > font engine: otf 3.116, afm 1.513, tfm 1.000, 8 instances, 3 shared in backend, 3 common vectors, 0 common hashes, load time 0.135 seconds mkiv lua stats > result saved in file: helloworld.pdf, compresslevel 1, objectcompresslevel 3 mkiv lua stats > used platform: osx-64, type: unix, binary subtree: texmf-osx-64 mkiv lua stats > used engine: luametatex version: 2.0911, functionality level: 20210527, format id: 575, compiler: clang mkiv lua stats > tex properties: 740032 hash slots used of 2097152, 46307 control sequences, approximate memory usage: 30 MB mkiv lua stats > lua properties: engine: lua 5.4, used memory: 53 MB, ctx: 51 MB, max: 51 MB, symbol mask: utf (τεχ) mkiv lua stats > runtime: 0.443 seconds tex error > tex error on line 0 in file : lua error: run callback [33]: ...ext/tex/texmf-context/tex/context/base/mkxl/trac-deb.lmt:267: attempt to index a nil value (global 'file') stack traceback: ...ext/tex/texmf-context/tex/context/base/mkxl/trac-deb.lmt:267: in upvalue 'function_0x7f904d6370f0' [string "local function_0x7f904d4962a0 = utilities.seq..."]:7: in local 'runner' ...ext/tex/texmf-context/tex/context/base/mkxl/luat-run.lmt:129: in function <...ext/tex/texmf-context/tex/context/base/mkxl/luat-run.lmt:126> system | total runtime: 1.654 seconds of 1.705 seconds ```

[Witiko]

Let's expand the command, this should produce the same result as the above code:

This is more interesting:

That was just a syntax error on my part, sorry.

This is even more interesting:

That really is interesting. How about just this then:

\usemodule[t][markdown]

\starttext

\directlua{
  local print = tex.print
}

\stoptext

Surely, this will run fine?

[andreiborisov]

This was just a syntax error on my part, sorry.

That was my thought! I must say, that's quite a journey and I'm enjoying helping you to pinpoint the problem: I feel like I'm Dr. Watson😂

This really is interesting. How about just this then

This compiles fine.

[Witiko]

That was my thought! I must say, that's quite a journey and I'm enjoying helping you to pinpoint the problem: I feel like I'm Dr. Watson😂

It's all rather thrilling, isn't it? 😉

This compiles fine.

Let's unpack the \markdownPrepare version then. The following code should again produce the attempt to index a nil value (global 'file') error:

\usemodule[t][markdown]

\starttext

\directlua{
  local print = tex.print
  local lfs = require("lfs")
  local cacheDir = "\markdownOptionCacheDir"
  if not lfs.isdir(cacheDir) then
    assert(lfs.mkdir(cacheDir))
  end
  local md = require("markdown")
  local convert = md.new(\markdownLuaOptions)
}

\stoptext

If it does, then we just start deleting the individual lines starting with local convert = md.new(\markdownLuaOptions), until we've found the culprit. When you have eliminated the impossible, whatever remains, however improbable, must be the truth!

It might also be useful to take a look at what is in file ...ext/tex/texmf-context/tex/context/base/mkxl/trac-deb.lmt on line 267 and the surrounding lines, which is where the error is eventually raised. It may not help us, but it will definitely interest ConTeXt developers if this is their bug.

[andreiborisov]

The following code should again produce the attempt to index a nil value (global 'file') error

You're correct!

When you have eliminated the impossible, whatever remains, however improbable, must be the truth!

And the improbable is this line: local convert = md.new(\markdownLuaOptions), when I comment it out, compilation proceeds without errors.

[Witiko]

That's what I was afraid of: the issue is somewhere in the Markdown parser. This will be a little difficult to bisect without some knowledge of the syntax of Lua. Perhaps I will need to reproduce your setup afterall.

If you'd like to dig in, there is the following definition at the end of markdown.lua:

function M.new(options)
  local writer = M.writer.new(options)
  local reader = M.reader.new(writer, options)
  return reader.convert
end

If you could comment out return reader.convert as follows:

function M.new(options)
  local writer = M.writer.new(options)
  local reader = M.reader.new(writer, options)
  -- return reader.convert
end

and continue upwards until the error ceases. This will help us narrow down whether the error originates in the Markdown reader of the TeX writer. 🙂

[Witiko]

Perhaps I will need to reproduce your setup afterall.

If I could have a ZIP archive with your working directory, I would try to reproduce your error with the LMTX Docker image.

[andreiborisov]

If I could have a ZIP archive with your working directory, I would try to reproduce your error with the LMTX Docker image.

It's just a helloworld.tex file:

\usemodule[t][markdown]

\starttext

\directlua{
  local print = tex.print
  local lfs = require("lfs")
  local cacheDir = "\markdownOptionCacheDir"
  if not lfs.isdir(cacheDir) then
    assert(lfs.mkdir(cacheDir))
  end
  local md = require("markdown")
  local convert = md.new(\markdownLuaOptions)
}

\stoptext

Run it with context helloworld.tex.

[andreiborisov]

This will help us narrow down whether the error originates in the Markdown reader of the TeX writer. 🙂

Seems to be the TeX writer.

[Witiko]

Seems to be the TeX writer.

Well, to crack the case, we will have to look at the following lines in the markdown.lua file and start deleting the insides of the M.writer.new function from the bottom (return self):

function M.writer.new(options)
  local self = {}
  options = options or {}
  -- another 380 lines of code
  return self
end

Going one by one would take ages, so we should bisect, i.e. always delete half of what remains if the error persists or restore half of what we deleted if the error is gone. This way, we should find the culprit in no more than 9 steps. We just must be careful not to cause a syntax error by our deletions.

[Witiko]

If I should place a bet on whodunit, I would say that the culprit is the following function:

  local languages_json = (function()
    local ran_ok, kpse = pcall(require, "kpse")
    ...
  end)()

This seems to be the only part that interfaces with LuaMetaTeX and tries to load the KPathSea library. The rest of the code should be pure in the sense that it does not really touch any external libraries.

[andreiborisov]

$ git clone 'https://gitlab.com/islandoftex/images/context.git'
$ cd context
$ docker build --file Dockerfile.lmtx -t context:lmtx .
$ wget 'https://github.com/Witiko/markdown/releases/download/2.9.0/markdown.zip'
$ unzip markdown.zip

After this you need to add the package files into the ConTeXt installation, it's easier to do with TDS archive:

$ unzip markdown.tds.zip -d tds
$ docker run -it -v "$PWD"/tds:/context/tex/texmf-modules -w /context/tex/texmf-modules --rm context:lmtx
root@303f8191a9b8:/context/tex/texmf-modules# mtxrun --generate

[andreiborisov]

If I should place a bet on whodunit, I would say that the culprit is the following function

And you'd win this bet!) When I comment it out, there are no errors.

[Witiko]

So in the end, it was just my carelessness, sorry. Unlike in languages such as Python, where all variable declarations are local unless explicitly stated to be global, it's the opposite in Lua. A single missing local keyword in 575a1e0 caused the Markdown package to override the internal file variable of LuaMetaTeX. I fixed it in https://github.com/Witiko/markdown/commit/1153afb5a13a95bae11c1c95dcfb4f6d0705db9e, many thanks for the assistance, Dr. Watson would have been proud!

[Witiko]

I can't reproduce this on my TeX Live 2021 installation and neither do I see anything particularly suspicious in the logs. It's not too worrying, because all the outputs are produced correctly, otherwise the tests would fail, but I would be happier if I understood. 😅

This is now also solved. 😉

[andreiborisov]

Brilliant!

Dr. Watson would have been proud!

😎

Would you mind making a GitHub release with the changes?

[Witiko]

That's the plan, but there is still a couple of milestones I'd like to hit before 2.10.0. Should I hurry it up?

[andreiborisov]

@Witiko no, don't worry, I can compile from the source for now.

Thank you again for taking the time to tackle this!🚀

[Witiko]

@andreiborisov You can download the up-to-date ZIP files with the compiled package from GitHub Actions. They are produced as artifacts and only deleted after 90 days. For some reason (see https://github.com/actions/upload-artifact/issues/39), GitHub Actions puts each of my ZIP archives into another ZIP archive, which is a little annoying, but definitely better than having to compile manually!