search

latex3 / l3build

A testing and building system for LaTeX
LaTeX Project Public License v1.3c
83 stars 14 forks source link

Cannot `check` or `save` with ConTeXt. #232

Closed gucci-on-fleek closed 11 months ago

gucci-on-fleek commented 2 years ago

l3build modifies the $TEXMFCNF variable, which makes ConTeXt tests break completely. Commenting out these lines:

https://github.com/latex3/l3build/blob/fb53b22f1eb2f69ded2f129540a9ee8e9344537a/l3build-check.lua#L794-L796

allows ConTeXt to run, although this causes tdsdirs to be ignored. This issue is present in both ConTeXt LMTX/MkXL and MkIV.

This mailing list thread suggests that you can use the $TEXMFAUXTREES variable instead, although I have not tried this.

Setup

I'm currently running TL 2021, l3build 2022-03-15, and ConTeXt MkXL 2022.04.20 19:18 on Windows 10 x86_64.

Files

Create these files to test:

build.lua

module = "test"

testfiledir = "."
checkengines = { "luatex" }
checkformat = "context"

context.lvt

\starttext
    Hello!
\stoptext

context.tlg

<empty file>

Unmodified l3build

Saving

l3build save context
Creating and copying context.tlg
mtxrun          | forcing cache reload
resolvers       | resolving | looking for regular 'texmfcnf.lua' on weird path '.' from specification '.'
resolvers       | resolving | looking for fallback 'contextcnf.lua' on weird path '.' from specification '.'
resolvers       | resolving |
resolvers       | resolving | warning: no lua configuration files found
resolvers       | resolving | no texmf paths are defined (using TEXMF)
resolvers       | resolving |
mtxrun          | the resolver databases are not present or outdated
resolvers       | resolving | using suffix based filetype 'scripts'
resolvers       | resolving | remembering file 'mtx-context.lua' using hash 'lua::mtx-context.lua'
resolvers       | resolving | using suffix based filetype 'scripts'
resolvers       | resolving | remembering file 'mtx-contexts.lua' using hash 'lua::mtx-contexts.lua'
resolvers       | resolving | remembered file 'mtx-context.lua'
resolvers       | resolving | using suffix based filetype 'scripts'
resolvers       | resolving | remembering file 'mtx-t-mtx-context.lua' using hash 'lua::mtx-t-mtx-context.lua'
resolvers       | resolving | using suffix based filetype 'scripts'
resolvers       | resolving | remembering file 'mtx-t-mtx-contexts.lua' using hash 'lua::mtx-t-mtx-contexts.lua'
resolvers       | resolving | remembered file 'mtx-t-mtx-context.lua'
resolvers       | resolving | remembered file 'mtx-context.lua'
mtxrun          | unknown script 'mtx-context.lua' or 'mtx-mtx-context.lua'
C:\texlive\2021\bin\win32\runscript.tlu:935: ...exlive/2021/texmf-dist/scripts/l3build/l3build-check.lua:91: ./build/test/context.log: No such file or directory

Testing

l3build check
Running checks on
  context (1/1)
C:\texlive\2021\bin\win32\runscript.tlu:935: ...exlive/2021/texmf-dist/scripts/l3build/l3build-check.lua:91: ./build/test/context.log: No such file or directory

Patchedl3build

Saving

l3build save context
reating and copying context.tlg

resolvers       | formats | executing runner 'run luametatex format': C:/context/tex/texmf-win64/bin/luametatex --jobname="./context.lvt" --fmt=C:/context/tex/texmf-cache/luametatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.fmt --lua=C:/context/tex/texmf-cache/luametatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luametatex/cont-en.lui  --c:currentrun=1 --c:fulljobname="./context.lvt" --c:input="./context.lvt" --c:interaction="nonstopmode" --c:kindofrun=1 --c:maxnofruns=9 --c:texmfbinpath="C:/context/tex/texmf-win64/bin"
system          >
system          > ConTeXt  ver: 2022.04.20 19:18 LMTX  fmt: 2022.5.1  int: english/english
system          >
system          > 'cont-new.mkxl' loaded
open source     > level 1, order 1, name 'C:/context/tex/texmf-context/tex/context/base/mkxl/cont-new.mkxl'

[…]

mkiv lua stats  > lua properties: engine: lua 5.4, used memory: 54 MB, ctx: 52 MB, max: 55 MB, symbol mask: utf (¤ä╬Á¤ç)
mkiv lua stats  > runtime: 0.783 seconds, 1 processed pages, 1 shipped pages, 1.277 pages/second
system          | total runtime: 1.593 seconds of 1.667 second

Testing

l3build check
Running checks on
  context (1/1)

  All checks passed
u-fischer commented 2 years ago

This mailing list thread suggests that you can use the $TEXMFAUXTREES variable instead,

I don't think that this is an option. I would not like it at all if l3build would overwrite my texmfauxtrees settings. And using tlmgr conf auxtrees would be too slow.

l3build modifies the $TEXMFCNF variable, which makes ConTeXt tests break completely.

l3build could probably add some option which suppresses the setting. But as I commented in the thread, it is imho more a context problem. As far as I know context doesn't use kpathsea anymore so it is not quite clear why they make use of a kpathsea variable and break if it is changed. I stopped to use TEXMFCNF years ago as every careless call to context to test something resulted in havoc.

gucci-on-fleek commented 2 years ago

As far as I know context doesn't use kpathsea anymore

The ConTeXt formats (luametatex, luatex) don't use kpathsea, but the I believe that the context executable (texlua) still does. I could be completely wrong here though.

it is not quite clear why they make use of a kpathsea variable and break if it is changed

There's this comment in the code:

https://github.com/contextgarden/context-mirror/blob/b155eafa1e2d73692eed08ecabeb5ad589ac11dc/tex/context/base/mkiv/data-res.lua#L83-L136

but I can't say that I understand it at all. There's also this:

https://github.com/contextgarden/context-mirror/blob/b155eafa1e2d73692eed08ecabeb5ad589ac11dc/web2c/readme.txt#L1-L5

but its suggestion of copying contextcnf.lua and texmfcnf.lua doesn't actually work.

I did manage to find a way to get l3build to run ConTeXt and to successfully pass it a texmf tree to search:

diff --git a/regression-test.old b/regression-test.tex
index 0a1ef5f..e2db047 100755
--- a/regression-test.old
+++ b/regression-test.tex
@@ -292,7 +292,15 @@
     \fi
   \fi
 \fi
+
+\ifdefined\startluacode
+    \startluacode
+        local hashes = resolvers.gethashes()
+        hashes[#hashes + 1] = {cache = false, type = "file", name = os.getenv("EXTRATEXMF")}
+        resolvers.generators.file({filename = os.getenv("EXTRATEXMF")})
+    \stopluacode
+\fi
 \reset@catcodes
diff --git a/l3build-check.old b/l3build-check.lua
index f016c25..d267f1c 100755
--- a/l3build-check.old
+++ b/l3build-check.lua
@@ -747,8 +747,16 @@ function runtest(name, engine, hide, ext, test_type, breakout)
   local function setup(file)
     return " -jobname=" .. name .. tokens .. ' "\\input ' .. file .. '" '
   end
+  local texmfcnf
   if match(checkformat,"^context$") then
     function setup(file) return tokens .. ' "' .. file .. '" '  end
+    -- Only works for a `localtexmf()` that has a single path
+    texmfcnf = os_setenv .. " EXTRATEXMF=" .. localtexmf():gsub(";", ""):gsub("//", "/")
+    .. os_concat
+  else
+      -- Allow for local texmf files
+      texmfcnf = os_setenv .. " TEXMFCNF=." .. os_pathsep
+        .. os_concat
   end
   local basename = testdir .. "/" .. name
   local gen_file = basename .. test_type.generated
@@ -781,9 +789,7 @@ function runtest(name, engine, hide, ext, test_type, breakout)
       -- Avoid spurious output from (u)pTeX
       os_setenv .. " GUESS_INPUT_KANJI_ENCODING=0"
         .. os_concat ..
-      -- Allow for local texmf files
-      os_setenv .. " TEXMFCNF=." .. os_pathsep
-        .. os_concat ..
+      texmfcnf ..
       set_epoch_cmd(epoch, forcecheckepoch) ..
       -- Ensure lines are of a known length
       os_setenv .. " max_print_line=" .. maxprintline

This patch is fairly awful though; it seems to work, but it's also really ugly and fragile. Another solution that also works is setting $TEXMFHOME, which is likely easier. Let me know if you want me to clean this up into a proper PR.

josephwright commented 11 months ago

I've committed a partial fix: this simply avoids setting TEXMCNF for ConTeXt at all. If you could test, that would be great. (I also want to address LuaTeX vs LuaMetaTeX - coming up.)

gucci-on-fleek commented 11 months ago

@josephwright

I've committed a partial fix: this simply avoids setting TEXMCNF for ConTeXt at all. If you could test, that would be great

I had been using a similar patch for just over a year and it worked great. I've just installed 2023-07-17 and my standard l3build config now works with ConTeXt, so I believe that this issue is resolved.

(I also want to address LuaTeX vs LuaMetaTeX - coming up.)

I've been using l3build to test MkIV+MkXL for almost a year and a half now, and my earlier patch was all that I needed to make it work. If you're curious, you can look at my l3build config at https://github.com/gucci-on-fleek/lua-widow-control, but the way I'm using l3build is very idiosyncratic.

I've had a pretty good experience with using l3build to test ConTeXt. My only complaint is that it's quite annoying to set up testing if you need to support 4 different formats like I do. Although I'm probably just doing something wrong.

Anyways, thanks for the fix!

josephwright commented 11 months ago

I've committed a partial fix: this simply avoids setting TEXMCNF for ConTeXt at all. If you could test, that would be great

I had been using a similar patch for just over a year and it worked great. I've just installed 2023-07-17 and my standard l3build config now works with ConTeXt, so I believe that this issue is resolved.

Indeed, but leaves open the question of what happens when you need non-standard settings.

(I also want to address LuaTeX vs LuaMetaTeX - coming up.)

I've been using l3build to test MkIV+MkXL for almost a year and a half now, and my earlier patch was all that I needed to make it work.

I was thinking of the GitHub Actions side of things: something is not quite right.