Hyphen character missing when using luatex with DVI output

jeremy-engel commented 7 months ago

When using lualatex --output-format=dvi, if the hyphen character is followed by a non-space character, then the hyphen is missing from the output. Consider the following example:

\documentclass{article}

\begin{document}

A-B

-

A-\hspace{0pt}B

A-

-B

\end{document}

Compiling with latex and converting the DVI file with dvisvgm results in with-latex

With lualuatex --output-format=dvi, the resulting SVG file after conversion is with-lualatex

This is with the following versions:

This is LuaTeX, Version 1.17.0 (TeX Live 2023)
Development id: 7581

dvisvgm 3.0.3 (x86_64-pc-linux-gnu)
-----------------------------------
brotli:      1.0.9
clipper:     6.2.1
freetype:    2.13.0
Ghostscript: 9.55.0
kpathsea:    6.3.5
potrace:     1.16
xxhash:      0.8.1
zlib:        1.2.13

muzimuzhi commented 7 months ago

Looks like \automatichyphenmode=2 (defaults 0) works, but I don't know why.

The LuaTeX dvi outputs with \automatichyphenmode set to 0, 1, and 2 are already different, so the culprit might not be dvisvgm.

\documentclass{article}
\pagestyle{empty}
\def\testhyphens{%
  A-B\par
  -\par
  A-\hspace{0pt}B\par
  A-\par
  -B\par\bigskip}

\begin{document}
\automatichyphenmode=0
\testhyphens

\automatichyphenmode=1
\testhyphens

\automatichyphenmode=2
\testhyphens
\end{document}

Running

lualatex --output-format=dvi dvisvgm-gh255
dviasm dviasm dvisvgm-gh255.dvi > dvisvgm-gh255.dump

then in dvisvgm-gh255.dump

\automatichyphenmode=0 \automatichyphenmode=1 \automatichyphenmode=2

`\automatichyphenmode=0`	`\automatichyphenmode=1`	`\automatichyphenmode=2`
push: right: 77pt down: 63pt fnt: [...] at 10pt set: '\x1b"' pop: push: right: 77pt down: 75pt set: '@' pop: push: right: 77pt down: 87pt set: '\x1b@"' pop: push: right: 77pt down: 99pt set: '\x1b@' pop: push: right: 77pt down: 111pt set: '"' pop:	push: right: 77pt down: 135pt set: '\x1b"' pop: push: right: 77pt down: 147pt set: '@' pop: push: right: 77pt down: 159pt set: '\x1b@"' pop: push: right: 77pt down: 171pt set: '\x1b@' pop: push: right: 77pt down: 183pt set: '@"' pop:	push: right: 77pt down: 207pt set: '\x1b@"' pop: push: right: 77pt down: 219pt set: '@' pop: push: right: 77pt down: 231pt set: '\x1b@"' pop: push: right: 77pt down: 243pt set: '\x1b@' pop: push: right: 77pt down: 255pt set: '@"' pop:

push:
  right: 77pt
  down: 63pt
  fnt: [...] at 10pt
  set: '\x1b"'
pop:
push:
  right: 77pt
  down: 75pt
  set: '@'
pop:
push:
  right: 77pt
  down: 87pt
  set: '\x1b@"'
pop:
push:
  right: 77pt
  down: 99pt
  set: '\x1b@'
pop:
push:
  right: 77pt
  down: 111pt
  set: '"'
pop:

push:
  right: 77pt
  down: 135pt
  set: '\x1b"'
pop:
push:
  right: 77pt
  down: 147pt
  set: '@'
pop:
push:
  right: 77pt
  down: 159pt
  set: '\x1b@"'
pop:
push:
  right: 77pt
  down: 171pt
  set: '\x1b@'
pop:
push:
  right: 77pt
  down: 183pt
  set: '@"'
pop:

push:
  right: 77pt
  down: 207pt
  set: '\x1b@"'
pop:
push:
  right: 77pt
  down: 219pt
  set: '@'
pop:
push:
  right: 77pt
  down: 231pt
  set: '\x1b@"'
pop:
push:
  right: 77pt
  down: 243pt
  set: '\x1b@'
pop:
push:
  right: 77pt
  down: 255pt
  set: '@"'
pop:

You can see the first and last line of each test group with \automatichyphenmode=2 are different from when \automatichyphenmode is set to other values.

mgieseki commented 7 months ago

@muzimuzhi Awesome, thank you for the detailed analysis!

dvisvgm indeed converts the DVI files correctly. The initial one created by LuaTeX doesn't contain the missing hyphens so that dvisvgm can't process them.

muzimuzhi commented 7 months ago

With lualuatex --output-format=dvi and \automatichyphenmode=(0|1), one also got

Missing character: There is no @ (U+0040) in font nullfont!
Missing character: There is no @ (U+0040) in font nullfont!

(\tracinglostchars=3 can be used to convert them into errors.)

muzimuzhi commented 7 months ago

Turns out it's a luaotfload problem, see latex3/luaotfload#270. luaotfload maintainer Marcel Krüger has already kindly pushed a fix to dev branch, which you can test with locally by executing

git clone --branch=dev --depth=1 git@github.com:latex3/luaotfload.git
cd luaotfload
# needs l3build package
l3build install

cd /path/to/tex/files
# needs latex-base-dev and latex-bin-dev packages perhaps
# should be equivalent to "lualatex-dev --output-format=dvi"
dvilualatex-dev dvisvgm-gh255
dvisvgm --font-format=woff2 dvisvgm-gh255

jeremy-engel commented 7 months ago

Great, thanks for the detailed explanation and for hunting down that luaotfload issue. I just confirmed that the dev branch for luaotfload does resolve this issue (and the problem I originally had that evolved into the MWE) as you said. I'll close this as the issue is in luaotfload and already fixed pending release. Thanks again.

mgieseki / dvisvgm

Hyphen character missing when using luatex with DVI output #255