Closed benwbrum closed 3 months ago
So I tested this on the old FtP server, then did a lot of comparison on the verbose outputs to figure out what might be the difference. While many packages were missing from production, and I installed them, I don't know that made any difference.
What I did eventually notice is that the working/old server encodes the problematic text like so:
\underline{'tis well}
while the new one encodes it like this:
\ul{\textquotesingle tis well}
Changing our test file on the production server to be this encoding works -- a pdf is generated without the error.
We believe this is being done by pandoc.
Pandoc comparison on the new vs old server:
new/production (broken) server:
fromthepage@fromthepage:~$ pandoc --version pandoc 3.1.3 Features: -server +lua Scripting engine: Lua 5.4 User data directory: /home/fromthepage/.local/share/pandoc Copyright (C) 2006-2023 John MacFarlane. Web: https://pandoc.org This is free software; see the source for copying conditions. There is no warranty, not even for merchantability or fitness for a particular purpose.
Old (working) server:
fromthepage@fromthepage:~$ ssh fromthepage@50.116.44.131 pandoc --version fromthepage@50.116.44.131's password: pandoc 2.13 Compiled with pandoc-types 1.22, texmath 0.12.2, skylighting 0.10.5, citeproc 0.3.0.9, ipynb 0.1.0.1 User data directory: /home/fromthepage/.local/share/pandoc Copyright (C) 2006-2021 John MacFarlane. Web: https://pandoc.org This is free software; see the source for copying conditions. There is no warranty, not even for merchantability or fitness for a particular purpose.
There's a lot of differences.
Confirmed this works on the new server as well:
\ul{'tis well}
So the problem is definitely the \textquotesingle
We've decided that the approach we should take is replacing (using gsub) \textquotesingle
with a '
somewhere around here: https://github.com/benwbrum/fromthepage/blob/029217f8c079a81907ddb7e32630dd0407b906b0/app/helpers/export_helper.rb#L71
(after the html to md conversion using pandoc, before md to latex to pdf)
Customer has decided to remove the apostrophe. Keeping this open, but not a high priority fix.
Customer has encountered an underlined apostrophe that's really important. Upping the priority on this again.
Xelatex on production (failing): XeTeX 3.141592653-2.6-0.999995 (TeX Live 2023/Debian)
Xelatex on old server (working): XeTeX 3.14159265-2.6-0.99998 (TeX Live 2017/Debian)
Isolating the pandoc command that fails on production produces this:
pandoc --from markdown+superscript+pipe_tables -o /tmp/bad_font.pdf /tmp/bad_font.md --pdf-engine=xelatex --verbose --abbreviations=/dev/null -V colorlinks=true 2> badlatex.out
[makePDF] temp dir:
/tmp/tex2pdf.-7cab912f892c775c
[makePDF] Command line:
xelatex "-halt-on-error" "-interaction" "nonstopmode" "-output-directory" "/tmp/tex2pdf.-7cab912f892c775c" "/tmp/tex2pdf.-7cab912f892c775c/input.tex"
[makePDF] Relevant environment variables:
("TEXINPUTS","/tmp/tex2pdf.-7cab912f892c775c:")
("TEXMFOUTPUT","/tmp/tex2pdf.-7cab912f892c775c")
("SHELL","/bin/bash")
("PWD","/home/fromthepage")
("HOME","/home/fromthepage")
("LANG","en_US.UTF-8")
("PATH","/home/fromthepage/bin:/home/fromthepage/.local/bin:/home/fromthepage/.rvm/gems/ruby-2.7.3/bin:/home/fromthepage/.rvm/gems/ruby-2.7.3@global/bin:/usr/local/rvm/rubies/ruby-2.7.3/bin:/usr/local/sbi
n:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/share/rvm/bin:/usr/local/texlive/2017/bin/x86_64-linux")
[makePDF] Source:
% Options for packages loaded elsewhere
\PassOptionsToPackage{unicode}{hyperref}
\PassOptionsToPackage{hyphens}{url}
\PassOptionsToPackage{dvipsnames,svgnames,x11names}{xcolor}
%
\documentclass[
]{article}
\usepackage{amsmath,amssymb}
\usepackage{iftex}
\ifPDFTeX
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{textcomp} % provide euro and other symbols
\else % if luatex or xetex
\usepackage{unicode-math} % this also loads fontspec
\defaultfontfeatures{Scale=MatchLowercase}
\defaultfontfeatures[\rmfamily]{Ligatures=TeX,Scale=1}
\fi
\usepackage{lmodern}
\ifPDFTeX\else
% xetex/luatex font selection
\fi
% Use upquote if available, for straight quotes in verbatim environments
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
\IfFileExists{microtype.sty}{% use microtype if available
\usepackage[]{microtype}
\UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
}{}
\makeatletter
\@ifundefined{KOMAClassName}{% if non-KOMA class
\IfFileExists{parskip.sty}{%
\usepackage{parskip}
}{% else
\setlength{\parindent}{0pt}
\setlength{\parskip}{6pt plus 2pt minus 1pt}}
}{% if KOMA class
\KOMAoptions{parskip=half}}
\makeatother
\usepackage{xcolor}
\ifLuaTeX
\usepackage{luacolor}
\usepackage[soul]{lua-ul}
\else
\usepackage{soul}
\fi
\setlength{\emergencystretch}{3em} % prevent overfull lines
\providecommand{\tightlist}{%
\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
\setcounter{secnumdepth}{-\maxdimen} % remove section numbering
\ifLuaTeX
\usepackage{selnolig} % disable illegal ligatures
\fi
\IfFileExists{bookmark.sty}{\usepackage{bookmark}}{\usepackage{hyperref}}
\IfFileExists{xurl.sty}{\usepackage{xurl}}{} % add URL line breaks if available
\urlstyle{same}
\hypersetup{
colorlinks=true,
linkcolor={Maroon},
filecolor={Maroon},
citecolor={Blue},
urlcolor={Blue},
pdfcreator={LaTeX via pandoc}}
\author{}
\date{}
\begin{document}
\hypertarget{advertisement-for-slave-auction}{%
\section{Advertisement for slave
auction}\label{advertisement-for-slave-auction}}
\ul{\textquotesingle tis well}
\end{document}
[makePDF] Run #1
This is XeTeX, Version 3.141592653-2.6-0.999995 (TeX Live 2023/Debian) (preloaded format=xelatex)
restricted \write18 enabled.
entering extended mode
(/tmp/tex2pdf.-7cab912f892c775c/input.tex
LaTeX2e <2023-11-01> patch level 1
L3 programming layer <2024-01-22>
(/usr/share/texlive/texmf-dist/tex/latex/base/article.cls
Document Class: article 2023/05/17 v1.4n Standard LaTeX document class
(/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo))
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsmath.sty
For additional information on amsmath, use the `?' option.
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amstext.sty
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsgen.sty))
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsbsy.sty)
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsopn.sty))
(/usr/share/texlive/texmf-dist/tex/latex/amsfonts/amssymb.sty
(/usr/share/texlive/texmf-dist/tex/latex/amsfonts/amsfonts.sty))
(/usr/share/texlive/texmf-dist/tex/generic/iftex/iftex.sty)
(/usr/share/texlive/texmf-dist/tex/latex/unicode-math/unicode-math.sty
(/usr/share/texlive/texmf-dist/tex/latex/l3kernel/expl3.sty
(/usr/share/texlive/texmf-dist/tex/latex/l3backend/l3backend-xetex.def))
(/usr/share/texlive/texmf-dist/tex/latex/unicode-math/unicode-math-xetex.sty
(/usr/share/texlive/texmf-dist/tex/latex/l3packages/xparse/xparse.sty)
(/usr/share/texlive/texmf-dist/tex/latex/l3packages/l3keys2e/l3keys2e.sty)
(/usr/share/texlive/texmf-dist/tex/latex/fontspec/fontspec.sty
(/usr/share/texlive/texmf-dist/tex/latex/fontspec/fontspec-xetex.sty
(/usr/share/texlive/texmf-dist/tex/latex/base/fontenc.sty)
(/usr/share/texlive/texmf-dist/tex/latex/fontspec/fontspec.cfg)))
(/usr/share/texlive/texmf-dist/tex/latex/base/fix-cm.sty
(/usr/share/texlive/texmf-dist/tex/latex/base/ts1enc.def))
(/usr/share/texlive/texmf-dist/tex/latex/unicode-math/unicode-math-table.tex)))
(/usr/share/texmf/tex/latex/lm/lmodern.sty)
(/usr/share/texlive/texmf-dist/tex/latex/upquote/upquote.sty
(/usr/share/texlive/texmf-dist/tex/latex/base/textcomp.sty))
(/usr/share/texlive/texmf-dist/tex/latex/microtype/microtype.sty
(/usr/share/texlive/texmf-dist/tex/latex/graphics/keyval.sty)
(/usr/share/texlive/texmf-dist/tex/latex/etoolbox/etoolbox.sty)
(/usr/share/texlive/texmf-dist/tex/latex/microtype/microtype-xetex.def)
(/usr/share/texlive/texmf-dist/tex/latex/microtype/microtype.cfg))
(/usr/share/texlive/texmf-dist/tex/latex/parskip/parskip.sty
(/usr/share/texlive/texmf-dist/tex/latex/kvoptions/kvoptions.sty
(/usr/share/texlive/texmf-dist/tex/generic/ltxcmds/ltxcmds.sty)
(/usr/share/texlive/texmf-dist/tex/latex/kvsetkeys/kvsetkeys.sty)))
(/usr/share/texlive/texmf-dist/tex/latex/xcolor/xcolor.sty
(/usr/share/texlive/texmf-dist/tex/latex/graphics-cfg/color.cfg)
(/usr/share/texlive/texmf-dist/tex/latex/graphics-def/xetex.def)
(/usr/share/texlive/texmf-dist/tex/latex/graphics/mathcolor.ltx))
(/usr/share/texlive/texmf-dist/tex/latex/graphics/dvipsnam.def)
(/usr/share/texlive/texmf-dist/tex/latex/xcolor/svgnam.def)
(/usr/share/texlive/texmf-dist/tex/latex/xcolor/x11nam.def)
(/usr/share/texlive/texmf-dist/tex/generic/soul/soul.sty
(/usr/share/texlive/texmf-dist/tex/generic/soul/soul-ori.sty)
(/usr/share/texlive/texmf-dist/tex/generic/infwarerr/infwarerr.sty)
(/usr/share/texlive/texmf-dist/tex/generic/etexcmds/etexcmds.sty))
(/usr/share/texlive/texmf-dist/tex/latex/bookmark/bookmark.sty
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/hyperref.sty
(/usr/share/texlive/texmf-dist/tex/generic/kvdefinekeys/kvdefinekeys.sty)
(/usr/share/texlive/texmf-dist/tex/generic/pdfescape/pdfescape.sty
(/usr/share/texlive/texmf-dist/tex/generic/pdftexcmds/pdftexcmds.sty))
(/usr/share/texlive/texmf-dist/tex/latex/hycolor/hycolor.sty)
(/usr/share/texlive/texmf-dist/tex/latex/auxhook/auxhook.sty)
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/nameref.sty
(/usr/share/texlive/texmf-dist/tex/latex/refcount/refcount.sty)
(/usr/share/texlive/texmf-dist/tex/generic/gettitlestring/gettitlestring.sty))
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/pd1enc.def)
(/usr/share/texlive/texmf-dist/tex/generic/intcalc/intcalc.sty)
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/puenc.def)
(/usr/share/texlive/texmf-dist/tex/latex/url/url.sty)
(/usr/share/texlive/texmf-dist/tex/generic/bitset/bitset.sty
(/usr/share/texlive/texmf-dist/tex/generic/bigintcalc/bigintcalc.sty))
(/usr/share/texlive/texmf-dist/tex/latex/base/atbegshi-ltx.sty))
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/hxetex.def
(/usr/share/texlive/texmf-dist/tex/generic/stringenc/stringenc.sty)
(/usr/share/texlive/texmf-dist/tex/latex/rerunfilecheck/rerunfilecheck.sty
(/usr/share/texlive/texmf-dist/tex/latex/base/atveryend-ltx.sty)
(/usr/share/texlive/texmf-dist/tex/generic/uniquecounter/uniquecounter.sty)))
(/usr/share/texlive/texmf-dist/tex/latex/bookmark/bkm-dvipdfm.def))
(/usr/share/texlive/texmf-dist/tex/latex/xurl/xurl.sty)
No file input.aux.
(/usr/share/texlive/texmf-dist/tex/latex/microtype/mt-LatinModernRoman.cfg)
Package hyperref Warning: Rerun to get /PageLabels entry.
! Cannot use \XeTeXglyph with ectt1000; not a native platform font.
\remove@tlig #1->\XeTeXglyph
\numexpr \XeTeXcharglyph #1\relax
l.73 \ul{\textquotesingle tis well}
No pages of output.
Transcript written on /tmp/tex2pdf.-7cab912f892c775c/input.log.
[makePDF] Run #2
This is XeTeX, Version 3.141592653-2.6-0.999995 (TeX Live 2023/Debian) (preloaded format=xelatex)
restricted \write18 enabled.
entering extended mode
(/tmp/tex2pdf.-7cab912f892c775c/input.tex
LaTeX2e <2023-11-01> patch level 1
L3 programming layer <2024-01-22>
(/usr/share/texlive/texmf-dist/tex/latex/base/article.cls
Document Class: article 2023/05/17 v1.4n Standard LaTeX document class
(/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo))
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsmath.sty
For additional information on amsmath, use the `?' option.
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amstext.sty
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsgen.sty))
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsbsy.sty)
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsopn.sty))
(/usr/share/texlive/texmf-dist/tex/latex/amsfonts/amssymb.sty
(/usr/share/texlive/texmf-dist/tex/latex/amsfonts/amsfonts.sty))
(/usr/share/texlive/texmf-dist/tex/generic/iftex/iftex.sty)
(/usr/share/texlive/texmf-dist/tex/latex/unicode-math/unicode-math.sty
(/usr/share/texlive/texmf-dist/tex/latex/l3kernel/expl3.sty
(/usr/share/texlive/texmf-dist/tex/latex/l3backend/l3backend-xetex.def))
(/usr/share/texlive/texmf-dist/tex/latex/unicode-math/unicode-math-xetex.sty
(/usr/share/texlive/texmf-dist/tex/latex/l3packages/xparse/xparse.sty)
(/usr/share/texlive/texmf-dist/tex/latex/l3packages/l3keys2e/l3keys2e.sty)
(/usr/share/texlive/texmf-dist/tex/latex/fontspec/fontspec.sty
(/usr/share/texlive/texmf-dist/tex/latex/fontspec/fontspec-xetex.sty
(/usr/share/texlive/texmf-dist/tex/latex/base/fontenc.sty)
(/usr/share/texlive/texmf-dist/tex/latex/fontspec/fontspec.cfg)))
(/usr/share/texlive/texmf-dist/tex/latex/base/fix-cm.sty
(/usr/share/texlive/texmf-dist/tex/latex/base/ts1enc.def))
(/usr/share/texlive/texmf-dist/tex/latex/unicode-math/unicode-math-table.tex)))
(/usr/share/texmf/tex/latex/lm/lmodern.sty)
(/usr/share/texlive/texmf-dist/tex/latex/upquote/upquote.sty
(/usr/share/texlive/texmf-dist/tex/latex/base/textcomp.sty))
(/usr/share/texlive/texmf-dist/tex/latex/microtype/microtype.sty
(/usr/share/texlive/texmf-dist/tex/latex/graphics/keyval.sty)
(/usr/share/texlive/texmf-dist/tex/latex/etoolbox/etoolbox.sty)
(/usr/share/texlive/texmf-dist/tex/latex/microtype/microtype-xetex.def)
(/usr/share/texlive/texmf-dist/tex/latex/microtype/microtype.cfg))
(/usr/share/texlive/texmf-dist/tex/latex/parskip/parskip.sty
(/usr/share/texlive/texmf-dist/tex/latex/kvoptions/kvoptions.sty
(/usr/share/texlive/texmf-dist/tex/generic/ltxcmds/ltxcmds.sty)
(/usr/share/texlive/texmf-dist/tex/latex/kvsetkeys/kvsetkeys.sty)))
(/usr/share/texlive/texmf-dist/tex/latex/xcolor/xcolor.sty
(/usr/share/texlive/texmf-dist/tex/latex/graphics-cfg/color.cfg)
(/usr/share/texlive/texmf-dist/tex/latex/graphics-def/xetex.def)
(/usr/share/texlive/texmf-dist/tex/latex/graphics/mathcolor.ltx))
(/usr/share/texlive/texmf-dist/tex/latex/graphics/dvipsnam.def)
(/usr/share/texlive/texmf-dist/tex/latex/xcolor/svgnam.def)
(/usr/share/texlive/texmf-dist/tex/latex/xcolor/x11nam.def)
(/usr/share/texlive/texmf-dist/tex/generic/soul/soul.sty
(/usr/share/texlive/texmf-dist/tex/generic/soul/soul-ori.sty)
(/usr/share/texlive/texmf-dist/tex/generic/infwarerr/infwarerr.sty)
(/usr/share/texlive/texmf-dist/tex/generic/etexcmds/etexcmds.sty))
(/usr/share/texlive/texmf-dist/tex/latex/bookmark/bookmark.sty
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/hyperref.sty
(/usr/share/texlive/texmf-dist/tex/generic/kvdefinekeys/kvdefinekeys.sty)
(/usr/share/texlive/texmf-dist/tex/generic/pdfescape/pdfescape.sty
(/usr/share/texlive/texmf-dist/tex/generic/pdftexcmds/pdftexcmds.sty))
(/usr/share/texlive/texmf-dist/tex/latex/hycolor/hycolor.sty)
(/usr/share/texlive/texmf-dist/tex/latex/auxhook/auxhook.sty)
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/nameref.sty
(/usr/share/texlive/texmf-dist/tex/latex/refcount/refcount.sty)
(/usr/share/texlive/texmf-dist/tex/generic/gettitlestring/gettitlestring.sty))
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/pd1enc.def)
(/usr/share/texlive/texmf-dist/tex/generic/intcalc/intcalc.sty)
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/puenc.def)
(/usr/share/texlive/texmf-dist/tex/latex/url/url.sty)
(/usr/share/texlive/texmf-dist/tex/generic/bitset/bitset.sty
(/usr/share/texlive/texmf-dist/tex/generic/bigintcalc/bigintcalc.sty))
(/usr/share/texlive/texmf-dist/tex/latex/base/atbegshi-ltx.sty))
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/hxetex.def
(/usr/share/texlive/texmf-dist/tex/generic/stringenc/stringenc.sty)
(/usr/share/texlive/texmf-dist/tex/latex/rerunfilecheck/rerunfilecheck.sty
(/usr/share/texlive/texmf-dist/tex/latex/base/atveryend-ltx.sty)
(/usr/share/texlive/texmf-dist/tex/generic/uniquecounter/uniquecounter.sty)))
(/usr/share/texlive/texmf-dist/tex/latex/bookmark/bkm-dvipdfm.def))
(/usr/share/texlive/texmf-dist/tex/latex/xurl/xurl.sty)
(/tmp/tex2pdf.-7cab912f892c775c/input.aux)
(/usr/share/texlive/texmf-dist/tex/latex/microtype/mt-LatinModernRoman.cfg)
Package hyperref Warning: Rerun to get /PageLabels entry.
! Cannot use \XeTeXglyph with ectt1000; not a native platform font.
\remove@tlig #1->\XeTeXglyph
\numexpr \XeTeXcharglyph #1\relax
l.73 \ul{\textquotesingle tis well}
No pages of output.
Transcript written on /tmp/tex2pdf.-7cab912f892c775c/input.log.
Error producing PDF.
! Cannot use \XeTeXglyph with ectt1000; not a native platform font.
\remove@tlig #1->\XeTeXglyph
\numexpr \XeTeXcharglyph #1\relax
l.73 \ul{\textquotesingle tis well}
CA State Library is experiencing errors exporting a facing page PDF. These only seem to happen on production, with the following transcription:
Bulk Export
I have reproduced this on production, in this bulk export:
Pandoc Command
Here is the command-line replication of this problem: