search

MiKTeX / miktex

the MiKTeX source code
https://miktex.org
Other
791 stars 94 forks source link

xetex: unrestricted command has trailing null byte #1447

Closed DE3000 closed 7 months ago

DE3000 commented 8 months ago

Possibly related to https://github.com/MiKTeX/miktex/issues/532 Poorly worded issue https://github.com/MiKTeX/miktex/issues/1444 maybe related as well

miktex-report.txt ``` ReportDate: 2024-02-16 20:49:46 CurrentVersion: 24.1 SetupDate: 2020-02-26 22:56:00 SetupVersion: 2.9 Configuration: Portable GitInfo: 7b52272 / 2024-01-27 12:26:15 OS: Windows 10.0.19045 SharedSetup: no LinkTargetDirectory: E:\miktex-portable\texmfs\install\miktex\bin\x64 PathOkay: yes LastUpdateCheck: 2024-02-16 16:54:13 LastUpdate: 2024-02-16 16:54:50 LastUpdateDb: 2024-02-16 16:54:01 SystemAdmin: yes RootPrivileges: no AdminMode: no Root0: E:\miktex-portable\texmfs\config Root1: E:\miktex-portable\texmfs\data Root2: E:\miktex-portable\texmfs\install Root3: C:\Dropbox\Environments\localtexmf UserInstall: E:\miktex-portable\texmfs\install UserConfig: E:\miktex-portable\texmfs\config UserData: E:\miktex-portable\texmfs\data ```

I am using the minted package for displaying formatted code in a document. Prior to updating I was on MiKTeX 23.5 (full log below) and everything ran fine. However, after updating to MiKTeX 24.1 there's an issue with minted calling an external program pygmentize by passing a few parameters.

I have been discussion the issue on minted package repository (https://github.com/gpoore/minted/issues/386) where we tried to isolate the issue to either the minted package or is it a more general problem.

I have isolated the issue to running commands with \ShellEscape where the command has quoted options.

The test file shelltest.tex is as follows:

\documentclass{article}
\usepackage{shellesc}

\begin{document}
\ShellEscape{mkdir "quoted"}
\ShellEscape{copy nul "quoted/test.txt"}
\ShellEscape{robocopy "quoted" unquoted}
\end{document}

Log output of running xelatex --version and xelatex -shell-escape --interaction=nonstopmode shelltest.tex

With MiKTeX 23.5

```log MiKTeX-XeTeX 4.10 (MiKTeX 23.5 Portable) © 1994-2008 SIL International, © 2009-2023 Jonathan Kew, © 2010-2012 Hàn Thế Thành, © 2012-2013 Khaled Hosny TeX is a trademark of the American Mathematical Society using bzip2 version 1.0.8, 13-Jul-2019 compiled with curl version 7.87.0; using libcurl/7.87.0 Schannel compiled with expat version 2.5; using expat_2.5.0 compiled with fontconfig version 2.13.1; using 2.13.1 compiled with freetype2 version 2.12.1; using 2.12.1 compiled with graphite2 version 1.3.14; using 1.3.14 compiled with harfbuzz version 6.0.0; using 6.0.0 compiled with icu version 72.1; using 72.1 compiled with jpeg version 9.5 compiled with liblzma version 50040002; using 50040002 compiled with libpng version 1.6.39; using 1.6.39 compiled with libressl version LibreSSL 3.1.4; using LibreSSL 3.1.4 compiled with MiKTeX Application Framework version 4.7; using 4.7 compiled with MiKTeX Core version 4.20; using 4.20 compiled with MiKTeX Archive Extractor version 4.0; using 4.0 compiled with MiKTeX Package Manager version 4.9; using 4.9 compiled with pplib version v2.05 less toxic i hope using teckit version 2.4 compiled with uriparser version 0.9.7 compiled with zlib version 1.2.13; using 1.2.13 This is XeTeX, Version 3.141592653-2.6-0.999995 (MiKTeX 23.5 Portable) (preloaded format=xelatex.fmt) \write18 enabled. entering extended mode (shelltest.tex LaTeX2e <2023-06-01> patch level 1 L3 programming layer <2023-08-11> (E:\miktex-portable\texmfs\install\tex/latex/base\article.cls Document Class: article 2023/05/17 v1.4n Standard LaTeX document class (E:\miktex-portable\texmfs\install\tex/latex/base\size10.clo)) (E:\miktex-portable\texmfs\install\tex/latex/tools\shellesc.sty) (E:\miktex-portable\texmfs\install\tex/latex/l3backend\l3backend-xetex.def) (shelltest.aux) (E:\miktex-portable\texmfs\install\tex/latex/base\ts1cmr.fd) 1 file(s) copied. ------------------------------------------------------------------------------- ROBOCOPY :: Robust File Copy for Windows ------------------------------------------------------------------------------- Started : Monday, February 12, 2024 22:13:05 Source : C:\temp\latex-testing\quoted\ Dest : C:\temp\latex-testing\unquoted\ Files : *.* Options : *.* /DCOPY:DA /COPY:DAT /R:1000000 /W:30 ------------------------------------------------------------------------------ 1 C:\temp\latex-testing\quoted\ Newer 0 test.txt 100% ------------------------------------------------------------------------------ Total Copied Skipped Mismatch FAILED Extras Dirs : 1 0 1 0 0 0 Files : 1 1 0 0 0 0 Bytes : 0 0 0 0 0 0 Times : 0:00:00 0:00:00 0:00:00 0:00:00 Ended : Monday, February 12, 2024 22:13:05 (shelltest.aux) ) No pages of output. Transcript written on shelltest.log. ```

With MiKTeX 24.1 ```log MiKTeX-XeTeX 4.10 (MiKTeX 24.1 Portable) © 1994-2008 SIL International, © 2009-2023 Jonathan Kew, © 2010-2012 Hàn Thế Thành, © 2012-2013 Khaled Hosny TeX is a trademark of the American Mathematical Society using bzip2 version 1.0.8, 13-Jul-2019 compiled with curl version 8.4.0; using libcurl/8.4.0 Schannel compiled with expat version 2.5; using expat_2.5.0 compiled with fontconfig version 2.13.1; using 2.13.1 compiled with freetype2 version 2.12.1; using 2.12.1 compiled with graphite2 version 1.3.14; using 1.3.14 compiled with harfbuzz version 8.2.1; using 8.2.1 compiled with icu version 74.2; using 74.2 compiled with jpeg version 9.5 compiled with liblzma version 50040002; using 50040002 compiled with libpng version 1.6.39; using 1.6.39 compiled with libressl version LibreSSL 3.8.1; using LibreSSL 3.8.1 compiled with MiKTeX Application Framework version 4.8; using 4.8 compiled with MiKTeX Core version 4.23; using 4.23 compiled with MiKTeX Archive Extractor version 4.1; using 4.1 compiled with MiKTeX Package Manager version 4.10; using 4.10 compiled with pplib version v2.05 less toxic i hope using teckit version 2.4 compiled with uriparser version 0.9.7 compiled with zlib version 1.2.13; using 1.2.13 This is XeTeX, Version 3.141592653-2.6-0.999995 (MiKTeX 24.1 Portable) (preloaded format=xelatex.fmt) \write18 enabled. entering extended mode (shelltest.tex LaTeX2e <2023-11-01> patch level 1 L3 programming layer <2024-01-22> (E:\miktex-portable\texmfs\install\tex/latex/base\article.cls Document Class: article 2023/05/17 v1.4n Standard LaTeX document class (E:\miktex-portable\texmfs\install\tex/latex/base\size10.clo)) (E:\miktex-portable\texmfs\install\tex/latex/tools\shellesc.sty) (E:\miktex-portable\texmfs\install\tex/latex/l3backend\l3backend-xetex.def) (shelltest.aux) (E:\miktex-portable\texmfs\install\tex/latex/base\ts1cmr.fd) 1 file(s) copied. ------------------------------------------------------------------------------- ROBOCOPY :: Robust File Copy for Windows ------------------------------------------------------------------------------- Started : Monday, February 12, 2024 22:59:11 Source : C:\Dropbox\Eclipse workspace\latex-testing\quoted unquoted\ Dest - Files : *.* Options : *.* /DCOPY:DA /COPY:DAT /R:1000000 /W:30 ------------------------------------------------------------------------------ ERROR : No Destination Directory Specified. Simple Usage :: ROBOCOPY source destination /MIR source :: Source Directory (drive:\path or \\server\share\path). destination :: Destination Dir (drive:\path or \\server\share\path). /MIR :: Mirror a complete directory tree. For more usage information run ROBOCOPY /? **** /MIR can DELETE files as well as copy them ! (shelltest.aux) ) No pages of output. Transcript written on shelltest.log. ```

Robocopy outputs the options passed to the command so we can see the difference between the two runs.

MiKTex 23.5 robocopy

Source : C:\temp\latex-testing\quoted\
Dest : C:\temp\latex-testing\unquoted\

MiKTex 24.1 robocopy

Source : C:\temp\latex-testing\quoted unquoted\
Dest -

With MiKTeX 24.1 the quoted option gets lumped together with the next option.

Interesting to note that if the last option is quoted then it works fine, as \ShellEscape{copy nul "quoted/test.txt"} works in both cases.

When working with the original minted tex file, the issue of incorrect options being passed to the pygmentize program as it has some options that are quoted.

edocevoli commented 8 months ago

Please attach the log file(s) so we can see what's executed. You will find log files in the ...\miktex\log directory. (replace ... with the data root directory)

DE3000 commented 8 months ago

MiKTex 23.5 log except

2024-02-19 15:59:16,167+0200 INFO  xelatex - this process (29176) started by cmd in directory C:\temp\latex-testing with command line: xelatex --version
2024-02-19 15:59:16,167+0200 INFO  xelatex - running on Windows 10.0.19045
2024-02-19 15:59:16,170+0200 INFO  xelatex - this is MiKTeX-XETEX 4.10.25 (0.999995) (MiKTeX 23.5 Portable)
2024-02-19 15:59:16,172+0200 INFO  xelatex - allowing known shell commands
2024-02-19 15:59:16,184+0200 INFO  xelatex - this process (29176) finishes with exit code 0
2024-02-19 15:59:16,186+0200 WARN  xelatex - minor issue: It has been a long time since you have checked for MiKTeX updates.
2024-02-19 15:59:16,612+0200 INFO  xelatex - this process (28628) started by cmd in directory C:\temp\latex-testing with command line: xelatex -shell-escape -interaction=nonstopmode shelltest.tex
2024-02-19 15:59:16,613+0200 INFO  xelatex - running on Windows 10.0.19045
2024-02-19 15:59:16,615+0200 INFO  xelatex - this is MiKTeX-XETEX 4.10.0 (0.999995) (MiKTeX 23.4 Portable)
2024-02-19 15:59:16,617+0200 INFO  xelatex - allowing known shell commands
2024-02-19 15:59:16,628+0200 INFO  xelatex - allowing all shell commands
2024-02-19 15:59:16,817+0200 WARN  xelatex - executing unrestricted write18 shell command: mkdir "quoted"
2024-02-19 15:59:16,818+0200 INFO  xelatex.core - start process: C:\WINDOWS\system32\cmd.exe /c "mkdir "quoted""
2024-02-19 15:59:16,903+0200 INFO  xelatex - write18 exit code: 1
2024-02-19 15:59:16,903+0200 WARN  xelatex - executing unrestricted write18 shell command: copy nul "quoted/test.txt"
2024-02-19 15:59:16,903+0200 INFO  xelatex.core - start process: C:\WINDOWS\system32\cmd.exe /c "copy nul "quoted/test.txt""
2024-02-19 15:59:16,940+0200 INFO  xelatex - write18 exit code: 0
2024-02-19 15:59:16,941+0200 WARN  xelatex - executing unrestricted write18 shell command: robocopy "quoted" unquoted
2024-02-19 15:59:16,941+0200 INFO  xelatex.core - start process: C:\WINDOWS\system32\cmd.exe /c "robocopy "quoted" unquoted"
2024-02-19 15:59:17,001+0200 INFO  xelatex - write18 exit code: 1
2024-02-19 15:59:17,007+0200 INFO  xelatex - this process (28628) finishes with exit code 0
2024-02-19 15:59:17,008+0200 WARN  xelatex - minor issue: It has been a long time since you have checked for MiKTeX updates.

MiKTex 24.1 log except

2024-02-19 15:52:10,711+0200 INFO  xelatex - this process (28844) started by cmd in directory C:\temp\latex-testing with command line: xelatex --version
2024-02-19 15:52:10,712+0200 INFO  xelatex - running on Windows 10.0.19045
2024-02-19 15:52:10,714+0200 INFO  xelatex - this is MiKTeX-XETEX 4.10.0 (0.999995) (MiKTeX 24.1 Portable)
2024-02-19 15:52:10,717+0200 INFO  xelatex - allowing known shell commands
2024-02-19 15:52:10,729+0200 INFO  xelatex - this process (28844) finishes with exit code 0
2024-02-19 15:52:11,097+0200 INFO  xelatex - this process (4992) started by cmd in directory C:\temp\latex-testing with command line: xelatex -shell-escape -interaction=nonstopmode shelltest.tex
2024-02-19 15:52:11,097+0200 INFO  xelatex - running on Windows 10.0.19045
2024-02-19 15:52:11,099+0200 INFO  xelatex - this is MiKTeX-XETEX 4.10.0 (0.999995) (MiKTeX 24.1 Portable)
2024-02-19 15:52:11,101+0200 INFO  xelatex - allowing known shell commands
2024-02-19 15:52:11,113+0200 INFO  xelatex - allowing all shell commands
2024-02-19 15:52:11,124+0200 INFO  xelatex - going to create file: xelatex.fmt
2024-02-19 15:52:11,124+0200 INFO  xelatex.core - start process: miktex --enable-installer formats build xelatex --engine xetex
2024-02-19 15:52:23,496+0200 WARN  xelatex - executing unrestricted write18 shell command: mkdir "quoted" 
2024-02-19 15:52:23,497+0200 INFO  xelatex.core - start process: C:\WINDOWS\system32\cmd.exe /c "mkdir "quoted" "
2024-02-19 15:52:23,597+0200 INFO  xelatex - write18 exit code: 1
2024-02-19 15:52:23,597+0200 WARN  xelatex - executing unrestricted write18 shell command: copy nul "quoted/test.txt" 
2024-02-19 15:52:23,597+0200 INFO  xelatex.core - start process: C:\WINDOWS\system32\cmd.exe /c "copy nul "quoted/test.txt" "
2024-02-19 15:52:23,638+0200 INFO  xelatex - write18 exit code: 0
2024-02-19 15:52:23,639+0200 WARN  xelatex - executing unrestricted write18 shell command: robocopy "quoted" unquoted 
2024-02-19 15:52:23,639+0200 INFO  xelatex.core - start process: C:\WINDOWS\system32\cmd.exe /c "robocopy "quoted" unquoted "
2024-02-19 15:52:23,700+0200 INFO  xelatex - write18 exit code: 16
2024-02-19 15:52:23,707+0200 INFO  xelatex - this process (4992) finishes with exit code 0

From what I can see since it's not clear just from the output above, MiKTex 24.1 always has an extra space and sometimes an extra quotation mark (") at the end of the command to run.

2024-02-19 15:59:16,941+0200 WARN  xelatex - executing unrestricted write18 shell command: robocopy "quoted" unquoted
2024-02-19 15:59:16,941+0200 INFO  xelatex.core - start process: C:\WINDOWS\system32\cmd.exe /c "robocopy "quoted" unquoted"

VS

2024-02-19 15:52:23,639+0200 WARN  xelatex - executing unrestricted write18 shell command: robocopy "quoted" unquoted 
2024-02-19 15:52:23,639+0200 INFO  xelatex.core - start process: C:\WINDOWS\system32\cmd.exe /c "robocopy "quoted" unquoted "
edocevoli commented 7 months ago

Seems that both runs use different package versions. For example: LaTeX2e <2023-06-01> patch level 1 vs. LaTeX2e <2023-11-01> patch level 1

At the moment I cannot reproduce the issue so closing the ticket for now.

DE3000 commented 7 months ago

I took the portable 23.5 directory that is working and copied to a new folder. The new folder was then updated via the MiKTex console to 24.1. Any version changes came from the update.

DE3000 commented 7 months ago

Fresh portable install of 24.1

xelatex.log

The extra character is not a space but a NUL character.

muzimuzhi commented 7 months ago

I have been discussion the issue on minted package repository (gpoore/minted#386) where we tried to isolate the issue to either the minted package or is it a more general problem.

I have isolated the issue to running commands with \ShellEscape where the command has quoted options.

According to feedback https://github.com/gpoore/minted/issues/386#issuecomment-1999132975 from @ohly87, author of original issue, this problem has been resolved in MiKTeX 24.3.

Thank you!