DocTeX syntax highlighting breaks on environments

[BizarrePenguin]

Pre-checks*

• [x] The issue has not been reported in this repository.
• [x] The issue remains after disabling all other extensions and restarting Visual Studio Code.
• [x] The FAQ cannot address the issue.
• [x] The issue is not related to compiling a document, or the document can be successfully compiled in the OS terminal but not in Visual Studio Code with this extension.

Environment*

• Operating System: macOS Sonoma 14.1.1 (23B81)
• Visual Studio Code Version: 1.90.0
• LaTeX Workshop Version: 9.20.1
• TeX Distribution Version: MiKTeX 22.1

The Issue*

The DocTeX syntax highlighting stops working when starting an environment in the documentation part (the part that starts every line with percent signs), i.e.

% \foo
% \begin{bar}
% \foo
% \end{bar}
% \foo

displays as

Reproduction Steps

1. Create a .dtx file in vscode with the latex-workshop extension enabled
2. Make sure DocTeX is selected as the language for syntax highlighting
3. Write any LaTeX code with percent signs at the start of the lines (should be highlighted as expected)
4. Start an environment with a percent sign at the start of the line
5. Write any LaTeX code with percent signs at the start of the lines

Expected Behavior

LaTeX code inside the environment and after ending the environment should be highlighted as expected with .dtx files.

Logs

LaTeX Workshop Output*

[12:00:07.815][Extension] Initializing LaTeX Workshop.
[12:00:07.921][Build][Recipe] Set $LATEXWORKSHOP_DOCKER_LATEX: ""
[12:00:07.945][Server] Creating LaTeX Workshop http and websocket server.
[12:00:07.991][Format][Bib] Bibtex format config: {"tab":"  ","case":"lowercase","left":"{","right":"}","trailingComma":false,"sort":["key"],"alignOnEqual":true,"sortFields":false,"fieldsOrder":[],"firstEntries":["string","xdata"]}
[12:00:08.017][Server] Server successfully started: {"address":"127.0.0.1","family":"IPv4","port":62291} .
[12:00:08.021][Extension] Extension root: /Users/$USERNAME/.vscode/extensions/james-yu.latex-workshop-9.20.1
[12:00:08.021][Extension] $PATH: /opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/usr/local/MacGPG2/bin:/Users/$USERNAME/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/$USERNAME/opt/anaconda3/bin:/Users/$USERNAME/opt/anaconda3/condabin:/Users/$USERNAME/Documents/dev/flutter/bin
[12:00:08.021][Extension] $SHELL: /bin/zsh
[12:00:08.021][Extension] $LANG: undefined
[12:00:08.021][Extension] $LC_ALL: undefined
[12:00:08.021][Extension] process.platform: darwin
[12:00:08.022][Extension] process.arch: arm64
[12:00:08.022][Extension] vscode.env.appName: Visual Studio Code
[12:00:08.022][Extension] vscode.env.remoteName: undefined
[12:00:08.022][Extension] vscode.env.uiKind: 1
[12:00:08.022][Config] latex-workshop.latex.recipes: [{"name":"pdflatex 🔃","tools":["pdflatex"]},{"name":"pdflatex (x2)","tools":["pdflatex","pdflatex"]},{"name":"lualatex","tools":["lualatex"]},{"name":"latexmk","tools":["latexmk"]},{"name":"latexmk (latexmkrc)","tools":["latexmk_rconly"]},{"name":"latexmk (lualatex)","tools":["lualatexmk"]},{"name":"latexmk (xelatex)","tools":["xelatexmk"]},{"name":"pdflatex ➞ bibtex ➞ pdflatex × 2","tools":["pdflatex","bibtex","pdflatex","pdflatex"]},{"name":"Compile Rnw files","tools":["rnw2tex","latexmk"]},{"name":"Compile Jnw files","tools":["jnw2tex","latexmk"]},{"name":"tectonic","tools":["tectonic"]}] .
[12:00:08.023][Config] latex-workshop.latex.recipe.default: "lastUsed" .
[12:00:08.023][Config] latex-workshop.latex.tools: [{"name":"lualatex","command":"lualatex","args":["-synctex=1","-interaction=nonstopmode","-file-line-error","-output-format=pdf","-output-directory=%OUTDIR%","%DOC%"],"env":{}},{"name":"latexmk","command":"latexmk","args":["-synctex=1","-interaction=nonstopmode","-file-line-error","-shell-escape","-pdf","-outdir=%OUTDIR%","%DOC%"],"env":{}},{"name":"lualatexmk","command":"latexmk","args":["-synctex=1","-interaction=nonstopmode","-file-line-error","-lualatex","-outdir=%OUTDIR%","%DOC%"],"env":{}},{"name":"xelatexmk","command":"latexmk","args":["-synctex=1","-interaction=nonstopmode","-file-line-error","-xelatex","-outdir=%OUTDIR%","%DOC%"],"env":{}},{"name":"latexmk_rconly","command":"latexmk","args":["%DOC%"],"env":{}},{"name":"pdflatex","command":"pdflatex","args":["-synctex=1","-interaction=nonstopmode","-file-line-error","-shell-escape","%DOC%"],"env":{}},{"name":"bibtex","command":"bibtex","args":["%DOCFILE%"],"env":{}},{"name":"rnw2tex","command":"Rscript","args":["-e","knitr::opts_knit$set(concordance = TRUE); knitr::knit('%DOCFILE_EXT%')"],"env":{}},{"name":"jnw2tex","command":"julia","args":["-e","using Weave; weave(\"%DOC_EXT%\", doctype=\"tex\")"],"env":{}},{"name":"jnw2texminted","command":"julia","args":["-e","using Weave; weave(\"%DOC_EXT%\", doctype=\"texminted\")"],"env":{}},{"name":"pnw2tex","command":"pweave","args":["-f","tex","%DOC_EXT%"],"env":{}},{"name":"pnw2texminted","command":"pweave","args":["-f","texminted","%DOC_EXT%"],"env":{}},{"name":"tectonic","command":"tectonic","args":["--synctex","--keep-logs","%DOC%.tex"],"env":{}}] .
[12:00:08.024][Config] latex-workshop.latex.autoBuild.run: "never" .
[12:00:08.040][Extension] Trigger characters for intellisense of LaTeX documents: ["\\",",","{"]
[12:00:08.041][Root] Current workspace folders: undefined
[12:00:08.041][Extension] LaTeX Workshop initialized.
[12:00:08.041][Root] Current workspaceRootDir:  .
[12:00:08.042][Root] No root file found.
[12:00:08.042][Event] ROOT_FILE_SEARCHED
[12:00:08.045][Server] valdOrigin is http://127.0.0.1:62291
[12:00:08.206][Structure] Structure updated with 0 entries for /Users/$USERNAME/Desktop/test.dtx .
[12:00:08.206][Event] STRUCTURE_UPDATED
[12:00:15.346][Structure] Structure updated with 0 entries for /Users/$USERNAME/Desktop/test.dtx .
[12:00:15.347][Event] STRUCTURE_UPDATED

Note, that for privacy reasons I removed some paths from the log of the $PATH variable that have nothing to do with vscode or LaTeX and also changed my computers username to $USERNAME in the log output.

Developer Tools Console

Please paste the whole log messages below, not parts of ones. This console logs can sometimes be very important in many cases. To access the log, click Help -> Toggle Developer Tools -> Console.

log.ts:419  INFO Initial editor state Object
log.ts:419  INFO Started local extension host with pid 83211.
log.ts:439   ERR Ignoring latex-workshop.synctex.indicator.enabled as latex-workshop.synctex.indicator is "rectangle"
TMScopeRegistry.ts:46 Overwriting grammar scope name to file mapping for scope text.tex.
Old grammar file: file:///Applications/Visual%20Studio%20Code.app/Contents/Resources/app/extensions/latex/syntaxes/TeX.tmLanguage.json.
New grammar file: file:///Users/$USERNAME/.vscode/extensions/james-yu.latex-workshop-9.20.1/syntax/TeX.tmLanguage.json
register @ TMScopeRegistry.ts:46
TMScopeRegistry.ts:46 Overwriting grammar scope name to file mapping for scope text.tex.latex.
Old grammar file: file:///Applications/Visual%20Studio%20Code.app/Contents/Resources/app/extensions/latex/syntaxes/LaTeX.tmLanguage.json.
New grammar file: file:///Users/$USERNAME/.vscode/extensions/james-yu.latex-workshop-9.20.1/syntax/LaTeX.tmLanguage.json
register @ TMScopeRegistry.ts:46
TMScopeRegistry.ts:46 Overwriting grammar scope name to file mapping for scope text.bibtex.
Old grammar file: file:///Applications/Visual%20Studio%20Code.app/Contents/Resources/app/extensions/latex/syntaxes/Bibtex.tmLanguage.json.
New grammar file: file:///Users/$USERNAME/.vscode/extensions/james-yu.latex-workshop-9.20.1/syntax/Bibtex.tmLanguage.json
register @ TMScopeRegistry.ts:46
TMScopeRegistry.ts:46 Overwriting grammar scope name to file mapping for scope text.tex.markdown_latex_combined.
Old grammar file: file:///Applications/Visual%20Studio%20Code.app/Contents/Resources/app/extensions/latex/syntaxes/markdown-latex-combined.tmLanguage.json.
New grammar file: file:///Users/$USERNAME/.vscode/extensions/james-yu.latex-workshop-9.20.1/syntax/markdown-latex-combined.tmLanguage.json
register @ TMScopeRegistry.ts:46
TMScopeRegistry.ts:46 Overwriting grammar scope name to file mapping for scope source.cpp.embedded.latex.
Old grammar file: file:///Applications/Visual%20Studio%20Code.app/Contents/Resources/app/extensions/latex/syntaxes/cpp-grammar-bailout.tmLanguage.json.
New grammar file: file:///Users/$USERNAME/.vscode/extensions/james-yu.latex-workshop-9.20.1/syntax/cpp-grammar-bailout.tmLanguage.json
register @ TMScopeRegistry.ts:46
TMScopeRegistry.ts:46 Overwriting grammar scope name to file mapping for scope text.tex.
Old grammar file: file:///Applications/Visual%20Studio%20Code.app/Contents/Resources/app/extensions/latex/syntaxes/TeX.tmLanguage.json.
New grammar file: file:///Users/$USERNAME/.vscode/extensions/james-yu.latex-workshop-9.20.1/syntax/TeX.tmLanguage.json
register @ TMScopeRegistry.ts:46
TMScopeRegistry.ts:46 Overwriting grammar scope name to file mapping for scope text.tex.latex.
Old grammar file: file:///Applications/Visual%20Studio%20Code.app/Contents/Resources/app/extensions/latex/syntaxes/LaTeX.tmLanguage.json.
New grammar file: file:///Users/$USERNAME/.vscode/extensions/james-yu.latex-workshop-9.20.1/syntax/LaTeX.tmLanguage.json
register @ TMScopeRegistry.ts:46
TMScopeRegistry.ts:46 Overwriting grammar scope name to file mapping for scope text.bibtex.
Old grammar file: file:///Applications/Visual%20Studio%20Code.app/Contents/Resources/app/extensions/latex/syntaxes/Bibtex.tmLanguage.json.
New grammar file: file:///Users/$USERNAME/.vscode/extensions/james-yu.latex-workshop-9.20.1/syntax/Bibtex.tmLanguage.json
register @ TMScopeRegistry.ts:46
TMScopeRegistry.ts:46 Overwriting grammar scope name to file mapping for scope text.tex.markdown_latex_combined.
Old grammar file: file:///Applications/Visual%20Studio%20Code.app/Contents/Resources/app/extensions/latex/syntaxes/markdown-latex-combined.tmLanguage.json.
New grammar file: file:///Users/$USERNAME/.vscode/extensions/james-yu.latex-workshop-9.20.1/syntax/markdown-latex-combined.tmLanguage.json
register @ TMScopeRegistry.ts:46
TMScopeRegistry.ts:46 Overwriting grammar scope name to file mapping for scope source.cpp.embedded.latex.
Old grammar file: file:///Applications/Visual%20Studio%20Code.app/Contents/Resources/app/extensions/latex/syntaxes/cpp-grammar-bailout.tmLanguage.json.
New grammar file: file:///Users/$USERNAME/.vscode/extensions/james-yu.latex-workshop-9.20.1/syntax/cpp-grammar-bailout.tmLanguage.json
register @ TMScopeRegistry.ts:46
log.ts:419  INFO [perf] Render performance baseline is 15ms
log.ts:429  WARN [remote-ssh]: Couldn't find message for key setting.usseExecServer.description.
log.ts:429  WARN [perf] Renderer reported VERY LONG TASK (169ms), starting profiling session '7650c87f-eead-49ed-af82-f605fa72bdde'
nls.ts:319 The factory function of "vs/base/browser/window" has thrown an exception
v @ nls.ts:319
nls.ts:319 ReferenceError: window is not defined
at eval (window.ts:21:27)
at t._safeInvokeFunction (nls.ts:319:2)
at t._invokeFactory (nls.ts:319:2)
at t.complete (nls.ts:319:2)
at v._onModuleComplete (nls.ts:319:2)
at v._resolve (nls.ts:319:2)
at v.defineModule (nls.ts:319:2)
at n (nls.ts:319:2)
at eval (fake:9:1)
at anonymous (fake:1:1)
v @ nls.ts:319
nls.ts:319 Here are the modules that depend on it:
v @ nls.ts:319
nls.ts:319 Array(0)
v @ nls.ts:319

Note, that for privacy reasons I changed my computers username to $USERNAME in the log output.

Anything Else?

• Interestingly this is not an issue when using the macrocode environment with the correct amount of spaces

  % \foo
  %    \begin{macrocode}
  % \foo
  \foo
  %    \end{macrocode}
  % \foo

  as here it is desirable that inside the environment the percent sign actually acts as a comment indicator and in contrast to other environments the highlighting goes back to normal after \end{macrocode}:

• It is also different with the verbatim environment:

  % \foo
  % \begin{verbatim}
  % \foo
  % \end{verbatim}
  % \foo

where it highlights as expected after \end{verbatim} while printing everything white in between:

[James-Yu]

Thank you for filing this issue. We have outsourced our LaTeX syntax highlighting grammar to vscode-latex-basics, could you please post your issue there?

[jlelong]

I cannot reproduce the issue

[BizarrePenguin]

Thank you, I did not notice the syntax highlighting was outsourced, sorry.

With the development version of vscode-latex-basics I don't have the issue either. So I guess I will just wait and see if it is resolved automatically in future releases of the LaTeX Workshop extension.