latex: commit 8e780a4c highlights all text within environments

[shuber2]

Describe the highlighting problem

Starting with commit 8e780a4c all text with environments (\begin{x} … \end{x}), including the document environment. Now everything (except preamble) is highlighted.

Example snippet that causes the problem

\documentclass[]{article}

\begin{document} hello world is highlighted \end{document}

Tree-sitter parsing result

class_include [0, 0] - [0, 25] options: brack_group_key_value [0, 14] - [0, 16] path: curly_group_path [0, 16] - [0, 25] path: path [0, 17] - [0, 24] generic_environment [2, 0] - [4, 14] begin: begin [2, 0] - [2, 16] name: curly_group_text [2, 6] - [2, 16] text: text [2, 7] - [2, 15] word: word [2, 7] - [2, 15] text [3, 0] - [3, 11] word: word [3, 0] - [3, 5] word: word [3, 6] - [3, 11] end: end [4, 0] - [4, 14] name: curly_group_text [4, 4] - [4, 14] text: text [4, 5] - [4, 13] word: word [4, 5] - [4, 13]

Example screenshot

colorscheme is gruvbox

Expected behavior

Having text in document environment (and may others, like beamer frames) in normal highlighting (i.e., no highlighting).

Output of :checkhealth nvim-treesitter

nvim-treesitter: require("nvim-treesitter.health").check()
========================================================================
## Installation
  - WARNING: `tree-sitter` executable not found (parser generator, only needed for :TSInstallFromGrammar, not required for :TSInstall)
  - OK: `node` found v14.17.6 (only needed for :TSInstallFromGrammar)
  - OK: `git` executable found.
  - OK: `cc` executable found. Selected from { vim.NIL, "cc", "gcc", "clang", "cl", "zig" }
    Version: cc (Gentoo 11.2.0 p1) 11.2.0
  - OK: Neovim was compiled with tree-sitter runtime ABI version 13 (required >=13). Parsers must be compatible with runtime ABI.

## Parser/Features H L F I J
  - typescript     ✓ ✓ ✓ ✓ ✓ 
  - c              ✓ ✓ ✓ ✓ ✓ 
  - scala          ✓ . ✓ . ✓ 
  - fish           ✓ ✓ ✓ ✓ ✓ 
  - yaml           ✓ ✓ ✓ ✓ ✓ 
  - eex            ✓ . . . ✓ 
  - llvm           ✓ . . . . 
  - hcl            ✓ . ✓ ✓ ✓ 
  - bibtex         ✓ . ✓ ✓ . 
  - kotlin         ✓ ✓ ✓ . ✓ 
  - nix            ✓ ✓ ✓ . ✓ 
  - css            ✓ . ✓ ✓ ✓ 
  - python         ✓ ✓ ✓ ✓ ✓ 
  - jsonc          ✓ ✓ ✓ ✓ ✓ 
  - ocaml_interface✓ ✓ ✓ . ✓ 
  - bash           ✓ ✓ ✓ . ✓ 
  - go             ✓ ✓ ✓ ✓ ✓ 
  - json           ✓ ✓ ✓ ✓ . 
  - erlang         . . . . . 
  - c_sharp        ✓ ✓ ✓ . ✓ 
  - supercollider  ✓ ✓ ✓ ✓ ✓ 
  - gowork         ✓ . . . ✓ 
  - sparql         ✓ ✓ ✓ ✓ ✓ 
  - ledger         ✓ . ✓ ✓ ✓ 
  - julia          ✓ ✓ ✓ ✓ ✓ 
  - tsx            ✓ ✓ ✓ ✓ ✓ 
  - pascal         ✓ ✓ ✓ ✓ ✓ 
  - vala           ✓ . . . . 
  - fusion         ✓ ✓ ✓ ✓ . 
  - surface        ✓ . ✓ ✓ ✓ 
  - norg           . . . . . 
  - gomod          ✓ . . . ✓ 
  - rasi           ✓ ✓ ✓ ✓ . 
  - ocaml          ✓ ✓ ✓ . ✓ 
  - make           ✓ . . . ✓ 
  - vim            ✓ ✓ . . ✓ 
  - clojure        ✓ ✓ ✓ . ✓ 
  - pug            ✓ . . . ✓ 
  - http           ✓ . . . ✓ 
  - html           ✓ ✓ ✓ ✓ ✓ 
  - hocon          ✓ . . . ✓ 
  - hjson          ✓ ✓ ✓ ✓ ✓ 
  - dart           ✓ ✓ . ✓ ✓ 
  - pioasm         ✓ . . . ✓ 
  - json5          ✓ . . . ✓ 
  - yang           ✓ . ✓ . . 
  - regex          ✓ . . . . 
  - zig            ✓ . ✓ ✓ ✓ 
  - comment        ✓ . . . . 
  - graphql        ✓ . . ✓ ✓ 
  - cuda           ✓ ✓ ✓ ✓ ✓ 
  - glsl           ✓ ✓ ✓ ✓ ✓ 
  - javascript     ✓ ✓ ✓ ✓ ✓ 
  - toml           ✓ ✓ ✓ ✓ ✓ 
  - glimmer        ✓ . . . . 
  - r              ✓ ✓ . ✓ . 
  - heex           ✓ ✓ ✓ ✓ ✓ 
  - svelte         ✓ . ✓ ✓ ✓ 
  - fennel         ✓ ✓ . . ✓ 
  - gdscript       ✓ ✓ . ✓ ✓ 
  - dockerfile     ✓ . . . ✓ 
  - jsdoc          ✓ . . . . 
  - commonlisp     ✓ ✓ ✓ . . 
  - elixir         ✓ ✓ ✓ ✓ ✓ 
  - latex          ✓ . ✓ . ✓ 
  - query          ✓ ✓ ✓ ✓ ✓ 
  - java           ✓ ✓ . ✓ ✓ 
  - cpp            ✓ ✓ ✓ ✓ ✓ 
  - ql             ✓ ✓ . ✓ ✓ 
  - scss           ✓ . . ✓ . 
  - lua            ✓ ✓ ✓ ✓ ✓ 
  - perl           ✓ . . . . 
  - php            ✓ ✓ ✓ ✓ ✓ 
  - rust           ✓ ✓ ✓ ✓ ✓ 
  - rst            ✓ ✓ . . ✓ 
  - dot            ✓ . . . ✓ 
  - turtle         ✓ ✓ ✓ ✓ ✓ 
  - ninja          ✓ . ✓ ✓ . 
  - cmake          ✓ . ✓ . . 
  - beancount      ✓ . ✓ . . 
  - prisma         ✓ . . . . 
  - tlaplus        ✓ ✓ ✓ . ✓ 
  - vue            ✓ . ✓ ✓ ✓ 
  - ruby           ✓ ✓ ✓ ✓ ✓ 

  Legend: H[ighlight], L[ocals], F[olds], I[ndents], In[j]ections
         +) multiple parsers found, only one will be used
         x) errors found in the query, try to run :TSUpdate {lang}

Output of nvim --version

NVIM v0.6.0
Build type: Gentoo
Lua 5.1
Compilation: /usr/bin/x86_64-pc-linux-gnu-gcc -O2 -pipe -march=native -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DNVIM_TS_HAS_SET_MATCH_LIMIT -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=always -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -DMIN_LOG_LEVEL=3 -I/var/tmp/portage/app-editors/neovim-0.6.0/work/neovim-0.6.0_build/config -I/var/tmp/portage/app-editors/neovim-0.6.0/work/neovim-0.6.0/src -I/usr/include -I/var/tmp/portage/app-editors/neovim-0.6.0/work/neovim-0.6.0_build/src/nvim/auto -I/var/tmp/portage/app-editors/neovim-0.6.0/work/neovim-0.6.0_build/include
Compiled by portage@localhost

Features: +acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "/etc/vim/sysinit.vim"
  fall-back for $VIM: "/usr/share/nvim"

Run :checkhealth for more info

Additional context

No response

[clason]

I cannot reproduce this. Make sure the latex parser is updated (:TSUninstall latex, :TSInstall latex) and you don't have any stale parsers or queries lying around in other directories.

(Especially from colorschemes; latexTSP is not a standard highlight group! The output from TSHighlightCapturesUnderCursor looks very fishy; there should not be any @punctuation.delimiter in that tree.)

[clason]

The only thing I can think of is that your Neovim is too old to understand the #eq? directive (which is used to highlight words that are & as delimiter) -- but then you should get an error, not a wrong capture. Can you try upgrading your outdated Neovim to at least 0.6.1, and ideally master/nightly?

[shuber2]

I can confirm that it indeed works for NVIM v0.7.0-dev+1132-g15004473b.

[shuber2]

I would also like to remark that it does not work with nvim-0.6.1 as provided by Gentoo linux. This is kind of a pity that we need development version of neovim. Do you see any chance to make it work for 0.6.1, too?

[clason]

No

[clason]

(sorry, that's part of what "experimental" means.)

You could try the 0.6.1 appimage, though -- maybe it's just an issue with the Gentoo build (which has other drawbacks, like not building with LuaJIT).

[shuber2]

Interesting, it also does not work here: NVIM v0.7.0-dev+1135-gfdea15723-dirty Build type: Gentoo Lua 5.1

[shuber2]

Switchted to luajit, still having the issue:

NVIM v0.7.0-dev+1135-gfdea15723-dirty Build type: Gentoo LuaJIT 2.0.5

I will open up a bug report for Gentoo.

[clason]

Oh god, that's even worse, that version is years out of date.

The issue is the bundled tree-sitter library, which is probably outdated as well.

[shuber2]

The tree-sitter library is of version 0.20.0.

[clason]

Yeah, we expect 0.20.4

[shuber2]

You are the best. Upgraded to dev-libs/tree-sitter to nightly (9999), and it works! Furthermore, it also works after downgrading to neovim-0.6.1. So it seems this is entirely about dev-libs/tree-sitter. I will open a bug request for Gentoo suggesting to provide tree-sitter-0.20.4.

Thanks for sticking with me on that issue!

[shuber2]

One more thing: Would it make sense to explicitly mention the >=tree-sitter-0.20.4 requirement in the requirements section of nvim-treesitter's README?

[clason]

I don't think it has to be 0.20.4 precisely. The official 0.6.1 release is built with 0.20.1, which should work as well (although I can't test at the moment).

The README already states that 0.6.1 or nightly is required; if distributions provide modified and broken releases, there's nothing we can do about it.

[shuber2]

I installed tree-sitter 0.20.1 now on Gentoo, and it still works, and tree-sitter ABI version 14 (instead of 13 as for tree-sitter 0.20.0) is available.