index out of range error in `nvim/runtime/lua/vim/lsp/semantic_tokens.lua`

[cskeeters]

A typst file with contents:

Bob’s car

Will cause the error index out of range @ semantic_tokens.lua:149 in the neovim runtime.

Error executing vim.schedule lua callback: ...0.2_1/share/nvim/runtime/lua/vim/lsp/semantic_tokens.lua:304: ...0.2_1/share/nvim/runtime/lua/vim/lsp/semantic_tokens.lua:149: index out of range
stack traceback:
        [builtin#36]: at 0x0102d65078
        ...0.2_1/share/nvim/runtime/lua/vim/lsp/semantic_tokens.lua:304: in function 'handler'
        ...eovim/0.10.2_1/share/nvim/runtime/lua/vim/lsp/client.lua:687: in function ''
        vim/_editor.lua: in function <vim/_editor.lua:0>

NOTE: No error with non-smart quotes.

It looks like Neovim's runtime is parsing tokens from the tinymist LSP and erroring.

Package/Software version:

• MacOS Sequoia 15.0.1
• Neovim v0.10.2 (Release)
• LuaJIT 2.1.1727870382
• tinymist 0.11.28

tinymist extension version: v0.11.0. Get it by tinymist --version in terminal.

tinymist
Build Timestamp:     2024-10-05T07:02:18.000000000Z
Build Git Describe:  VERGEN_IDEMPOTENT_OUTPUT
Commit SHA:          VERGEN_IDEMPOTENT_OUTPUT
Commit Date:         None
Commit Branch:       None
Cargo Target Triple: aarch64-apple-darwin
Typst Version:       0.11.1
Typst Source:        git+https://github.com/Myriad-Dreamin/typst.git?tag=tinymist-v0.11.1-2#314749ea26ac3f62622bc3b3c30d3a7d6a457a74

[Myriad-Dreamin]

Downgrading neovim can mitigate this error.

[Myriad-Dreamin]

The above are not quite related, but fixing warnings that were detected by vscode. One of them, "allowing overlapping tokens", is not implemented by all clients.

[cskeeters]

See https://github.com/neovim/neovim/issues/30675

Setting offset_encoding = "utf-8" addresses the issue for me.

lspconfig.tinymist.setup{
    on_attach = on_attach,
    flags = lsp_flags,
    capabilities = capabilities,
    offset_encoding = "utf-8",
    settings = {
    }
}

https://github.com/igorlfs/dotfiles/commit/51199d1a33a5fa28f198a6dcd6858aad8e5e106c#diff-4294c316cad3cabb99eaf4879fa9556e579283822b1c4dab0f92f18d2ce32f0eR3

[Myriad-Dreamin]

Setting offset_encoding = "utf-8" addresses the issue for me.

It is rare to set offset_encoding manually, so this looks like a workaround.

https://github.com/neovim/neovim/issues/30675#issuecomment-2395543389

Yes, the LSP appears to be sending back UTF-8 encoded positions, not UTF-16 encoded positions (default) so you would need to specify that in the set up.

It was previously being masked by this pcall 003b8a2#diff-3b3a433b8799463f3cb0bcfbb0f1b699c0b949e7e31516e82b94a9c6aea3b395L81

But, i think this is the correct behaviour

This brings some smell of neovim (implementation or config) bugs because language server is likely to not respond with utf-8 encoding if the server had handshaked with agreeing position_encoding: Utf16 (by default). I checked it just now, language server responds tokens with utf-16 position encoding in VS Code (by default). Three possible causes:

• the handshake is not performed correctly.
• the default configuration is not aligned correct.
• not a position encoding problem, but some other bugs of tinymist about semantic tokens.

Hint: You can check the handshake result in the log:

[2024-10-07T05:16:39Z INFO tinymist::server] LanguageState: initialized with config Config { const_config: ConstConfig { position_encoding: Utf16, ... }, ... }

[Myriad-Dreamin]

Neovim developers are fixing this bug. We can check it again in the next release of neovim.