Myriad-Dreamin / tinymist

Tinymist [ˈtaɪni mɪst] is an integrated language service for Typst [taɪpst].
https://myriad-dreamin.github.io/tinymist
Apache License 2.0
716 stars 30 forks source link

index out of range error in `nvim/runtime/lua/vim/lsp/semantic_tokens.lua` #638

Open cskeeters opened 3 weeks ago

cskeeters commented 3 weeks ago

A typst file with contents:

Bob’s car

Will cause the error index out of range @ semantic_tokens.lua:149 in the neovim runtime.

Error executing vim.schedule lua callback: ...0.2_1/share/nvim/runtime/lua/vim/lsp/semantic_tokens.lua:304: ...0.2_1/share/nvim/runtime/lua/vim/lsp/semantic_tokens.lua:149: index out of range
stack traceback:
        [builtin#36]: at 0x0102d65078
        ...0.2_1/share/nvim/runtime/lua/vim/lsp/semantic_tokens.lua:304: in function 'handler'
        ...eovim/0.10.2_1/share/nvim/runtime/lua/vim/lsp/client.lua:687: in function ''
        vim/_editor.lua: in function <vim/_editor.lua:0>

NOTE: No error with non-smart quotes.

It looks like Neovim's runtime is parsing tokens from the tinymist LSP and erroring.

Package/Software version:

tinymist extension version: v0.11.0. Get it by tinymist --version in terminal.

tinymist
Build Timestamp:     2024-10-05T07:02:18.000000000Z
Build Git Describe:  VERGEN_IDEMPOTENT_OUTPUT
Commit SHA:          VERGEN_IDEMPOTENT_OUTPUT
Commit Date:         None
Commit Branch:       None
Cargo Target Triple: aarch64-apple-darwin
Typst Version:       0.11.1
Typst Source:        git+https://github.com/Myriad-Dreamin/typst.git?tag=tinymist-v0.11.1-2#314749ea26ac3f62622bc3b3c30d3a7d6a457a74
Myriad-Dreamin commented 3 weeks ago

Downgrading neovim can mitigate this error.

Myriad-Dreamin commented 3 weeks ago

The above are not quite related, but fixing warnings that were detected by vscode. One of them, "allowing overlapping tokens", is not implemented by all clients.

cskeeters commented 3 weeks ago

See https://github.com/neovim/neovim/issues/30675

Setting offset_encoding = "utf-8" addresses the issue for me.

lspconfig.tinymist.setup{
    on_attach = on_attach,
    flags = lsp_flags,
    capabilities = capabilities,
    offset_encoding = "utf-8",
    settings = {
    }
}

https://github.com/igorlfs/dotfiles/commit/51199d1a33a5fa28f198a6dcd6858aad8e5e106c#diff-4294c316cad3cabb99eaf4879fa9556e579283822b1c4dab0f92f18d2ce32f0eR3

Myriad-Dreamin commented 3 weeks ago

Setting offset_encoding = "utf-8" addresses the issue for me.

It is rare to set offset_encoding manually, so this looks like a workaround.

https://github.com/neovim/neovim/issues/30675#issuecomment-2395543389

Yes, the LSP appears to be sending back UTF-8 encoded positions, not UTF-16 encoded positions (default) so you would need to specify that in the set up.

It was previously being masked by this pcall 003b8a2#diff-3b3a433b8799463f3cb0bcfbb0f1b699c0b949e7e31516e82b94a9c6aea3b395L81

But, i think this is the correct behaviour

This brings some smell of neovim (implementation or config) bugs because language server is likely to not respond with utf-8 encoding if the server had handshaked with agreeing position_encoding: Utf16 (by default). I checked it just now, language server responds tokens with utf-16 position encoding in VS Code (by default). Three possible causes:

Hint: You can check the handshake result in the log:

[2024-10-07T05:16:39Z INFO tinymist::server] LanguageState: initialized with config Config { const_config: ConstConfig { position_encoding: Utf16, ... }, ... }

Myriad-Dreamin commented 2 weeks ago

Neovim developers are fixing this bug. We can check it again in the next release of neovim.