[Crash] Invalid Lexer bump

[MAGNAT2645]

Basic informations

• OS: Arch Linux
• VSCode version: Latest (vscodium-bin from AUR)
• Extension version: Latest

Further Information

The extension started to crash and stops loading after 5 crashes. Log says Invalid Lexer bump. Happened today and still happening, was working fine earlier.

There's also another error but it doesn't break anything, just annoying. Failed to run spcomp. No such file or directory (os error 2) This error happens only when you open (or switch to) any workspace (or just restart LSP). SourcePawnLanguageServer.spcompPath is set up to spcomp with absolute path and compiling using Ctrl+Shift+D works fine.

To Reproduce

Just open any workspace or restart LSP. That's all.

Code to reproduce the behaviour

None.

Expected behaviour

Extension should not crash like this. I don't know what could have been changed to make the extension crash now.

Error messages

thread 'main' panicked at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/logos-0.12.1/src/lexer.rs:162:9:
Invalid Lexer bump
stack backtrace:
   0:     0x559a9c191d1c - std::backtrace_rs::backtrace::libunwind::trace::he43a6a3949163f8c
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x559a9c191d1c - std::backtrace_rs::backtrace::trace_unsynchronized::h50db52ca99f692e7
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x559a9c191d1c - std::sys_common::backtrace::_print_fmt::hd37d595f2ceb2d3c
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x559a9c191d1c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h678bbcf9da6d7d75
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x559a9c1c0c4c - core::fmt::rt::Argument::fmt::h3a159adc080a6fc9
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/fmt/rt.rs:138:9
   5:     0x559a9c1c0c4c - core::fmt::write::hb8eaf5a8e45a738e
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/fmt/mod.rs:1094:21
   6:     0x559a9c18dade - std::io::Write::write_fmt::h9663fe36b2ee08f9
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/io/mod.rs:1714:15
   7:     0x559a9c191b04 - std::sys_common::backtrace::_print::hcd4834796ee88ad2
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x559a9c191b04 - std::sys_common::backtrace::print::h1360e9450e4f922a
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x559a9c193763 - std::panicking::default_hook::{{closure}}::h2609fa95cd5ab1f4
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panicking.rs:270:22
  10:     0x559a9c19347c - std::panicking::default_hook::h6d75f5747cab6e8d
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panicking.rs:290:9
  11:     0x559a9c193e5e - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h05249de742e1768e
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/alloc/src/boxed.rs:2021:9
  12:     0x559a9c193e5e - std::panicking::rust_panic_with_hook::h57e78470c47c84de
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panicking.rs:711:13
  13:     0x559a9bffc81a - std::panicking::begin_panic::{{closure}}::h346122c98c29cb9b
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panicking.rs:628:9
  14:     0x559a9bffc7e6 - std::sys_common::backtrace::__rust_end_short_backtrace::h77720955aae04cc0
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/sys_common/backtrace.rs:170:18
  15:     0x559a9bad782c - std::panicking::begin_panic::h89bae906e6c80884
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panicking.rs:627:12
  16:     0x559a9bffafa6 - logos::lexer::Lexer<Token>::bump::hda2943b888f7034f
                               at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/logos-0.12.1/src/lexer.rs:162:9
  17:     0x559a9bffafa6 - sourcepawn_lexer::pragma::lex_pragma_arguments::h6144cc4586b50d66
  18:     0x559a9c005dc6 - <sourcepawn_lexer::token::Token as logos::Logos>::lex::goto478_ctx489_x::h34ea0dd4d214d5a6
                               at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sourcepawn_lexer-0.1.1/src/token.rs:4:10
  19:     0x559a9c005dc6 - <sourcepawn_lexer::token::Token as logos::Logos>::lex::goto937_at1_ctx489_x::hcbab5aa3bcc4384a
                               at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sourcepawn_lexer-0.1.1/src/token.rs:4:10
  20:     0x559a9c005dc6 - <sourcepawn_lexer::token::Token as logos::Logos>::lex::goto965_ctx489_x::h070fa7f18ddd8845
                               at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sourcepawn_lexer-0.1.1/src/token.rs:4:10
  21:     0x559a9bff9a88 - <logos::lexer::Lexer<Token> as core::iter::traits::iterator::Iterator>::next::hb55e745b8f27a9e4
                               at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/logos-0.12.1/src/lexer.rs:193:9
  22:     0x559a9bff9a88 - <sourcepawn_lexer::lexer::SourcepawnLexer as core::iter::traits::iterator::Iterator>::next::hc842a2cc135ea2a6
                               at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sourcepawn_lexer-0.1.1/src/lexer.rs:268:32
  23:     0x559a9bf7e168 - store::graph::<impl store::Store>::get_include_ids_from_document::h89245f4535a60f48
                               at /home/runner/work/sourcepawn-vscode/sourcepawn-vscode/crates/store/src/graph.rs:61:23
  24:     0x559a9bf7e9e2 - store::graph::<impl store::Store>::load_projects_graph::h5e1d21c8b0f28089
                               at /home/runner/work/sourcepawn-vscode/sourcepawn-vscode/crates/store/src/graph.rs:102:41
  25:     0x559a9be86366 - sourcepawn_lsp::server::files::<impl sourcepawn_lsp::server::Server>::reparse_all::h28f4e864ed90fdc6
                               at /home/runner/work/sourcepawn-vscode/sourcepawn-vscode/crates/sourcepawn_lsp/src/server/files.rs:25:24
  26:     0x559a9be8da91 - sourcepawn_lsp::server::Server::process_messages::h3436bc22efc781af
                               at /home/runner/work/sourcepawn-vscode/sourcepawn-vscode/crates/sourcepawn_lsp/src/server/mod.rs:317:51
  27:     0x559a9be9288b - sourcepawn_lsp::server::Server::run::h57364a3911cdb52d
                               at /home/runner/work/sourcepawn-vscode/sourcepawn-vscode/crates/sourcepawn_lsp/src/server/mod.rs:344:9
  28:     0x559a9bae9f53 - sourcepawn_lsp::main::hb18084b55bf78ef6
                               at /home/runner/work/sourcepawn-vscode/sourcepawn-vscode/crates/sourcepawn_lsp/src/main.rs:62:5
  29:     0x559a9baf5a53 - core::ops::function::FnOnce::call_once::h18fba4c19fb171ba
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/ops/function.rs:250:5
  30:     0x559a9baf5a53 - std::sys_common::backtrace::__rust_begin_short_backtrace::hd0669cef2ac17b38
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/sys_common/backtrace.rs:154:18
  31:     0x559a9baf7549 - std::rt::lang_start::{{closure}}::h457b271042a58271
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/rt.rs:166:18
  32:     0x559a9c184eeb - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hbcc4f8a3f5ada78c
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/ops/function.rs:284:13
  33:     0x559a9c184eeb - std::panicking::try::do_call::he5f117a9e13dadde
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panicking.rs:502:40
  34:     0x559a9c184eeb - std::panicking::try::h2f3af9afce3a0443
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panicking.rs:466:19
  35:     0x559a9c184eeb - std::panic::catch_unwind::h6d6c387f38ef05ea
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panic.rs:142:14
  36:     0x559a9c184eeb - std::rt::lang_start_internal::{{closure}}::h6ca09d5905711415
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/rt.rs:148:48
  37:     0x559a9c184eeb - std::panicking::try::do_call::ha9fd18ea06654a4b
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panicking.rs:502:40
  38:     0x559a9c184eeb - std::panicking::try::hda5c2a4432362341
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panicking.rs:466:19
  39:     0x559a9c184eeb - std::panic::catch_unwind::h440f731b142bc235
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panic.rs:142:14
  40:     0x559a9c184eeb - std::rt::lang_start_internal::hc0b4e50f058f62ce
                               at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/rt.rs:148:20
  41:     0x559a9baecd5c - main
  42:     0x7f3cafde3cd0 - <unknown>
  43:     0x7f3cafde3d8a - __libc_start_main
  44:     0x559a9bae4f5e - _start
  45:                0x0 - <unknown>
[Error - 16:48:18] The SourcePawn Language Server server crashed 5 times in the last 3 minutes. The server will not be restarted.

I guess these crashes happen due to assert check in the logos package (lexer.rs) but idk how to fix this

    pub fn bump(&mut self, n: usize) {
        self.token_end += n;

        assert!(
            self.source.is_boundary(self.token_end),
            "Invalid Lexer bump",
        )
    }

[Sarrus1]

Hello! Thanks for reporting this, I was looking for a way to replicate the issue!

What's probably happening is that there is a mistake in the lexer grammar somewhere in the preprocessor, that causes it to to one token too far.

Would it be possible for you to send me one of the workspace that crashes your lsp please? With all the files

[MAGNAT2645]

I found possible issue. #pragma deprecated "Устаревшая функция. Плагин автоматически очищает всё, что создал другой выгруженный плагин." If i comment out this line, extension doesn't crash. So i guess this happens due to using Unicode symbols (but the compiler itself seems to support these).

[MAGNAT2645]

But this one is starting to annoy every time i open workspaces. Failed to run spcomp. No such file or directory (os error 2)

[Sarrus1]

Thanks for helping me replicate this! Fixed in https://github.com/Sarrus1/sourcepawn-lexer/releases/tag/0.1.2 I will merge this when I merge feat/salsa (which might take some time...)

[Sarrus1]

Fixed in v7.1.0, which is stable.