nimsuggest crashes for specific file in project

[observant2]

In this project I get suggestions for components.nim and engine.nim, but not for mex.nim.

I debugged this for an hour now, but cannot find the place where it fails.

Here's the relevant stacktrace

Got frame:
{"jsonrpc":"2.0","id":"2","method":"textDocument/completion","params":{"textDocument":{"uri":"file:///C:/Users/Admin/Documents/Code/nim/mex/src/mex.nim"},"position":{"line":171,"character":1}}}
Got valid Request message of type textDocument/completion
Got request for URI: file:///C:/Users/Admin/Documents/Code/nim/mex/src/mex.nim copied to C:\Users\Admin\AppData\Local\Temp\nimlsp\6A10B233.nim
Running equivalent of: sug /C:/Users/Admin/Documents/Code/nim/mex/src/mex.nim;C:\Users\Admin\AppData\Local\Temp\nimlsp\6A10B233.nim:172:1
C:\Users\Admin\Documents\Code\nim\nimlsp\src\nimlsp.nim(228) nimlsp
C:\Users\Admin\Documents\Code\nim\nimlsp\src\nimlsppkg\suggestlib.nim(88) sug
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\nimsuggest\nimsuggest.nim(750) runCmd
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\nimsuggest\nimsuggest.nim(191) executeNoHooks
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\modules.nim(144) compileProject
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\modules.nim(85) compileModule
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\passes.nim(210) processModule
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\passes.nim(86) processTopLevelStmt
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\sem.nim(606) myProcess
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\sem.nim(569) semStmtAndGenerateGenerics
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\semstmts.nim(2224) semStmt
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\semexprs.nim(987) semExprNoType
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\semexprs.nim(2742) semExpr
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\semstmts.nim(2164) semStmtList
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\semexprs.nim(2746) semExpr
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\semstmts.nim(598) semConst
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\sem.nim(363) semConstExpr
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\sem.nim(300) fixupTypeAfterEval
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\semmacrosanity.nim(56) annotateType
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\compiler\semmacrosanity.nim(38) ithField
C:\Users\Admin\.choosenim\toolchains\nim-1.0.2\lib\system\fatal.nim(39) sysFatal
Error: unhandled exception: index out of bounds, the container is empty [IndexError]

The weird thing is that when I run nimsuggest itself on the command line I get valid suggestions for mex.nim, including the functions from the imported modules (engine.nim and components.nim).

Any ideas why this happens?

[PMunch]

Hmm, I don't observe the same crash here. Might be something that only happens on Windows machines.. I do however sometimes see that it the debug logging from nimlsp says that it has some suggestions, but I don't see them in my editor. Not quite sure what causes that though. Might be something with my editor.

[observant2]

Okay, I'll try to investigate further. If anyone else uses Windows, feel free to investigate too. 🙈

[observant2]

It seems like nimsuggest always crashes on compileProject or compileModule when fed with mex.nim.

It is hard for me to debug this. Is there a way to output variable values inside of nimsuggest.nim? I cannot use debugEcho there and a simple echo isn't output to nimlsp it seems. 🤷‍♂

[PMunch]

Hmm, if it is inside nimsuggest.nim it should crass with nimsuggest as well. Have you tried to build nimlsp yourself with nimble debug? This makes it quite a bit more verbose, including showing the nimsuggest command equivalents it tries to run.

[PMunch]

Otherwise you can have a look at what debugEcho actually does. It writes it out to stderr (jsonrpc communication with the LSP server goes over stdout) and a file. So you could probably do something similar in nimsuggest.nim.

[cyraxjoe]

I am also getting some random cashes while using it from emacs.

Even with nimlsp.nim from this project I get some evaluation errors in the evaluation of the JsonSchema macro after the include (https://github.com/PMunch/nimlsp/blob/master/src/nimlsp.nim#L2), eventually https://github.com/nim-lang/Nim/blob/devel/compiler/vm.nim#L2267 this offset evaluates to -1 which... ends up in a call like: newSeq(tos.slots, -1) breaking everything, nothing catches that error and terminates nimlsp.

I'm using Debian testing, with the latest nim. I still haven't wrap my head around all the pieces to have an clear hypothesis, so far is only speculation and crashes that I'm tracing with strace and logging internals from the nim evaluation with a locally modified nim.

I will update later if I have any new clue.

[cyraxjoe]

I modified the logging functionality and this is the full context of the recurrent error that I'm testing as an example, I'm now wrapping the numsuggest calls in a try/except block and logging the failures, the line numbers are a bit off, given the extra logging lines that I added.

[21:55:33] - DEBUG: Got request for URI: file:///home/joe/repos/nimlsp/src/nimlsp.nim copied to /tmp/nimlsp/00000000FF444B37.nim
[21:55:33] - DEBUG: Running equivalent of: def /home/joe/repos/nimlsp/src/nimlsp.nim;/tmp/nimlsp/00000000FF444B37.nim:507:0
[21:55:33] - DEBUG: Full command file = /home/joe/repos/nimlsp/src/nimlsp.nim dirtyfile = /tmp/nimlsp/00000000FF444B37.nim line=507 col=0
[21:55:33] - DEBUG: Full command is ideDef
[21:55:33] - DEBUG: sym: jsonSchema@28735081
[21:55:33] - DEBUG: sym kind: skMacro
[21:55:33] - DEBUG: sym typ: macro (pattern: untyped): untyped
[21:55:33] - DEBUG: sym typ.len: 2
[21:55:33] - DEBUG: tos.slots: @[]
[21:55:33] - DEBUG: maxSlots: -1
[21:55:33] - DEBUG: n.safeLen: 2
[21:55:33] - ERROR: Unable to obtain suggestions: 'value out of range: -1 notin 0 .. 9223372036854775807'
[21:55:33] - ERROR: Stacktrace:
[21:55:33] - ERROR:                     
                    /home/joe/repos/nimlsp/src/nimlsp.nim(268) nimlsp
                    /home/joe/repos/nimlsp/src/nimlsppkg/suggestlib.nim(96) def
                    /home/joe/repos/Nim/nimsuggest/nimsuggest.nim(760) runCmd
                    /home/joe/repos/Nim/nimsuggest/nimsuggest.nim(194) executeNoHooks
                    /home/joe/repos/Nim/compiler/modules.nim(143) compileProject
                    /home/joe/repos/Nim/compiler/modules.nim(91) compileModule
                    /home/joe/repos/Nim/compiler/passes.nim(210) processModule
                    /home/joe/repos/Nim/compiler/passes.nim(86) processTopLevelStmt
                    /home/joe/repos/Nim/compiler/sem.nim(609) myProcess
                    /home/joe/repos/Nim/compiler/sem.nim(572) semStmtAndGenerateGenerics
                    /home/joe/repos/Nim/compiler/semstmts.nim(2303) semStmt
                    /home/joe/repos/Nim/compiler/semexprs.nim(1006) semExprNoType
                    /home/joe/repos/Nim/compiler/semexprs.nim(2768) semExpr
                    /home/joe/repos/Nim/compiler/semstmts.nim(2243) semStmtList
                    /home/joe/repos/Nim/compiler/semexprs.nim(2810) semExpr
                    /home/joe/repos/Nim/compiler/semstmts.nim(2171) evalInclude
                    /home/joe/repos/Nim/compiler/semstmts.nim(2149) incMod
                    /home/joe/repos/Nim/compiler/semstmts.nim(2303) semStmt
                    /home/joe/repos/Nim/compiler/semexprs.nim(1006) semExprNoType
                    /home/joe/repos/Nim/compiler/semexprs.nim(2768) semExpr
                    /home/joe/repos/Nim/compiler/semstmts.nim(2243) semStmtList
                    /home/joe/repos/Nim/compiler/semexprs.nim(2665) semExpr
                    /home/joe/repos/Nim/compiler/semexprs.nim(988) semDirectOp
                    /home/joe/repos/Nim/compiler/semexprs.nim(880) afterCallActions
                    /home/joe/repos/Nim/compiler/sem.nim(471) semMacroExpr
                    /home/joe/repos/Nim/compiler/vm.nim(2288) evalMacroCall
                    /home/joe/repos/Nim/lib/system/fatal.nim(49) sysFatal

[bung87]

this should go to nim repo see https://github.com/nim-lang/Nim/issues/14461
I have invested this last year, it's basically nim's compiler problem, the simple solution is avoiding using complex template.

[johnnovak]

Confirmed, I'm also seeing regular crashes in my moderately template/macro heavy projects, which makes nimsuggest virtually unusable.

[PMunch]

This is unfortunately a nimsuggest issue, which isn't maintained as part of nimlsp but as a separate project. So please raise all of these errors in the Nim repository so they can get fixed.

[bung87]

likely fixed in https://github.com/nim-lang/Nim/commit/2cbfc1e51470bae7bc3570caf4432bd062f11599