Developer tooling

[mratsim]

The steady pace of Nim development, the language and the communication around the new features like ARC and ORC is attracting goodwill and devs are trying Nim out.

However as mentioned in many many places, Nim current success is not supported by the tooling and is actually despite it.

Situation

• https://forum.nim-lang.org/t/6682#41440 (User left with no linting, no resolution)
• https://forum.nim-lang.org/t/6777#42197 (New user surprised by the weak IDE support)
• https://forum.nim-lang.org/t/6756#42080 (Nim success is despite a good toolset)
• https://forum.nim-lang.org/t/6862#42899 (Starting a pure Nim Vscode extension) but code archived https://github.com/saem/vscode-nim-fork edit: living at https://github.com/saem/vscode-nim
• https://forum.nim-lang.org/t/6985#43816 Nimsuggest or VScode hindering module organization
• https://forum.nim-lang.org/t/7014#44138 "Editor support is the biggest gripe I have with the language"
• https://forum.nim-lang.org/t/7140#45056 Nim suggest doesn't work with Jester and Karax
• Personally I have to deactivate nimsuggest while working on Weave because it gets stuck at 100% CPU almost immediately. This means no syntax highlighting.
• @dom96 thread: https://github.com/nim-lang/Nim/issues/16001
• No bug involving nimsuggest was ever fixed the only closed was dom's thread

Additionally this question on Telegram sparked my RFC:

No data from Telegram and IRC for now feel free to browse https://www.google.com/search?q=nimsuggest+site:irclogs.nim-lang.org

Analysis

1. I believe there are 2 issues here: nimsuggest and the vscode extensions. AFAIK I didn't see issues with LSP or vim/nvim or emacs that aren't due to nimsuggest.

2. From the various threads, I understand that the plan is to rewrite nimsuggest once incremental compilation lands.

3. Visual Studio Code is one of the most popular editor for developers and is the most popular Nim editor by far despite the issues reported over and over with nimsuggest and the extensions: https://nim-lang.org/blog/2020/02/18/community-survey-results-2019.html

Opinions

1. The tools around a language are an integral part of the whole developer experience. The fact that the extension for the number 1 Nim dev environment has so much issues is a problem and will be a minus for developers evaluating Nim against other languages.

2. Incremental compilation has been deprioritized vs other features even though it is part of Milestones 2020 roadmap at https://github.com/nim-lang/RFCs/milestone/1 (https://github.com/nim-lang/RFCs/issues/46). From the look of it, incremental compilation has architectural unknown that makes it unlikely to be delivered in the next 6 months.

3. From a prioritization point of view, I believe developers tools are significantly more important than incremental compilation. If compilation is slow, you can always throw faster hardware at it. We're not in C++ land here. However if a developer is slow, throwing more developers doesn't work: there is communication and synchronization friction, getting to know the code base and ... tooling frictions.

4. Companies like Status or the one screenshoted are recruiting Nim talents but also Go, C++, Rust talents for their team. While those team members are eager to learn Nim, they need help to get started. Tooling can go a long way.

5. One of the top critic of Nim is the seemingly absence of mandatory namespacing for imports which makes it hard to know where a function came from, there are 2 replies:

   • enforce myModule.myImport(args) syntax in your codebase.
   • use better tooling

   Unfortunately there is no better tooling right now.

6. I think VSCode has some smart Intellisense that doesn't rely on the language but just the project code. Experimented users probably don't need a full nimsuggest. However they do enjoy syntax highlighting working and working with macros. Unfortunately if nimsuggest crashes due to macros, there is no highlighting and all suggestions disappear even pure Intellisense. Case in point, I just type spawn( in Weave and autocompletion dies and VScode intellisense (cpp-tools) gets stuck at 160% CPU usage (and if in macros it's nimsuggest)

   edit: It probably is TabNine, a deep learning powered auto-completion tool: https://www.tabnine.com/, https://www.tabnine.com/blog/deep/

Action items

1. We have a couple of full-time developers working on Nim, I think working on developer tooling should be one of 2021 focus. I do think a VScode extension under Nim umbrella is in order.

2. An extension with just syntax highlighting and uses VScode auto-complete builtins that doesn't crash or use 100% CPU on Karax, Jester, Weave and other codebase using generics, statics and macros would please seasoned developers.

3. For a full extension that integrates with or embed the compiler, don't wait for IC, there are workaround for slow machines but no workaround for discouraged developers. This is important for new joiners

4. Good debugger integration (viewing variable contents, GDB/LLDB) would also be a boon for debugging complex code. debugEcho gets old.

5. Regarding architecture, I'm not sure if a CLI tool or a service with a RPC API is better so another RFC might be in order to discuss that

[moigagoo]

I wholeheartedly support the intention to improve the tooling for Nim development as it has been as huge pain for me as well.

But the focus purely on VSCode I don't agree with. I think we should:

1. focus on LSP and Nimsuggest support
2. remove error checking from Nimsuggest and let it do the code navigation, completions, and signature tooltips.

[liquidev]

I agree with @moigagoo, not everyone wants to use a resource-eating Electron behemoth like VS Code. I'm currently working on creating tooling for the lite text editor, other people use Vim and Emacs, it's best if we target the common ground.

[moigagoo]

LSP support is already quite good at least for Vim, Emacs, and Sublime Text. So not much work is required here.

The only big problem I'm having now is that Nimsuggest constantly tries to do what it can't: that is, error-check code with macros.

So maybe we could have a fork of Nimsuggest, e.g. NimsuggestLite, and remove error checking from it for good. Then users of tools like nimlsp could start using that fork instead of the original Nimsuggest right away, enjoying code completion while not having to constantly see false positive alarms.

[mratsim]

But the focus purely on VSCode I don't agree with. I think we should:

To be clear, I'm not focusing purely on VScode but:

1. There are twice more Nim users using VScode than the 2nd most used platform, and even if we combine VIM + Neovim, vscode is ahead
2. New joiners are more likely to use VScode as it's the dominant code editor out there https://insights.stackoverflow.com/survey/2019#technology-_-most-popular-development-environments
3. VScode is the environment with the most problems (though I didn't test the IntelliJ plugin) as nimsuggest's problems are compounded with the extension problems.

[Araq]

For me nimsuggest often fails yet syntax highlighting always keeps working, I don't understand this remark.

[mratsim]

For me nimsuggest often fails yet syntax highlighting always keeps working, I don't understand this remark.

I need to deactivate completely the Nim extension to be able to work on macro heavy code like in Weave because it slows laptop to a crawl, generates a lot of heat and is basically plain useless.

[Araq]

VSCode's sytnax highlighting isn't done by nimsuggest.

[mratsim]

But because I deactivate the extension, I have nothing at all.

[Araq]

Pretty sure you can disable nimsuggest without disabling the extension altogether.

[moigagoo]

VSCode's sytnax highlighting isn't done by nimsuggest.

 But because I deactivate the extension, I have nothing at all.

In Sublime Text, the highlighting is done with Nimlime, while code nav and error checking is done with nimlsp. So, at least with ST Araq's suggestion is viable.

@mratsim maybe having to separate extensions for VSCode like we have for ST would be a good bet? This way we could focus on improving one tool—nimsuggest—and one protocol—LSP—instead of solving issues that are specific to a particular platform.

[Araq]

And to make it clear: I fully support the idea of focussing on VSCode plugin quality first and foremost.

[moigagoo]

 There are twice more Nim users using VScode than the 2nd most used platform, and even if we combine VIM + Neovim, vscode is ahead

OK that's an argument I can't argue with. Just trying to find solutions that would work for all editors, including VSCode.

[mratsim]

VSCode's sytnax highlighting isn't done by nimsuggest.

But because I deactivate the extension, I have nothing at all.

In Sublime Text, the highlighting is done with Nimlime, while code nav and error checking is done with nimlsp. So, at least with ST Araq's suggestion is viable.

@mratsim maybe having to separate extensions for VSCode like we have for ST would be a good bet? This way we could focus on improving one tool—nimsuggest—and one protocol—LSP—instead of solving issues that are specific to a particular platform.

Yes this what I suggested here:

2. An extension with just syntax highlighting and uses VScode auto-complete builtins that doesn't crash or use 100% CPU on Karax, Jester, Weave and other codebase using generics, statics and macros would please seasoned developers.

3. For a full extension that integrates with or embed the compiler, don't wait for IC, there are workaround for slow machines but no workaround for discouraged developers. This is important for new joiners

An extension that just does the minimal stuff that can be done without nimsuggest. This is a temporary (hopefully) workaround so that seasoned Nim developers who tend to use macro, templates, static generics are not hindered by nimsuggest bugs on those features. I'm not sure what are all the features independent from nimsuggest but as long as it doesn't crash on Karax/Jester/Weave and other packages tagged as important in Nim CI the extension is filling its role.

Then we need to work on nimsuggest+extension or lsp+extension or maybe those can be solved by lsp/nimsuggest alone to bring the tooling to some standard we have to define so that it helps the newbies.

Basically I see 2 needs:

1. Seasoned devs need a tool that doesn't hinder them because it crashes or gets stuck at 100% CPU. In my case I would be satisfied for starter with just syntax highlighting + suggestions just based on parameters/local variables declared which doesn't need nimsuggest to scan imports AFAIK. This should be possible if we strip nimsuggest from the current extension.

   I think Vim, Neovim, Sublime text are at that stage (?). They might be using tricks like restarting nimsuggest often to get there though.

2. Newbies Nim devs need a tool to help them navigate errors, types, a codebase, the standard library, imports, ... This one looks like a long-term complex development, and support project.

[moigagoo]

Seasoned devs need a tool that doesn't hinder them because it crashes or gets stuck at 100% CPU. In my case I would be satisfied for starter with just syntax highlighting + suggestions just based on parameters/local variables declared which doesn't need nimsuggest to scan imports AFAIK. This should be possible if we strip nimsuggest from the current extension.

 Newbies Nim devs need a tool to help them navigate errors, types, a codebase, the standard library, imports, ... This one looks like a long-term complex development, and support project.

The thing is, Nimsuggest is already good at code navigation, completions, and more. It's only the error checking that is broken really.

So, the long-term goal should be reduced only to error checking. Navigation, completion, suggestions we can already have. And highlighting, as you mentioned, is a totally separate thing done with editor-specific tools to each editor.

[liquidev]

@moigagoo

The thing is, Nimsuggest is already good at code navigation, completions, and more. It's only the error checking that is broken really.

I strongly disagree with this. Nimsuggest's biggest hindrance is compiler speed and general stability. And again, no extension we know of uses nimsuggest for error checking, only nim check.

The bigger and more instantiation-heavy your project is, the slower nimsuggest runs, and it crashes much more often. You can observe this on projects such as my Planet Overgamma, where in its main file suggestions become practically unusable – when using nim.nvim, it constantly crashes, suggestions appear after way too long for them to be useful. The overall instability of nimsuggest led me to having to restart it much too often, which is why I dropped it altogether in favor of a more lightweight solution like @mratsim suggested.

[moigagoo]

@liquid600pgm

And again, no extension we know of uses nimsuggest for error checking, only nim check.

I'm using nimlsp with Sublime Text, and it used Nimsuggest to report errors. And I get a ton of errors all the time just by using Karax. It also crashes all the time.

Apart from that, it's OK for me. My projects aren't large so maybe I'm just not seeing the problems you see

[liquidev]

Oh, you're right. Thanks for reminding me, I have indeed used NimLSP, though not for long because of the buggy error reporting. But then iirc NimLSP uses nimsuggest in a non-intended way (executing nimsuggest procs directly from itself, rather than as a separate process), so that may be somewhat related.

[moigagoo]

But then iirc NimLSP uses nimsuggest in a non-intended way (executing nimsuggest procs directly from itself, rather than as a separate process), so that may be somewhat related.

Nimlsp uses Nimsuggest as a library, which is completely legal. Not sure if that's an intended way but nimlsp works pretty well with Nimsuggest.

[disruptek]

From the look of it, incremental compilation has architectural unknown that makes it unlikely to be delivered in the next 6 months.

I will be surprised if it's not delivered in the next 6 weeks, even though I cannot afford to work on it. What is the "architectural unknown" you mention?

[RSDuck]

https://forum.nim-lang.org/t/6862#42899 (Starting a pure Nim Vscode extension) but code archived https://github.com/saem/vscode-nim-fork

it's still alive here: https://github.com/saem/vscode-nim

EDIT:

No bug involving nimsuggest was ever fixed the only closed was dom's thread

nimsuggest also has it's own bug tracker with a similar situation, atleast in the not so far away past https://github.com/nim-lang/nimsuggest/issues

[liquidev]

@disruptek I just hope that your IC efforts are gonna pay off with nimsuggest finally becoming less of a nightmare to use.

[disruptek]

To recap here, IC introduces a so-called Packed AST format that is fast to de/serialize. This is compressed and stored in a rodfile, one per module. A modification to nimsuggest could allow it to work on these fully sem-checked artifacts instead of having to recompile the entire project continuously, which should help CPU dramatically -- and eventually, memory consumption as well.

[Araq]

Yeah, IMHO we have a really solid plan. It's still a lot of work though.

[saem]

https://forum.nim-lang.org/t/6862#42899 (Starting a pure Nim Vscode extension) but code archived https://github.com/saem/vscode-nim-fork

@mratsim as @RSDuck pointed out the Nim extension for VS Code written in Nim is still alive: https://github.com/saem/vscode-nim. If you could update your original post that'd be appreciated as I'd like to avoid giving people the wrong impression. I've not been super active with it, mostly because I'm experimenting with a bunch of different things to see how to get a more pleasant language server architecture, one that can be used by the aforementioned VS Code extension.

And to make it clear: I fully support the idea of focussing on VSCode plugin quality first and foremost.

I purposefully named the Nim port's nimble name nimvscode to be inline with nimsuggest, etc... because I'm holding out hope that it might at some point become an "official" project.

For the curious here are my notes while playing around with trying to make better tools for Nim:

As part of the above language server experiements, I ported Aporia's approach to firing up nimsuggest in a background process with threads for IO. The separate processes is honeslty a huge pain. Nimsuggest is probably the furthest along and it still doesn't have sequence/correlation ids for everything, data formats aren't all that clear and occasionally ambiguous. It's all very brittle. I'd much rather have have the ability to kill threads in the stdlib and then encourage more direct useage of the underlying code. Getting people into the code and comfortable is an avenue to improving nimsuggest and the compiler in general. The more the compiler and friends push towards a semi-stable API (it statically checks so don't have to try too hard) with lots of querying capabilities the more tooling that can be built.

The "declarative" stuff like nimble, cfg, nims, etc... are the next biggest hurdle. Everything involved with them seems to require brittle parsing or making sure you guess the correct paths when invoking various tools. As an example, last I used nimble dump --json it has better output than the default, but bin for example outputs only the final mapped version. This is unfortunate if you want to do something smart like use the various modules you are declaring as part of the binary package to be candidate project files.

Ambiguity with what are all the candiate project files in a directory structure is difficult. It's not anywhere as clear as it's been made out to be and choosing a good candidate is the difference between nimsuggest working or not. Even if nimsuggest had fewer bugs being able to determine all the various projects and either using user behaviour or explicit input changing between them based on what needs to be done is a must. The determination of candidate projects and at least a consensus on their relative priority/preference should be unambiguous.

As another example of ambiguous formats: calling nimble tasks outputs the various tasks one can execute, awesome. Hopefully the user didn't put a newline or tab characters in the help text because that makes the output ambiguous and hopefully whatever the user added doesn't match the regex used to match the output. This is also not part of the json output, and even though json has lots of issues it'd still be better than present.

For when one is parsing process output, there should simply be a versioned data models (API, file, and IO formats) and serdes for it. Just keep these around "forever", basically from now on every input/output to a tool should have a name, version, associated data types, and serdes, even if they're no longer supported this allows tooling to use the latest code as a library to work with older versions easily. I'm hoping I've just missed this and perhaps it already exists.

Even if things aren't perfect if I can get a language server that's relatively easy to test and iterate on working with the VS Code extension, then I'd like to switch gears to working on nimsuggest et al even if it's mostly focused on tests and simple fixes. Right now it feels rather daunting without that.

[dom96]

Thank you for taking the time to write this up @mratsim. A lot of stuff here but I do agree with essentially all of it. Developer efficiency should be a top focus for those working on Nim. Since VS Code is the most popular and has a lot of momentum behind it, the focus should be placed on it to create an example of what a great Nim coding experience looks like.

I see a lot of talk around nimsuggest ending up either spinning the CPU or leaking memory. This is the reason I myself have disabled it in the VS Code extension. I've been saying this for a while now, and will continue to say it until I successfully inspire someone to implement it: let's have a Language Server implementation which executes nimsuggest and manages it as a process, ensuring that it gets restarted when it starts using 100% CPU or more than 500mb of RAM (configurable). We can even detect the project where this happened on and blocklist it automatically (while making the user aware that this is happening). I think this can go a long way to ensuring that people can make use of nimsuggest for the smaller projects they have, while also having a chance to use it on bigger projects too. More than that, I cannot imagine this would be difficult to implement.

There is even lots of room for expansion here, we can even ask the user if they are willing to share the code that nimsuggest becomes unstable on. We could get easily reproducible bug reports that way.

I strongly believe that putting effort into this would be far more impactful than some of the efforts I am seeing for Nim, in particular the implementation of incremental compilation.

[saem]

@dom96

I've been saying this for a while now, and will continue to say it until I successfully inspire someone to implement it: let's have a Language Server implementation which executes nimsuggest and manages it as a process,

I've read this a few times, in fact the initial code for this might have already been written by you or Araq, I ported from Aporia. After even a light bit of debugging I very quickly run into issues with simple things like correlating errors. Was the message from the nimsuggest process informational? Was it in fact an error based on the issued command? Can I trust my results? What's the taxonomy of errors I should care about? Was the error related to the current command's execution or did the user happen to do something to the file system and it's unrelated? Will my crap code continue to work as output formats change and how do I manage to version that? Wait, I can't be the only person who has to build up these structures, is there a shared lib that has all this? This is like working with poorly behaved distributed http service layer with poor interfaces: no schemas or versioning, poor use of status codes, half-implemented distributed tracing, spotty coverage of metrics and health checks, and instead of decent http clients we use selenium to orchestrate it.

All the above would be far better if we could kill threads and used nimsuggest as a library. In the future once the interfaces are a lot more solid consider supporting a newer language server version running with an older suggest version, which processes allow but one can also side by side libraries. Considering the myriad of many to many relations already exist in this: operating systems, nim versions, language clients (think browser quirks), LSP versions, LSP capabilities, data format versions, nimble versions, the myriad of ways someone can intimate a project, ... locking down a number of those variables in the beginning would go a long way.

Bonus issue, the more stdlib someone uses that doesn't have JS support the more of that they need to implement later with nodejs, but I've done enough of that type of stuff that it's not an issue for me beyond one more thing.

[liquidev]

@dom96

let's have a Language Server implementation which executes nimsuggest and manages it as a process, ensuring that it gets restarted when it starts using 100% CPU or more than 500mb of RAM (configurable).

This is merely a band-aid solution. Nimsuggest's biggest problem right now is that it has to recompile the entire project every time a file is modified, an issue that's also present in the Nim compiler itself. Hence why I, and many other people believe that incremental compilation is the right way to go about this. Fixing the issue at its core is key.

[dom96]

@liquid600pgm yes, it is a band-aid solution. But it's a quick and impactful way to make the developer experience better. It also has a chance to give us the necessary data to make nimsuggest better.

I don't see how incremental compilation will help here. As far as I know, nimsuggest can already reparse only the file that has changed. But even if it can't, the problem isn't its speed. It is general stability of nimsuggest. Can incremental compilation help with this?

[auxym]

Thanks a lot for writing this @mratsim, it nicely sums up my feelings too.

I just wanted to added some extra weight to debugging, which wasn't yet brought up by other commenters. I do agree that debugEcho gets old, and even gdb gets old. I recently came across this (HN post) where the author attempts to debug some Zig code and gets frustrated.

It might just be my own style, but I feel that an efficient debugging workflow adds a lot of productivity. I think the best experience I've had so far in that aspect is Groovy/Java in IntelliJ. Features like a nice "watch" window with arbitrary expressions and conditional breakpoints contribute a lot to this.

The (new?) nim-gdb script already gets us part of the way there (unmangled variable names? yay!). Plus, I have yet to try it, but it appears that nim-gdb can be used to debug nim code using VS Code's GUI debugging. So, it does look like we're actually not too far from a nice debugging workflow integrated in VS Code. Adding better support for it in the Nim plugin (so it works "out of the box") would help, as would improving the nim-gdb script to bettter support hovering over variables (as mentioned in the guide linked above).

[saem]

The (new?) nim-gdb script already gets us part of the way there (unmangled variable names? yay!). Plus, I have yet to try it, but it appears that nim-gdb can be used to debug nim code using VS Code's GUI debugging.

Can confirm this works with the nim version of the vs code extension at least. Top level variables in a module continue to be mangled, however. IIRC, an issue at the time was that choosenim doesn't expose the script in a good way, maybe I'm confusing that with something else? It wasn't too hard to work around but another pain point.

I haven't dug into the code or know much about gdb, but I believe it might be possible to remove the Python dependency and that'd be one less issue.

[Araq]

For when one is parsing process output, there should simply be a versioned data models (API, file, and IO formats) and serdes for it. Just keep these around "forever", basically from now on every input/output to a tool should have a name, version, associated data types, and serdes, even if they're no longer supported this allows tooling to use the latest code as a library to work with older versions easily. I'm hoping I've just missed this and perhaps it already exists.

Excuse me if you already know this:

• nimsuggest is backwards compatible to the point that you always should pass the --v2 switch. So that version 2 of the protocol is enabled.
• The output format is stable, table like structure with newlines and tabulators.
• The fact that also errors are reported is most unfortunate but was done because previous contributors demanded it. We should disable it IMO and make error queries separate.
• There is logic inside the compiler that specifies how the "main" Nim file should be found, but you don't have to use this, start the nimsuggest process with a directory and it will figure it out. If it gets it wrong, let's have more documentation and people should fix their projects.
• Nimsuggest has a test suite and we ensure it's green, also bugs do get fixed.

As far as nimsuggest itself is concerned, it does all these things you outline already. (Yes, I agree it doesn't work well, but the basic blocks for a robust tool are all there.)

[Araq]

I don't see how incremental compilation will help here. As far as I know, nimsuggest can already reparse only the file that has changed. But even if it can't, the problem isn't its speed. It is general stability of nimsuggest. Can incremental compilation help with this?

Yes it can. For multiple reasons:

• With IC nimsuggest doesn't even have to be a "server", it can be a simple one-shot command line tool. Much easier to write tests for then.
• As a one-shot command line tool it, it's harder to run into loops consuming 100% of CPU. It's much harder to run into memory leaks.
• With IC bug reports can contain the precompiled files and the nimsuggest command that produced the crash and voila, we can reproduce the bug.
• nimsuggest is multi-threaded and skips files and procs already. All for speed. If you think its speed is fine but its stability isn't, it's because we traded stability for speed. Immediate auto-competions for big systems was really important back then when I still heavily developed nimsuggest.

[disruptek]

Errors could be introduced into the PackedAst as error nodes prior to their support in the rest of the compiler; this would help nimsuggest reveal errors selectively while ticking off a compiler design goal at the same time.

[saem]

• nimsuggest is backwards compatible to the point that you always should pass the --v2 switch. So that version 2 of the protocol is enabled.

I didn't realize version 2 wasn't default and didn't know enough to put it all together when porting the extension, which does pass --v2.

• The output format is stable, table like structure with newlines and tabulators.

I've used both that and EPC, now that I'm digging into the code I can see more of the versioning bits in place. There are some areas for improvement but that can come later when there is enough driving a version 3.

• The fact that also errors are reported is most unfortunate but was done because previous contributors demanded it. We should disable it IMO and make error queries separate.

This is tricky and even though I'm on the receiving end of the input for a while now, with the extension and experiments with the LSP, this isn't super clear cut.

• There is logic inside the compiler that specifies how the "main" Nim file should be found, but you don't have to use this, start the nimsuggest process with a directory and it will figure it out. If it gets it wrong, let's have more documentation and people should fix their projects.

You've pointed out that logic before, the part that wasn't clear before was how much opinion came along with it. If it's as prescriptive as, "here is how it works, sure that might improve over time, but if you're not doing it this way your code is busted." Then that would drastically reduce the complexity.

• Nimsuggest has a test suite and we ensure it's green, also bugs do get fixed.

Yup, I'm studying each one and trying to resurrect the ones that for disabled, see the last five-ish commits: https://github.com/saem/Nim/commits/saem-nimsuggest-118 I'm trying to keep some notes along the way.

As far as nimsuggest itself is concerned, it does all these things you outline already. (Yes, I agree it doesn't work well, but the basic blocks for a robust tool are all there.)

Didn't mean to say it never could, after I posted I decided to go spelunking figured out some of this stuff along the way, still a ways to go.

[alaviss]

@saem If you ever gonna work on improving nimsuggest's protocol, please let me know as I would be very interested :)

[cmacmackin]

On the debugging topic, the current nim-gdb script/Python tools are helpful but not sufficient. After only some brief use, some issues I see:

• Top-level/global symbols are still name-mangled
• Procedure arguments are printed as pointers by default; it would be nice if they were automatically dereferenced
• Perhaps automatically apply the $ operator when pretty-printing? On the other hand, I appreciate there may well be use cases where the rawer representation is needed.
• Alternatively, provide some sort of hook which library-writers can provide to pretty-print their custom objects (this would be especially useful for libraries such as Arraymancer which work with raw pointers in the back-end)
• Often the $ operator is not found (perhaps in instances where it did not get compiled into the executable?)
• Inspecting object variants is cumbersome
• Code inserted by template expansion sometimes shows up when stepping through the program. I'm sure this is useful at times, but it can be confusing at others. Perhaps a switch could be introduced?
• Stepping through the program sometimes steps you into calls inserted by the compiler for low-level functionality
• We can't use overloaded operators in expressions, it seems. Supporting arbitrary overloaded operators probably isn't possible, but it would be good to have the [] operator for accessing contents of containers.
• The Python module does not get imported to GDB when nim-gdb is called from a choosenim installation (dom96/choosenim#247)

I haven't played with some of the more advanced language features like generics, templates, let alone macros, to have a clear understanding of if/how well they work with GDB.

[saem]

On the debugging topic, the current nim-gdb script/Python tools are helpful but not sufficient. After only some brief use, some issues I see:

• Top-level/global symbols are still name-mangled

yes, name mangling is a whole thing and mangling/demangling is non-trivial. Figuring out how to make this work would go a long way. The Nim Debug Information (ndi) files would be easy raw data that the python script could trivially pick up, better still since it's more data driven the script would be doing less work which is the right direction as it should be minimized ideally to the point wherein it disappears. Presently stitching the ndi information together isn't all that easy, but someone needs to have another run at the whole frame printing bit before that becomes easier. In an ideal world much of the script is simply generated by the compiler itself based on the various config as the memory layouts change and the overall thing is quite brittle.

• Procedure arguments are printed as pointers by default; it would be nice if they were automatically dereferenced

I didn't get a chance to get around to this, but that's something that's frustrating, lots of clicking around to open things as I'm debugging the compiler to figure out what's going on.

• Perhaps automatically apply the $ operator when pretty-printing? On the other hand, I appreciate there may well be use cases where the rawer representation is needed.

Unfortunately, I don't believe this is safe, the script does do something like this, IIRC. But I didn't dig into that part all too closely.

• Alternatively, provide some sort of hook which library-writers can provide to pretty-print their custom objects (this would be especially useful for libraries such as Arraymancer which work with raw pointers in the back-end)

This goes back to the whole generate some of this script via the compiler itself.

• Often the $ operator is not found (perhaps in instances where it did not get compiled into the executable?)

That sounds about right.

• Inspecting object variants is cumbersome

Yeah, there is some commented out code for handling those, I didn't think I needed it so I didn't write the tests and re-enable that code when I dug into it, but it shouldn't be too hard. I've gotten used to the PNode/TNode layout, but it's likely worth it.

• Code inserted by template expansion sometimes shows up when stepping through the program. I'm sure this is useful at times, but it can be confusing at others. Perhaps a switch could be introduced?

Honestly I think this should stay, it's disconcerting yes, but it's a good lesson as to what templates really are and that's the price of the abstraction. Maybe I'm misunderstanding you,but I'm not quite sure if this is entirely possible in gdb, though given how programmable it has been so far I wouldn't be surprised.

• Stepping through the program sometimes steps you into calls inserted by the compiler for low-level functionality

Or breakpoints get inserted above/below where you expect them. I think this might be an issue with code generation of debug information. Worth filing a bug report.

• We can't use overloaded operators in expressions, it seems. Supporting arbitrary overloaded operators probably isn't possible, but it would be good to have the [] operator for accessing contents of containers.

I haven't been able to get any of the expression stuff to work whatsoever, so if you have some working cases I'd be curious to see them. The question is which [] to expose if you have more than one, also what if it doesn't get compiled in? 😬

• The Python module does not get imported to GDB when nim-gdb is called from a choosenim installation (dom96/choosenim#247)

Yeah, this is rather unfortunate and I don't know why it ships broken by default, not to mention a bunch of other missing tools. But I don't want to touch it.

I haven't played with some of the more advanced language features like generics, templates, let alone macros, to have a clear understanding of if/how well they work with GDB.

Templates and macros barf out code, there isn't much to it. So you do get some fun stuff, for example, in the compiler there is a template roughly as follows template config*(c: ConfigRef) = c.graph.config, at which point the hovers/inspect all fall apart. So you have to remember tidbits like that so you can traverse the path manually.

@cmacmackin if you do want to improve the script and it's just the usual getting started hurdles, LMK and I can give you a hand (I'm on IRC).

[cmacmackin]

On the debugging topic, the current nim-gdb script/Python tools are helpful but not sufficient. After only some brief use, some issues I see:

• Top-level/global symbols are still name-mangled

yes, name mangling is a whole thing and mangling/demangling is non-trivial. Figuring out how to make this work would go a long way. The Nim Debug Information (ndi) files would be easy raw data that the python script could trivially pick up, better still since it's more data driven the script would be doing less work which is the right direction as it should be minimized ideally to the point wherein it disappears. Presently stitching the ndi information together isn't all that easy, but someone needs to have another run at the whole frame printing bit before that becomes easier. In an ideal world much of the script is simply generated by the compiler itself based on the various config as the memory layouts change and the overall thing is quite brittle.

The ideal situation would be to get the compiler to write DWARF debug information itself, although I don't think gcc supports injecting your own anyway, so that wouldn't be feasible. But getting it to output that information somewhere, whether in a separate file or embedded in the binary, would be useful. Of course, that would require us to change the compiler itself, which sounds intimidating.

• Procedure arguments are printed as pointers by default; it would be nice if they were automatically dereferenced

I didn't get a chance to get around to this, but that's something that's frustrating, lots of clicking around to open things as I'm debugging the compiler to figure out what's going on.

I took a look at the GDB Python documentation and this looks like it should be fairly straightforward.

• Perhaps automatically apply the $ operator when pretty-printing? On the other hand, I appreciate there may well be use cases where the rawer representation is needed.

Unfortunately, I don't believe this is safe, the script does do something like this, IIRC. But I didn't dig into that part all too closely.

• Alternatively, provide some sort of hook which library-writers can provide to pretty-print their custom objects (this would be especially useful for libraries such as Arraymancer which work with raw pointers in the back-end) This goes back to the whole generate some of this script via the compiler itself.

While I can see advantages to that, I don't think it would be necessary for this feature. What we could do is have users define a $debug operator, with the compiler providing a default that falls back to the $ operator. This would be a bit like how Python has both str and repr functions. The compiler would add __attribute__((used)) to the $debug procedures if building with the --debugger:native flag, to ensure it gets included in the final binary. Otherwise it would optimise them out. The pretty-printers could then just defer to the $debug functions and leave the print logic outside of Python.

• Often the $ operator is not found (perhaps in instances where it did not get compiled into the executable?)

That sounds about right.

As above, we could get the compiler to insert __attribute__((used)) if building with the --debugger:native flag,

• Inspecting object variants is cumbersome

Yeah, there is some commented out code for handling those, I didn't think I needed it so I didn't write the tests and re-enable that code when I dug into it, but it shouldn't be too hard. I've gotten used to the PNode/TNode layout, but it's likely worth it.

The above approach with $debug would give us this for free.

• Code inserted by template expansion sometimes shows up when stepping through the program. I'm sure this is useful at times, but it can be confusing at others. Perhaps a switch could be introduced?

Honestly I think this should stay, it's disconcerting yes, but it's a good lesson as to what templates really are and that's the price of the abstraction. Maybe I'm misunderstanding you,but I'm not quite sure if this is entirely possible in gdb, though given how programmable it has been so far I wouldn't be surprised.

Actually, this bothers me more when it happens with iterators than with templates. I think there's less justification for showing that. Ideally we could play with the stack a bit to make it appear like a function call.

Not sure whether showing templates or not is something which could be controlled from the debugger, but presumably a compile-time flag could alter the #line statements in the C code to control that.

• Stepping through the program sometimes steps you into calls inserted by the compiler for low-level functionality

Or breakpoints get inserted above/below where you expect them. I think this might be an issue with code generation of debug information. Worth filing a bug report.

I'll see if I can come up with a minimal reproducer at some point.

• We can't use overloaded operators in expressions, it seems. Supporting arbitrary overloaded operators probably isn't possible, but it would be good to have the [] operator for accessing contents of containers.

I haven't been able to get any of the expression stuff to work whatsoever, so if you have some working cases I'd be curious to see them. The question is which [] to expose if you have more than one, also what if it doesn't get compiled in? grimacing

I don't have experience with programming GDB so I can't really comment on how doable it would be, but I think it is worth looking at. Perhaps would could do something with "xmethods"? That would definitely require the compiler to generate the script though. I'd suggest exposing all available [] operators and forcing them all to be compiled in, as above. Adding in the basic math operators might be useful too.

• The Python module does not get imported to GDB when nim-gdb is called from a choosenim installation (dom96/choosenim#247)

Yeah, this is rather unfortunate and I don't know why it ships broken by default, not to mention a bunch of other missing tools. But I don't want to touch it.

Shouldn't be too hard to fix. We'd just need to have the nim-gdb script find the Python file relative to its location, rather than relative to which nim.

I haven't played with some of the more advanced language features like generics, templates, let alone macros, to have a clear understanding of if/how well they work with GDB.

Templates and macros barf out code, there isn't much to it. So you do get some fun stuff, for example, in the compiler there is a template roughly as follows template config*(c: ConfigRef) = c.graph.config, at which point the hovers/inspect all fall apart. So you have to remember tidbits like that so you can traverse the path manually.

@cmacmackin if you do want to improve the script and it's just the usual getting started hurdles, LMK and I can give you a hand (I'm on IRC).

I have a couple of small projects I'd need to complete first, but I would be interested in helping out with this. I can certainly help with the Python script. Changes to the compiler would be more challenging but I could probably give it a stab. What I'd like to avoid is having to contribute to GDB itself, which seems to be required to give first-class support to languages such as Rust. (I don't think GDB even accepts random pull requests, so it probably wouldn't be a possibility even if I were willing to brave the millions of lines of C.)

EDIT: I just took a look at the LLDB docs and it seems to have the goal that new language support can be added as a plugin, although it isn't there yet. Some projects do still seem to use it to add a debugger for their system, such as llnode for example. Perhaps this would ultimately provide a better debugging experience? Further research needed.

[cmacmackin]

After some reading I've concluded that to give truly first class debugger support for Nim we'd probably want to use LLDB. It doesn't look like the easiest thing to work with (API documentation isn't great) but it is clearly very extensible. See this discussion of adding Rust support. It looks like we could potentially wrap Nimscript to provide some basic expression evaluation (tutorials exist for embedding it in a Nim binary and that approach could be adapted.). We could also likely leverage some (many?) bits of the existing compiler.

Also note vadimcn/vscode-lldb#91, although nothing seems to have come from that.

[saem]

After some reading I've concluded that to give truly first class debugger support for Nim we'd probably want to use LLDB. It doesn't look like the easiest thing to work with (API documentation isn't great) but it is clearly very extensible. See this discussion of adding Rust support. It looks like we could potentially wrap Nimscript to provide some basic expression evaluation (tutorials exist for embedding it in a Nim binary and that approach could be adapted.). We could also likely leverage some (many?) bits of the existing compiler.

Also note vadimcn/vscode-lldb#91, although nothing seems to have come from that.

Welcome, you've almost retraced all my foot steps. 🤭

Even for the GDB stuff one could wrap the MI or use a tiny shim in Python, Guile, and the other language they support, as long as it's data driven it should be fine. Cleaning up the frames and leveraging the existing NDI would go far. As NDI seems to be barely used adding extra data or making bigger changes will likely be a non-issue.

I've read most of the relevant source code for that extension when it comes to rust support, including the debug adapter, etc... it would be impossible to port those boys over. Relatedly, LLDB has the Python Scripting Bridge and thankfully the bridge also has a C++ version. I've not looked too deeply at going down that binding road but that would be the first thing in order to start playing with it. Changes to the complier might not necessarily have to be to the main code base, but could start with a tool that lives side by side and needs a separate invocation to begin with. A clear deprecation policy stating it'll eventually get replaced means no real promises are made out broken. Depends how much program specific information is needed in the beginning and how far one can go by simply keeping up with relevant conventions and memory layouts (at least in the short term).

[EchoPouet]

I agree with the remarks of @mratsim. Now that Nim have a strong core, it's time to develop or improve tools to improve the language accessibility. Sometime, I have the impression that Nim is for geek or passionate developer only and not very user-friendly. It's a regrettable, and It's maybe not the objective.

Firstly, it's not a tool but proposing a native installer for Windows (exe or msi) and Mac (dmg) must be a minimum. Nim is available on Scoop but not everybody uses it. There is a Chocolatey package for Nim but out of date (https://chocolatey.org/packages/nim) for long time, maybe discuss with the maintainers to request the control. For Mac, it seems not very official when you read the website.

For tools, the priority order should be: 1 - Support officially an IDE. The @saem Visual Studio extension is the best choice because it writes in Nim, and it is the one that works best. "Officially", this means that the website talk about this plugin and Nim team work with @saem to improve plugin and fix bugs. 2 - A good multiplatform/multicompiler debugger. It's a big work but necessary to develop big projects.

These first two actions would be sufficient for this year.

As a bonus, there are these tools: 3 - A memory analyzer to track memory leak or consumption. Valgind for Nim by example but multiplatform. 4 - Code coverage analyzer.

When all these actions will be finished, a developer is ready to create a big project with Nim without difficulty.

Like @mratsim said I think working on developer tooling should be one of 2021 focus, and I totally agree with him.

[Araq]

Firstly, it's not a tool but proposing a native installer for Windows (exe or msi) and Mac (dmg) must be a minimum.

Most people who have a hard time with the existing installation would be equally lost after a successful installation -- the compiler is a command line tool. There is also no indication from our current Nim survey that installing Nim is too hard...

[EchoPouet]

Most people who have a hard time with the existing installation would be equally lost after a successful installation -- the compiler is a command line tool. There is also no indication from our current Nim survey that installing Nim is too hard...

I'm not saying it's difficult, fortunately a developer knows how to extract an archive and add a path in the PATH env. I'm just saying that it's a minimum for any software, even a compiler. Besides, there is already code to do this in the repository. If it's a problem of time I can take care of it for Windows.

As for the survey results, the people who answer it are already informed users. They didn't just make a hello world :) Besides, it's not the first thing you think of when you talk about a compiler.

[elliotwaite]

I believe @piotrtomiak is working on the JetBrains Nim plugin, so I just wanted to include him in this thread in case he had any suggestions for what could be improved that would help him in his efforts.

[piotrtomiak]

@elliotwaite Thanks for including me in the thread!

The Nim plugin for IntelliJ platform is actually an independent solution for code completion and suggestions based on the IntelliJ language support. There are various advantages of the platform itself, most notably built-in indexes (allowing for auto-import for example). For instance, you can already check, from which import a particular symbol comes from. It seems that nimsuggest would be useful only in case of macros.

As far as LSP approach is concerned. The LSP is pretty cool if you are focused on a single language, but if you start to mix languages it gets really tough. Let's take emerald templates for example: with IJ platform it is easy to support navigation/code completion/refactoring within expressions in HTML templates and related Nim files. I know for the fact, that there is a lot of struggle to do so with LSPs, even with a very popular web frameworks like Vue or Angular and TypeScript LSP.

I think that it might be worth considering the IntelliJ platform (which itself is free and open-source) as one of the basic platforms, as it will allow for the most sophisticated support. One thing though is that some of the existing support, which is not part of the open-source platform, like JavaScript, C/C++ or debugger, requires commercial license for the IDE. However, all the community driven efforts would be available for everyone for free. The Nim plugin itself is currently being developed only by me, it's free, but closed source. I would need to discuss that with management, but I think that if there is a community will to choose IJ as one of base platforms we can open-source the plugin and accept community contributions. The plugin is written entirely in Kotlin.

[elliotwaite]

In regards to the JetBrains Nim plugin, I've been using it with PyCharm since that's what I'm used to using for Python and I've been really enjoying it, especially with the recent updates. With Python users often ranking PyCharm as #‎1 or #‎2 for Python development, and since I can see a lot of future Nim users coming over from Python given Nim's familiar syntax and better performance, I think having strong Nim support in IntelliJ/PyCharm would be a good strategy for future growth. I know that in Nim's recent survey IntelliJ only ranked #‎6 in popularity amongst current Nim users, but as Nim's libraries mature, especially the data science ones, I could see there being a lot more Python users looking into Nim. And if they could use their familiar IDE with strong Nim support, that might be a key deciding factor in whether they continue to use the language.

[mratsim]

Important first experience with Nim tooling and the expectations new joiners have. Context not finding -d:release switch https://forum.nim-lang.org/t/7426#47148

So, retracing my steps, I think there are a few confusion points for newcomers with experience in other relatively "new" languages. In my case, coming from Go, then Rust, then Crystal, I am used to the language having a primary tool that handles all things related to the build, run, test, benchmark process. In go its the go command itself, in Rust it's cargo, and in Crystal a combination of the crystal command and shards. With this mental baggage, I looked for the same in Nim's case, and found nimble. I set up a project with nimble init and used nimble build to compile it. The output says nothing about debug or release builds:

 Verifying dependencies for fib@0.1.0
   Building fib/fib using c backend

Running nimble --help doesn't mention any release flags either, but does point to their documentation page, which does say that nimble build produces a debug build and to build in release mode use nimble install.

As for the nim command itself; as stated above, the fact that a debug build is the default is a little buried in that last Hint line. nim --help only mentions the use of -d:release as a note on the --opt flag:

--opt:none|speed|size     optimize not at all or for speed|size
                            Note: use -d:release for a release build!

which in my opinion is easy to overlook and kind of lessens its importance; almost mentioned like an afterthought. What I would have expected was a --release flag similar to Rust and Crystal. Although a debug build as default is the norm in most languages I've used, the existence of a --release flag is common too.

Although I suspect this type of experience is greatly subjective, depending on mental baggage from previous languages as in my case, I hope it helps in some way to improve usability and reduce future confusion.

I greatly appreciate all the discussion in this thread. I've learned quite a bit and that's a really important part of a programming language! Believe me, I've been in quite a few language forums where "hostile environment" for newcomers is sadly the norm. I'm glad that's not the case here. :-)

[elliotwaite]

@piotrtomiak

I would need to discuss that with management, but I think that if there is a community will to choose IJ as one of base platforms we can open-source the plugin and accept community contributions.

I think that would be a great idea. Open sourcing the plugin seems like a win/win for both JetBrains and the Nim community. It would allow the Nim community to more directly help you with the plugin, hopefully accelerating its development, and the better the plugin is, the more likely it will be that Nim users will choose to use JetBrains IDEs (both their free versions and their paid versions). I've already seen some Nim users asking how they can participate in its development, and there will likely be more in the future.

[alaviss]

Something we should improve for package makers too is the document generator. Currently:

• There are no built-in way to generate documents for all modules in the package (nim doc --project only follows the import graph, and if your package is consisted of multiple separated modules, that wouldn't work).
• There are no auto generator for a summary of a package like the stdlib's lib.rst: https://nim-lang.org/docs/lib.html. And the "index" is horrible for discovery.

Fixing these would be very helpful for people building packages.

[ShadowElf37]

I came to Nim from Python recently and I've been using Pycharm forever. The plugin for Jetbrains just really needs help. Frequently and arbitrarily misses syntax highlighting, especially on templates and macros. Weak highlighting support for Nim's syntax in general. The autocompletion is terrible – it gets variable names and sometimes imports, but even when it does, it'll try to autocomplete in weird places, like when I'm typing a number. Legitimately, I type let a = 5, and then when I press enter it autocompletes a in front of the 5 instead of going to a new line. It's not in a usable state, and I agree that it would attract a lot of Python users if they could just throw in the plugin and have it work well.