no more projectfile.nim as argument

[krux02]

Passing the correct projectfile.nim to nimsuggest in a non-trivial task that every editor that wants to support nim has to implement redundantly, and often leads to the situation that a project can be edited correctly in one editor but in other editors the tooling will not work out of the box. So my suggestion is, that nimsuggest should do it on it's own. Setting the projectfile should not be an editor setting anymore.

My suggestion of how this could work is the following:

When the nimsuggest server is started, it doesn't do anything yet. As soon as a file "/path/to/some/random/file.nim" should be checked, nimsuggest looks for the file "nimsuggest_project_file.txt" in that order in the following locations:

/path/to/some/random/nimsuggest_project_file.txt
/path/to/some/nimsuggest_project_file.txt
/path/to/nimsuggest_project_file.txt
/path/nimsuggest_project_file.txt
/nimsuggest_project_file.txt

The first nimsuggest_project_file.txt that is found should contain a single line with a relative path to the projectfile. The path is relative to the file it is written in. That projectfile should be used as the nimsuggest project file. If no such file could be found, then the file that should be checked will be used as it's own project file.

[Araq]

We already have a notion of how to mark project files, it's the foo.nim file which has a foo.nims or foo.nim.cfg file next to it.

When the nimsuggest server is started, it doesn't do anything yet.

Hard to implement and no IDE works in this way either. You open a "solution" / "project" with an IDE so that it can understand the directory layout. Why oh why is that so hard to pass to nimsuggest?

[krux02]

Because I use emacs and emacs does not have "solution" / "project". Yes emacs sucks, but I don't want to discuss that here, I just try to find a solution that works for any editor.

Emacs can only open files, and files are always opened in the folder the file is located, not relative to a project rood, or in the folder the emacs process runs. There is a package (non emacs people would call it a plugin) "projectile" that adds some concept of a project. But it does not change the way emacs opens files. it is mostly for project navigation. It does not provide something like a project "solution".

I wanted to use directory local variables to set my nim project file, but it turns out that it doesn't work that way. Variables can only be set either globally or buffer local (file local). The concept of directory variables are simply set in each file locally that is in a subdirectory of the .dir-locals.el. Because each file is opened in it's own directory, it simply means that paths in .dir-locals are relative to the opened file, not the .dir-locals.el file. Searching upwards the directory tree for the next .dir-locals.el file to make the paths absolute is incorrect. First of all I don't know if the variable has been sit via .dir-locals in the first place, nor do I know that when I find a .dir-locals.el that this file is the file where the path I have is actually set.

The only solution I have that would reliably work in emacs is the logic I just suggested above. But when I implement that in emacs, then this logic could also just happen in nimsuggest itself and therefore would reliably work in any editor, not just emacs.

[PMunch]

This is an issue for the language server protocol as well. As it is intended to work with IDEs and simple text editors alike it works on a file basis, so I have to write some logic to find the project file. And the definition "it's the file with a .nims or .nim.cfg file next to it" doesn't quite cut it either as there are many projects without such a file. It should possibly at least have a ".nimble" rule as well.

[alaviss]

The better option would be to remove the notion of project files from nimsuggest. Instead one instance can be used for as many projects as needed. This removes the need of plugin authors to invent ways to keep track of which instance can be used for which projects. However, given how the compiler is structured, I have doubt that this would be easy.

[saem]

Removing the project file requirement means that effectively on every call either an project file will be required or the file within which the query is being executed needs to be considered the project file -- that's totally fine IMO.

But the internals are likely to change quite a bit as nimsuggest would effectively have compliations per project (not just nim file but also the combination of defines, backend, and maybe more parameters) and query against that. This might be totally workable, but it does mean a fairly significant internal change.

A simplifying assumption would be to change the project behaviour to the following:

1. straight invocation returns the current project as it does today
2. invoking it with a nim file + current working directory (or assume the project's dir path) will configure/reconfigure subsequent queries to resolve in that context (see aside below for passing a cfg)

An approach I'd recommend using the above assumption:

1. create a nimsuggest2 (working title) as a copy of nimsuggest as this thing will likely change in many more ways
2. change the CLI to only handle a few nimsuggest params like --stdin, --port, --tester, --debug, ...
3. try having the project set mimic what the current handleCmdLine does
4. learn some stuff and see what to do next

As an aside it would be incredibly useful, if there was an "invocation cfg" (priority just below CLI params) allowing IDEs or other tools to specify any config overrides as part of setting the project above and in many other contexts. The cfg param would be entirely optional and only ever looked up at the specified file path foo/bar/xyz.cfg. This would allow tools to no only control the cfg themselves, but for tools to work with each other cohesively as it can be used as a standard compiler config interchange format. Specifically it means they can do this without polluting the source tree, overwriting/conflicting with repo defined cfgs, overwriting/conflicting with general user/system settings, allowing the storing and managing many such profiles, without trying to do this with raw strings of CLI params or yet another format that is effectively compiler cfgs, and the list goes on. :)

[Araq]

How does such a nimsugest ever support include files? How does it ever figure out whether it's "nim js" or "nim c"? How does it know about the --path? project files are important to keep around... Which is why every IDE I'm aware of has them... I don't care if "emacs" has trouble supporting this idea, so use an editor that isn't stuck in 1980.

[alaviss]

What we really want is auto-discovery. Plugin authors should not have to cook their own way to figure out which file corresponds to which project, or how to maintain a list of nimsuggest instances.

What we want is that nimsuggest automatically decides what project does the file we pass to it belongs to and do the right thing. findProjectNimFile is already a solid foundation for such a auto discovery system. nimlsp already do something similar afaik, so it is possible.

[saem]

project files are important to keep around

I never said they're going away, just that you can swap between them at run time without having to fire up a new process.

What we really want is auto-discovery.

No, no thank you. Editors like vim and emacs and other text editors that aren't IDEs, what you're suggesting ends up in headless IDE territory, no thanks.

[Araq]

No, no thank you. Editors like vim and emacs and other text editors that aren't IDEs, what you're suggesting ends up in headless IDE territory, no thanks.

I don't understand. We already have auto-discovery via findProjectNimFile and it works better than what we had before, IMHO.

[saem]

I don't understand. We already have auto-discovery via findProjectNimFile and it works better than what we had before, IMHO.

It's a heuristic to find the most project-y file given a starting point. We can have more projects conform so the heuristic is correct. However I specifically need to not only specific the project file but also the config as the two combined are what's specify all the projects at the static file system level. So that's not counting any parameters acting as build profiles which create yet further permutations to manage.

Fundamentally I still have to write a project discovery algorithm that enumerates all possible projects given some root. It has to work across Nim versions, at least 1.0 and devel. I need a way to switch between those projects and have all the tooling follow along. I need a way for the user or tool to manage different circumstances (build profiles). I need a way to not just strictly know them as a collection of strings but have something slightly more structured so things like, run the test associated to this module for this project are possible, then do it again but this time debug the test, ask the whole perhaps collecting and updating coverage information. It's a lot of management and I'm not begrudging editors that take a more simplistic approach but if we're aiming for a high bar:

• I'd like to avoid relying too much on a very robust cross-platform process manager (this is a fictional thing) and use this very sparingly
• if I have to fire up a new processes every time rather than be able to change the project+config, not only do I need to manage that many processes' lifetimes but also ask the granular reasons as to why they might need to be created, restarted, or stopped, as it's not just a process it's also the baked in config state.
• the whole stateful project+config as part of process launch and is thereafter immutable means the forced coupling of that state to process lifetime, it's creating a problem that never needed to exist in the first place
• If I can simply specify the project and config then I don't have to think as hard about should this process still be around? Do I need to send mod to this one? Did I track the cfgs this project started with correctly and did those change, because I need to restart it (lol, wut)
• nimsuggest and other tools take on slightly more variable state as complexity, but all tools including IDEs and editors (bigger chain of systems working together) are now overall less stateful

[alaviss]

While findProjectNimFile works pretty well with a standard nimble layout, it doesn't work at all when a collection of modules is concerned, for example the stdlib, since you cannot know whether a file is an include file or whether it is the main module file. And as @saem pointed out, the coupling of "project" with an instance can be limiting.

Now this will be in fictional territory, but here is what I imagined nimsuggest could do:

• Accept commands to any file and treat them a main module. Heuristics provided by findProjectNimFile can be used to see if the file is a part of a bigger project which should be one level up in the directory hierarchy, or is pin pointed by the existence of a nimble file. Detecting for possible projects via existence of configuration is very error prone (the stdlib is a prime example, the detection will always favor asyncdispatch as the main module since there's a .nim.cfg file for it). That data can be used to classify includes vs modules with higher confidence.
• Analyze the connections between files submitted, and classify them as includes vs modules. Configurations are employed on the modules as if they are being compiled by the compiler as the main file.
• Allow editors to provide context associated with a command, which can be employed by nimsuggest for better analysis:
  • If the editor submit a workspace, classify all standalone modules outside the workspace as modules to be imported. Additionally scope configuration to the workspace, and drop all configuration used to process projects outside of the workspace.
  • If a project file is submitted, only configuration used for the project file will be used, and modules other than the project file will be classified as "modules to be imported".

With IC this idea might be "slightly possible", but I haven't finalized the details.

[Araq]

Accept commands to any file and treat them a main module.

Already implemented.

Analyze the connections between files submitted, and classify them as includes vs modules

Already implemented.

Allow editors to provide context associated with a command,...

Already implemented.

[alaviss]

Now we need to do all that in one nimsuggest instance that's shared between all projects :)

[Araq]

No, that's all implemented... Editors don't use it so there are probably bugs left, chicken&egg problem. But every single enhancement you had in mind is in the compiler already.

[Araq]

Fundamentally I still have to write a project discovery algorithm that enumerates all possible projects given some root. It has to work across Nim versions, at least 1.0 and devel. I need a way to switch between those projects and have all the tooling follow along.

Maybe, or maybe not. I wouldn't mind the opportunity to clean up Nim's config system mess and tell users how to setup things so that nimsuggest works and what is unsupported and about to be deprecated. I'm always after good tradeoffs: Don't break somebody's build just because you wanted to remove 100 lines from the compiler. But if these 100 lines don't work with nimsuggest or are unreasonably costly to maintain, I'm all for cleaning up things. Users (myself included) are forgiving about breakages if the benefits are clearly communicated.

[alaviss]

No, that's all implemented... Editors don't use it so there are probably bugs left, chicken&egg problem. But every single enhancement you had in mind is in the compiler already.

Cool, so how do I use it? I don't see any documentation showing a magical flag to let us ignore the projectfile.nim argument so we can do one global instance of nimsuggest for everything.

I would love to throw out the mess I wrote to manage multiple instances.

[Araq]

Iirc, you use it by starting nimsuggest with a directory.

[alaviss]

That would still scope nimsuggest to one project, whichever it found in that directory.

If you try it somewhere that doesn't have a .nim:

$ nimsuggest /
cannot find file: /

[Araq]

Well, the idea is that it detects the project based on the directory. Files known to the project (including include files) are then handled precisely, but you can also give it unrelated .nim files and then it tries its best, obviously failing if it's an include file.

If you have better ideas, I'm all ears, but there simply is no way around the fact that you need project information.

[alaviss]

I want it to be optional. The project data can be supplied on-demand by editors as additional context when they send a command to nimsuggest (we will need a v3 interface for this).

And here's the thing: processes are expensive in editors plugins. Monitoring them is a pain, starting them is a pain, knowing when to stop them is a pain. Ideally we don't want to do it at all.

[Araq]

I always envisioned that the editor starts nimsuggest with the project's directory and the editor has a notion of "Project". One process per project is perfectly fine. But IMO we write "nimsuggest -v3" on top of IC. Then it's a "one shot" process invokation, no process management required.

[alaviss]

Then it's a "one shot" process invokation, no process management required.

On platforms like Windows, spawning a process can be a huge bottleneck, so I think a server is still a good thing to have, as long as I only ever need one instance of it.

I always envisioned that the editor starts nimsuggest with the project's directory and the editor has a notion of "Project"

The notion is:

• Not portable between editors
• Not the same between users: Most users would open Nim folder as their workspace and not Nim/lib or Nim/compiler
• Optional for most editors (even VSCode, you don't get a workspace unless you use "Open Folder" the last time I checked).
• Not always exist, for example: the user open a random file to edit, say opening Nim/compiler/suggest.nim to add some debugging code, then the editor don't know what is the project here.

[Araq]

On platforms like Windows, spawning a process can be a huge bottleneck, so I think a server is still a good thing to have, as long as I only ever need one instance of it.

Without measurements that is an empty statement. Process creation was slow in 1990, yes. 30 years later and it's "slow" but unlikely to be noticable.

The notion is: ...

So? Most users are willing to adapt their workflows minimally when communicated clearly. I don't expect to have auto-completion when I open random files.

[alaviss]

Without measurements that is an empty statement. Process creation was slow in 1990, yes. 30 years later and it's "slow" but unlikely to be noticable.

Then I guess all this depends on is whether IC is fast enough. But how would such nimsuggest deal with concurrent calls? Does the IC have lock files to prevent corruption due to multiple calls causing re-compilation in parallel?

And the structure of the current cache rely on only the module name I believe? Wouldn't that hinders the performance if the user develop on two separate projects that happen to have modules with the same name?

Most users are willing to adapt their workflows minimally when communicated clearly.

What is this clear message when there is no standard shared between one editor plugin and another? When people edit code they don't follow a set "procedure to open file" (trust me, I tried that with nim.nvim and it was a source of complaint), they just open the file and edit it. Consequentially, that's everything the editor (and its plugins) will ever know. Sure, we can do magical "auto-discovery", but why should we force editors to re-implement the same stuff over and over again?

How most languages solve this problem is simple: via the use of special configuration targeting the IDE tool, telling it what the project is instead of the tool trying to "guess", and editors don't have to worry about it. For projects without a configuration the tool just compile the file and provide whatever it can, it would not be "perfect" then but it would be better than nothing.

Since the configuration is for the tool itself, every user of the tool via the editor of their choice get the same experience. And again, the editor don't have to care about any of this. They just ask the tool and the tool answers.

See this for how TypeScript's language server does it. Similar architecture is found in Rust's language server too as far as I know.

I don't expect to have auto-completion when I open random files.

Maybe we should raise our bar a bit to be as good as others in this aspect.

[Araq]

You lost me. When we auto-detect the project via .nimble and nim.cfg files it's wrong, when we don't auto-detect it it's wrong, when TypeScript does it effectively in the same way we do it, it's suddenly good. So please tell use precisely how you think it should be done.

[alaviss]

TypeScript's tsconfig.json specify the project files, so the ide tool wouldn't have to guess. We can introduce a similar concept to nim.cfg or nimsuggest.cfg as well, then drop non-nimble detection from findProjectNimFile, since inferring from .nim.cfg is error prone, but .nimble inference seem to work well enough.

More importantly, it does all this without needing to specify a folder/file as parameter to the tool. You need exactly one instance and that instance will handle any file/project that you throw at it, which is what I am trying to propose.