Request for hsdev light

[rikvdkleij]

Hereby my proposal.

Hsdev light API

Commands

Project files

Function	Argument
Names of all modules which can be imported	target or file path
All top-level declarations	file path
All exported top-level declarations	file path
Info about identifier	location
Type of identifier	location
Type of selection	from location and to location
Location and module name of where identifier is defined	location
Check file	file path with ghc options and always clear

Libraries

Function	Argument
All exported top-level declarations	module name
Info about identifier	module name and name of identifier

If you can offer more (detailed) info for library modules and identifiers it would be great!

location is file path, row and column. target is stack target, stack ide targets

Remarks

1. I assume the scan command always has to be run after starting hsdev server but it would do only the minimal work to get the commands working.
2. Preloading or caching inside hsdev is not necessary.
3. It has to be clear when client knows when to stop reading output from hsdev.
4. The way how hsdev currently works with starting server, opening socket connection and request/respond json objects is okay.
5. It would be nice if asking for info and definition locations also works inside import module declarations.

Questions:

1. Is it possible to run multiple hsdev processes on different ports? One for each IntelliJ project.
2. Are multi-package projects supported? If so, how does scan have to be executed? Only in project root or for each package?
3. About scan command. What do flags cabal and infer mean? How does scan look like for cabal projects?

[mvoidex]

Remarks:

2. What do you mean by caching? Does hsdev use it now? It uses sqlite as primary storage, not cache.
3. It works

Questions:

1. I think yes. While experimenting I'm often hold 2-3 hsdev processes.
2. Probably not, as I haven't used such projects. Can you post some rtfm links? Is it stack's or cabal's feature?
3. infer doesn't affect installed modules. In fact this flag now disabled, you can infer types for files or projects by separate command infer. Inferring all types for all source modules is too slow, if you have several projects scanned.

[rikvdkleij]

Remarks:

What do you mean by caching? Does hsdev use it now? It uses sqlite as primary storage, not cache.

I mean, get info by demand and do not everything upfront. For example, I noticed for example with Hledge project (https://github.com/simonmichael/hledger) that it can take very long time before scan is finished. And after that, the server did not always respond to client commands.

Questions

2. It's a cabal and stack feature. For stack see https://docs.haskellstack.org/en/stable/GUIDE/#multi-package-projects. It are multiple cabal projects with one stack.yaml or cabal.project For example: Hledger (https://github.com/simonmichael/hledger) , Servant(https://github.com/haskell-servant/servant)
3. So infer is now disabled. What does cabal option do?

[mvoidex]

It took 13 seconds to scan hledger with current version (not pushed yet). SQLite defaults not good for inserting many rows. Another source of slowdown may be asking stack path each time. Fixed too.

3. Asks hsdev to scan installed modules in global-db and user-db

[rikvdkleij]

It took 13 seconds to scan hledger with current version (not pushed yet).

Did you first build the complete project? In which directory did you run the scan command? In my case it was running for minutes with 4 GHC processes at 100% CPU. I also noticed that the hsdev process can about 1,5 - 2 GB large. That would not be not a big problem, at least if it does not keep growing while using hsdev.

[mvoidex]

No, i haven't built it. I've ran:

hsdev start
hsdev scan --cabal
hsdev scan --path hledger

[rikvdkleij]

In which directory? Do you not have to add the sandbox path?

[mvoidex]

Ok, got it. There was a problem - hsdev called stack build --only-configure, but it builds all dependencies, which can last long. I've removed such implicit building.

There also was another bug: Cabal-2.0 changed namings of package-db in sandbox, now they use OS 'windows' for System.Info.os == "mingw32" too, and hsdev can't find it. Fixed both.

After cabal sandbox init and installing these packages I started hsdev used commands in project root (of git repo)

hledger§ hsdev start
hledger§ hsdev scan --sandbox . --project hledger-lib --project hledger --project hledger-ui --project hledger-web --project hledger-api

Took ~30 secs

You can use just hsdev scan --path . command, which will try find everything under current directory, but it also can some irrelevants files/projects. In this case it also found project hakyll-std under site\hakyll-std directory

Another problem is that hsdev detects this project as stack project, but it can be built with both cabal too. I think there should be command 'scan project' with flags for different options - whether to cabal or stack? Scan sandbox too?

[rikvdkleij]

Great!

Yes, I think it's good idea to denote if it's a stack or cabal project because there "sandbox" or "working" directory has a different structure. Btw, I have not used cabal for a long time since I switched to stack. But cabal 2 should be a big improvement over previous versions of cabal (with cabal I mean cabal-install).

Can you explain me the idea of various arguments of the scan command? Also in context of stack vs cabal project.

[mvoidex]

Have you read API.md#scan?

scan is command to inspect both installed modules and source files, it accepts flags to specify scan targets:

• project (flag can be repeated) — scan projects; it scans only sources and doesn't scan global-db/user-db or any other installed modules (*)
• cabal — unclear flag name, it's for scanning global-db/user-db
• sandbox (repeated) — scan installed modules in sandbox (cabal-sandbox or stack-work), it scans package-dbs for specified sandbox
• file (repeated) — scan standalone (non-project) file, it also doesn't scan any dependencies
• path (repeated) — scan anything within directory (projects, sandboxes, standalone files)

Normally you scan project(s) and dependent cabal/sandboxes.

(*) But it "detects" build tool (looking for .cabal-sandbox and stack.yaml) and expect corresponding installed modules as dependencies. As a result, if it "detects" stack, it won't use modules from cabal-sandbox, even if you scanned only cabal-sandbox and no stack-work.

For now you can't specify which build tool to use, there issue #77 for it.

[rikvdkleij]

Have you read API.md#scan?

Yes, but was not clear, for example difference between project and path.

Okay, so if I want to scan a Stack project, the most simple solution is to scan the whole project inside the project root is: scan --path . --cabal

[mvoidex]

Not sure about --cabal. It will scan user-db, which may not be related to project. I'd prefer

hsdev scan --project . --sandbox .

--project to scan project sources, and --sandbox to scan stack package-dbs (it will scan not only local package-db, but all related)

[rikvdkleij]

I wonder if you have time to work on this request. Otherwise I will look for an alternative solution. Same for issues #77, #78 and #74.

[mvoidex]

Hi! I indend to work on this.

77 i have a branch which supports specifying build tool when scanning project with scan project command (flags --cabal or --stack), will merge it soon

78 doesn't look hard to implement, i'll try to add benchmarks next week.

74 needs some investigation

Let's continue discussing details on gitter

[rikvdkleij]

Thanks!

Do you some specific issues to discuss on gitter?

When you have an initial light version developed I will have to integrate it with my plugin so I can test it.

[mvoidex]

Do you some specific issues to discuss on gitter?

Well, i don't know how to make it lighter than now (except by removing auto rescan) and meet your requirements :) I think, when discussing this in gitter, we will faster reach an understanding of exactly what the first version should be