search

haskell / haskell-language-server

Official haskell ide support via language server (LSP). Successor of ghcide & haskell-ide-engine.
Apache License 2.0
2.67k stars 364 forks source link

Major steps required (Roadmap) #1

Closed alanz closed 2 years ago

alanz commented 4 years ago

2020-09-29 Update (@jneira)

lukel97 commented 4 years ago

stack hie bios https://gist.github.com/chrisdone/7ef391e8445216f5e30b2665b85a6db2

jneira commented 4 years ago

Have you considered use precompiled executables to use plugins at runtime, instead compile at runtime them? I guess they should be executed as daemons (like the ide server itself), to avoid the start process overhead. We could use a subset (?) of lsp json format to comunicate between the ide main server and the plugin executables, to no reinvent any wheel. This way the plugins+lsp could be even used in a isolated way from a vscode extension. In fact maybe we could distribute them as separate vscode extensions, keeping in the server only the core interface with ghc: compile errors, goto def, etc

Caveats: we had to wrap each plugin lib (hlint, brittany, floskell, etc) or sets of plugins in a executable+lsp stdio+ghc interface wrapper and distribute the precompiled executables separately.

ndmitchell commented 4 years ago

@jneira many plugins are going to require a GHC session. Either you can duplicate the GHC session to different processes (which is incredibly expensive in CPU terms), or marshal every GHC API call over to the GHC session (incredibly expensive in terms of communication). It would be nice if we could figure out a way that works, but I can't think of one.

jneira commented 4 years ago

Mmm i see. Thanks for sharing.

So add to the ide server a bridge between the ghc session and the plugins using a lsp-like format is not viable :disappointed:

Still, could plugins not using the ghc session be isolated? Dont know much about each tool internals but maybe any of hoogle, hlint, liquid or any of formatters dont use a ghc session???

ndmitchell commented 4 years ago

Most plugins are likely to get something from the GHC environment - the parse tree, the text after preprocessing, the names on which to run searches. Looking at HLint and formatters like Ormulu, they should be using the GHC environment (they currently build their own GHC environment, which is a bit different).

alanz commented 4 years ago

@jneira the approach fpcomplete took in their initial web-ide backend took that approach, and I understand had major issues with just sharing data between the various processes.

alanz commented 4 years ago

Plus the prospect of the kinds of issues that will be raised because people are not able to get a set of separate processes running and talking to each other is not good. Think our current stuff x 10.

jneira commented 4 years ago

Sorry if i am adding a lot of nosense but i think @mpickering mentioned a library that tried to eval haskell code at runtime than maybe could be useful: not sure if it was https://github.com/stepcut/plugins

fendor commented 4 years ago

Out of curiosity, did you have time to touch debugger integration? Or is it too soon to talk about such a feature?

alanz commented 4 years ago

Out of curiosity, did you have time to touch debugger integration? Or is it too soon to talk about such a feature?

We had a brief conversation, which did not go anywhere in particular. I think it is something that needs to happen, sometime, but we first need to get over the initial process. Once haskell-ide is available for use, we can consider additional stuff.

alanz commented 4 years ago

Just dumping this here, thoughts on range formatting. https://github.com/haskell/haskell-ide-engine/pull/1602#discussion_r371497860

NightRa commented 4 years ago

Sounds like a perfect use case for dynamic linking @jneira @ndmitchell

jneira commented 4 years ago

Another question: could we continue using the hie solution to handle several ghcs using a wrapper? I think it could be more beginner friendly although it has a point of friction trying to guess the ghc version used in a project.

Somewhat philosophical side note: it seems to me that ghcide is more used by (suited to?) experienced haskell developers. I think we have the opportunity and the challenge to build a tool well-balanced between beginners and more experienced haskell devs.

jneira commented 4 years ago

@NightRa coming from the jvm world, where it is the default way to go, i think so. However i am not sure if it is widely used enough in the haskell world to not hit a lot of pain points (performance, bugs, etc, etc)

ndmitchell commented 4 years ago

Dynamic linking is not a well trodden path in Haskell. I would avoid it like the plague as it's no fun for support.

The plan is to do the HIE compiling lots of binaries thing.

jneira commented 4 years ago

The plan is to do the HIE compiling lots of binaries thing.

It has its own pain points especially for beginners, although i suppose they are better known and battle tested. Experience is king here (always? :wink:) and absolutly trust @ndmitchell, @alanz and the surely thoughtful discussions at Bristol. However to avoid too much (exponential?) slow static compilation combinations i would not totally abandon the alternative of break them via binaries wherever is factible. Even if we start to compile everything statically for the shake of simplicity.

alanz commented 4 years ago

@jneira As far as I am concerned, the initial effort should be to end up with an equivalent setup here as we have for hie, in terms of the wrapper and build system.

Also, to achieve parity by migrating our existing plugins.

It is not the time to try experimental stuff, I would like to see something usable ASAP.

alanz commented 4 years ago

As far as I am concerned, the initial effort should be to end up with an equivalent setup here as we have for hie, in terms of the wrapper and build system.

We are getting close to this, in the sense that we have copied it over, and done a smoke test.

I think the most important feature before we can recommend it to anyone is multi-component support.

Anton-Latukha commented 4 years ago

What is "Implicit Configuration" I guess "Settings", menus and buttons?

jneira commented 4 years ago

@Anton-Latukha if refers to the "automatic" configuration to start a ghc session with hie-bios, using cabal-helper (or other) to get the ghc flags instead an explicit hie.yaml file: see https://github.com/haskell/haskell-ide-engine#project-configuration

complyue commented 4 years ago

Either you can duplicate the GHC session to different processes (which is incredibly expensive in CPU terms), or marshal every GHC API call over to the GHC session (incredibly expensive in terms of communication). It would be nice if we could figure out a way that works, but I can't think of one.

I'd done some components with an idea called Hosting Based Interfacing, basically you re-organize fine-grained API calls into coarse-grained, business-oriented, parametric pieces of peer script, send it to a remote process for (hosted) execution, the result (naturally asynchronous and in free form) is conveyed with similar pieces of such script back to the originating peer process.

In context here it means to have a single GHC session hosting process, receiving and executing scripts from various plugin processes, those scripts help themselves to grab necessary (thus amount-wise, greatly reduced than raw GHC api transcription payload) data and do scripting back.

This technique won't reduce the overall complexity, actually, writing a single piece of code that would run interleaved among 2 or more processes, is confusing and uncomfortable at times. But it is especially effective for realtime communicating services / clients over WAN, insensitive to the delay of double latency imposed by the round-trip of request/response cycles, which may more because of the async nature, another significant optimization is that, per api-call marshal/unmarshal is avoided, the total payload over-wire is greatly reduced.

Hosting Based Interface won't make it easier or simpler for multiple plugins to share a common GHC session, but may help developing a minimal thus hopefully more stable contract for what artifacts (i.e. constants, procedures etc.) to be hosted at a single GHC session process, while leaving the plugin processes enough flexibility and space of imagination to script the transactions to accomplish their jobs.

jneira commented 4 years ago

//cc @alanz

alanz commented 4 years ago

I think we should add the plugin architecture change here too. #421, And possibly the overall view of things, #409.

jneira commented 4 years ago

@alanz Updated with those tasks, i've added use ghcide master too. Sorry i do not remember/know what stack/cabal hie-bios flags was about, being able to use one or another?.

alanz commented 4 years ago

I think the hie-bios flag was the intention to make upstream changes to stack/cabal to be able to call them with a flag so that they directly return the build info that hie-bios needs. And I think both patches were made, not sure if either has landed yet though. Maybe they need a prod.

jneira commented 4 years ago

thanks! i've updated the checklist with links to the relevant prs/branches about upstream changes to cabal/stack

fendor commented 4 years ago

There is a stack patch? 0.o

jneira commented 4 years ago

Yeah, afaik it was done in Bristol, @bubba posted the generated info in a comment above: https://github.com/haskell/haskell-language-server/issues/1#issuecomment-578507603 Btw, do you think show-build-info has chances to be merged at mid term? only curious :smile:

fendor commented 4 years ago

I think so, I think it will be merged after the cabal-3.4 release. cc @bubba

chrisdone commented 4 years ago

Context is no longer fresh in my L1 or L2 cache from Bristol, but I wrote the stack patch and was waiting from anyone to confirm whether the contents and format is acceptable. We weren't working to a specific schema or anything.

We could have a trivial module in a trivial package like hls-schema with a low dependency footprint, that defines the data types that are encoded/decoded via aeson by cabal-install or stack and HLS. Backwards-compatible updates can be made for additions with defaults, and breaking changes can yield a version bump. stack and cabal-install can depend on hls-schema==major.version. And if the package avoids derived JSON instances, we avoid incompat. between aeson releases.

lukel97 commented 4 years ago

@fendor Indeed, for reference the PR is here

jneira commented 4 years ago

There is the build server protocol that aims to be a standar like lsp. Maybe we could use it loosely instead create a new one: https://build-server-protocol.github.io/docs/specification.html Taking a quick look, it seems main stack/cabal concepts could be adapted to.

chrisdone commented 4 years ago

Without something type-safe enforced by Haskell I don't see much value in such aim-to-be-standard specifications over a trivial type-safe library, but ymmv. There's object-relational impedance to overcome, too.

alanz commented 4 years ago

I agree with @chrisdone . We have our own way of doing most of the things BSP currently offers, it is not a directly good fit. So lots of effort for no real reward, and massive unnecessary additional complexity.

The hls-schema concept makes sense.

jneira commented 4 years ago

maybe it is only matter of naming but i like hie-bios-schema, less tied to the final downstream

alanz commented 4 years ago

but i like hie-bios-schema

Perhaps wait to see if there is any renaming of hie-bios? I get the impression the word "hie" is now an anachronism, and confuses newcomers. I think that ship has sailed though, it is pretty much baked in all over the place.

alanz commented 4 years ago

And for the record, I agree that the schema belongs to hie-bios

chrisdone commented 4 years ago

No preference for naming from me.

But this sets a nice dependency order: the patch for stack/cabal-install ought to be essentially (1) import lib, (2) fill in the data structures, (3) provide command that dumps the data structure to stdout.

There might be some back and forth if there's a field I can't provide from Stack, those could be Maybed.

I don't have much bandwidth for tooling anymore. But I can get low overhead changes into Stack like this.

jneira commented 4 years ago

Nice, @chrisdone thanks for take care. @fendor @bubba how could this plan fit your work in cabal-install show-build-info?

fendor commented 4 years ago

I like the plan, I wasn't aware that we are actually this close to have tooling support in cabal and stack :open_mouth:. I think we should discuss the contents of the package hie-bios-schema (or similar) at the hie-bios repository to not pollute the big picture here, and since hie-bios and hie-bios-schema will have to work together. We also need to think about the API changes this entails to hie-bios, since now there are multiple trade offs to discuss, such as eager vs lazy loading of components, query by filepath or query by component, etc... We should converge on a datatype, ideally before https://github.com/haskell/cabal/pull/6241 is merged, to make adjustment to the PR to fit immediately what we need. I expect any backwards incompatible changes will be hard to get into cabal.

For the implementation plan in cabal, I am not sure they will add new dependencies to it, such as aeson or hie-bios-schema, and I think, usually such types are added to cabal-plan, which is fine I think.

@chrisdone I missing some information, such as path to ghc, libdir/topdir, maybe ghc version and the actual compilation options required to compile the component with. At least, judging from what I can see in the output https://gist.github.com/chrisdone/7ef391e8445216f5e30b2665b85a6db2 and the code in https://github.com/commercialhaskell/stack/compare/add-stack-ide-bios-cmd.

EDIT: BTW, I volunteer/want to maintain a library such as hie-bios-schema

jneira commented 4 years ago

We already are using ghcide master

jneira commented 3 years ago

@fendor out of curisoity, have you the opportunity of make some progress about the cabal/stack integration?

fendor commented 3 years ago

@jneira Unfortunately, not at all. I have been occupied with other stuff :(

jneira commented 3 years ago

no worries, maybe we can reactivate it via the next Gsoc?

fendor commented 3 years ago

That is definitely a good idea!

andys8 commented 3 years ago

Regarding module graph: the hiedb branch was merged.

jneira commented 3 years ago

@andys8 thanks for noting it, checklist updated

jneira commented 3 years ago

I've labeled it as Eligible for gsoc 2021 due to

stack/cabal hie-bios flag: upstream changes in cabal/stack to make them give precise build info to hie-bios

cabal @fendor/@bubba show-build-info: haskell/cabal#6241 stack @chrisdone branch https://github.com/commercialhaskell/stack/compare/add-stack-ide-bios-cmd

It has not a dedicated issue //cc @fendor

jneira commented 3 years ago

Plugin architecture check done!

jneira commented 3 years ago

cabal show-build-infohas a new pr, we hope it will be the definitive one! https://github.com/haskell/cabal/pull/7478