search

helix-editor / helix

A post-modern modal text editor.
https://helix-editor.com
Mozilla Public License 2.0
31.82k stars 2.35k forks source link

WebAssembly plugins system #122

Closed CBenoit closed 3 months ago

CBenoit commented 3 years ago

Basically load .wasm files.

Capabilities:

At first we could use a basic toml config file or CLI to feed .wasm files to the editor.

A way to configure permissions on a plugin basis could be investigated to use the sandboxing capabilities coming with WASM. Example with wasmtime:

$ wasmtime --dir=. --dir=/tmp demo.wasm [args…]

(reference)

I think the biggest challenge is to get well-defined interfaces down but let's not fear to break it during early stages.

Later we could investigate embedding a wasm-based scripting language such as Grain or AssemblyScript.

Here are some references:

I'm willing to experiment soon

kirawi commented 2 years ago

Well, the main reason is because it allows you to write plugins in your language of choice and interoperate with any other plugin. Lua, for example, can be supported through the plugin system as well by compiling the reference Lua interpreter to Wasm. The other side of this is more pragmatic: the Wasm ecosystem is generally just more alive compared to Lua. If there's an upstream bug or missing feature that affects us, then we can be sure that it'll be addressed within a timely manner. This isn't a dig into LuaJIT, but it looks like only one person is maintaining it at the moment (though it is battle-tested).

At any rate, don't underestimate what people will do even if it's a text editor :P

Serif-7 commented 2 years ago

@kirawi Ah, I had no idea it was as simple as compiling the interpreter to Wasm. I thought you would need a wasm interpreter made from scratch to support Lua.

kirawi commented 2 years ago

It'll get even easier in the future with an agreed upon GC standard.

matu3ba commented 2 years ago

This isn't a dig into LuaJIT, but it looks like only one person is maintaining it at the moment (though it is battle-tested).

git log --invert-grep --author=Mike shows that the whole jit compiler (size ~100k LOC) is written by Mike Pall with various contributions from individuals. So there are no other maintainers in the first place.

If you compare this to lua, there is also only 1 person contributing since 2019.

I am still failing to see a strategy how to mitigate lower performance, ie with multithreading (opens again attack vectors, but as I understand it wasm is chosen mainly for portability). So far from my perspective and conversations with kakoune users, luajit is mostly essential for minimum-latency applications and fuzzy-searching + introspection into the complete editor state. Take as examples the fzf algorithm ported to C being significant faster than go and lightspeed.

This could also include actions based on treesitter-based querying, which may fire on every keypress, since they slow down the complete editor. However probably those could be parallelized in wasm, but I am not familiar with realistic parallelization benchmark results in wasm yet. However this looks like a good start, but unfortunately without any results.

TLDR; Provide, ideally natively, a fast fuzzy-searcher. If people want to stick with wasm: Make clear that plugin system in wasm does not provide minimum latency stuff and that people should use the C API/Rust API for that.

kirawi commented 2 years ago

I think that it'll be low-latency with Wasm, but I'm not totally sure. I think we do want to provide a built-in fuzzy-searcher though, or at least an interface.

kirawi commented 2 years ago

Make clear that plugin system in wasm does not provide minimum latency stuff and that people should use the C API/Rust API for that.

I wonder if we will eventually need such a thing. It has value for things that absolutely need access to the system. It'll probably be a C API though since Rust's ABI isn't stable. It sounds like a massive complexity headache though...

I think it would be better to use serialization like in Neovim?

Edit: Never mind, it would completely defeat the point of Wasm. It would be better to just have a more powerful singular plugin system.

glyh commented 2 years ago

@archseer It's good to see people proposing new features for helix, but I found that many of them are the features provided by plugins in other editors, or not everybody needs them.

For this project to be maintainable, would you please clarify what should go into the plugin system and what should be built-in?

midnightexigent commented 2 years ago

Possibly unpopular opinion (and mostly a rant) here, but I want to get this out there. IMO, helix should not go the route of Vim/Neovim/Emacs where very basic functionality is builtin and all the rest needs to either be done by hand, or via a plugin.

TLDR at the end

That drastically decreases developer productivity because users tend to spend so much time trying to optimize their Vimscript or Lua code (for Neovim) instead of doing actual work. Basically, before being able to get any work done, users need to learn a whole different scripting language, read so many pages of documentations that explain what each function does and so on

Alternatively, in Neovim/Vim/Emacs, there are plugins that supposedly make it easy to get started quickly and by picking the right ones, one can have IDE-like functionality. The problem is that there are SO MANY of them. A lot of them do the same thing and compete with each other too. For Neovim, every other day there is a "new and improved/rewritten in lua" plugin.. just look at how many plugins there are in the awesome neovim repo which is supposed to be a CURATED list

Even tho it is pretty cool to have highly customizable software, most people don't want to spend X time learning some sort of API or scripting language. They want easily configurable, sane defaults that just work. Which is why frameworks like LunarVim or doom-emacs are so popular. Most users probably don't even like messing with the defaults because they assume that the default is the intended way of using the software.

Let's get back to those frameworks for a moment, aren't those just coming full circle from a full fledged IDE with everything builtin ? I mean they are.. they turn bare bones text editors into full fledged IDEs. Except that they have the JIT overhead of whatever scripting language is being used and the "sane defaults" are provided by a 3rd party instead of the base developer.

So instead of installing a software and being able to get to work right away, a new user would go through the following steps:

Going the Vim/Neovim/Emacs for helix would likely make users follow the above steps

So to reiterate, IMO, software like Vim/Neovim/Emacs has 2 problems that helix has avoided by design up until now

TLDR

Anyway, I've been rambling for long enough, so here is the tldr;

Adopting the same system as editors like Vim/Neovim/Emacs makes most of the useful editor-features rely on plugin developers while the core developers become providers of an infrastructure used by those plugin developers. Over time, the number of plugins will explode making them harder and harder to control/predict.

The APIs need to be documented and maintained/evolved according to all sorts of demands from plugin developers. It also significantly highers the entry barrier for new users (just like Vim/Neovim/Emacs) who will likely end up using a batteries-included configuration framework (also just like Vim/Neovim/Emacs) to get IDE-like functionality

helix on its own seems to be the equivalent of Vim/Neovim/Emacs + a configuration framework (minus some features here and there). Since those configuration frameworks are sufficient for most use cases, standalone helix should also be sufficient for most use cases. Features provided by those frameworks have even become essentials. So when a plugins system lands for helix, IMO, it shouldn't be stripped down because going the Vim/Neovim/Emacs route would run into the same aforementioned problems those editors have

adsick commented 2 years ago

Which is why frameworks like LunarVim or doom-emacs are so popular

Just first time heard of them. I agree on having good defaults and features out of the box but extensibility is the thing (proof me wrong with VSCode being probably the best pick for most programming languages)

makes most of the useful editor-features rely on plugin developers while the core developers become providers of an infrastructure used by those plugin developers

Why not develop default plugins (e.g. pickers, terminal managers) and give an option to swap them with third party or even opt out if user doesn't need that piece of functionality.

Basically, before being able to get any work done, users need to learn a whole different scripting language, read so many pages of documentations that explain what each function does and so on

For me a good plugin system makes it easy to create powerful extensions and then install and configure them (only if needed). From a user's perspective this should work as follows:

Also I really want to say that it is reasonable to think of editions model of distribution so we can have different sorts of functionality depending on what we need:

dead10ck commented 2 years ago

@midnightexigent I don't think a plug-in system and a good out-of-the-box experience are mutually exclusive. Your own example of VSCode has both a good OOB experience and a massive plugin ecosystem that extends its functionality for those who want or need it. vim and emacs being really bare by default has nothing to do with the fact that they are capable of running plugins, and more to do with the fact that they are decades old and still try to cater to the most arcane of old systems. This is a false dichotomy you're setting up in your head.

pianocomposer321 commented 2 years ago

Just wanted to chime in here for a minute, because @midnightexigent and @adsick have given me some ideas, and I wanted to get them out there.

For me I think the perfect situation would be one where things like a fuzzy finder, auto-closing pairs, something akin to vim-surround, etc., would be built in (as they are or are planned to be now), but it would be easy to disable them, as @adsick said. But even beyond what @adsick was saying, I think it would be awesome if rather than being completely a part of the editor and strongly coupled with it, they were maintained separately under different GH repos under the helix-editor organization, and would be merged back into the main helix repo whenever there's a new stable release.

That way they could be forked, contributed to, or just used as bases for other plugin authors just like any other repo. They could make it clear in their readme what functionality a replacement plugin would need to provide (maybe these could even be exposed as traits if plugins are able to be written in rust), and then anyone could either just fork the repo and add features they want or start from scratch to make a plugin that implements those required traits.

I would see this as similar to how neovim recently added vim.select, which has a builtin interface, but can be replaced with any interface that returns a value after a function is called. The difference with helix, of course, would be that the builtin interface would me much more feature rich to begin with, but the extensibility of having that be easily replaceable would still be there. It would be kinda like if Telescope was built in and the default vim.select interface, but it was still able to be replaced with, for example, fzf, if the user wanted to.

I'm not sure if this is practical, or even possible, with the way the helix codebase is currently structured, but I think that's how an ideal plugin community would work IMO - there are base plugins provided by the editor maintainers that are included by default and are probably "good enough" for most people, thereby providing an out-of-the-box experience, but these components would be easily disabled or swappable, and, ideally, maintained separately and completely decoupled from the rest of the editor. IMO, this would provide the ideal experience for all three types of editor users (because you know there's only three out there 😜): Those who don't want to tinker at all, and just want a good OOB experience; those who want to be able to customize it a bit or disable it or swap it out for something else; and those who want to be able to completely re-write the whole thing from scratch.

dead10ck commented 2 years ago

Yeah, I like the idea of possibly decoupling some of the features like auto pairs into plugins, but having a default set of plugins that are bundled by default. This is getting a little out of scope of this issue though, which is to discuss the plugin system itself. We're getting a little ahead of ourselves. 😛

pianocomposer321 commented 2 years ago

Yeah, I like the idea of possibly decoupling some of the features like auto pairs into plugins, but having a default set of plugins that are bundled by default. This is getting a little out of scope of this issue though, which is to discuss the plugin system itself. We're getting a little ahead of ourselves. 😛

I don't think it's totally out of scope...how the plugin system is designed will be significantly affected by what things people plan to do with it. Having a core set of plugins included out of the box will only be possible if the plugin system is designed with that in mind.

dead10ck commented 2 years ago

That's true, I take that back.

matu3ba commented 2 years ago

IDE-like functionality

You never specified what exactly IDE-like functionality is. Please provide concrete examples what you believe should be part of core and a usable metric how to decide this (for other potential plugins?).

For me IDE-like is (without opinion what should be part of core):

  1. Personally I think a graphical debugger is what most people expect from a good IDE, but unfortunately gdb and lldb have only slow and maintained python frontends for with no C API for pretty printing (or better navigation like goto next fn and filter for symbol searches etc).
  2. buffer navigation like harpoon, git worktree navigation and ideally a better solution to fast buffer handling (search, closing etc)
  3. fuzzy finder for everything
  4. auto-completion
  5. version control integration like gitsign or for folks doing frequent merge conflict resolving stuff like fugitive

BTW: Typescript is slower than luajit, which is one of the main reason VSCode is slow (after startup).

Faster jumping in text and file navigation are too diverse to unify (unless you create clunky monstrosities) with the first requiring minimal input delay (for smart visualization). The same holds for the plethora of fuzzy finder plugins and plugins based on tree-sitter. And dont even start with calendar or organization stuff or functionality to use certain programs more efficiently from the editor (ie fetching data etc).

A metric I do agree with is necessary complexity and API cost of switching parts out. If both are high, it belongs into a central repo.

It also significantly highers the entry barrier for new users

I feel like your rant is more about the intended philosophy rather than concrete examples. From my point of view your points do not take into account the higher cost of removing bloated code once it accumulated, but deprecated due to better functionality etc.

If you have the code sufficiently complex that no single person understand how it works due to too much "nice for the user" and on top of that it is inside a monorepo, you will have a hard time refactoring it. If you force the user to customize some parts or split those maintenance into other repos (less people need to read issues etc and talk to more users), you have a simpler refactoring.

One of the best counter-example is telescope: How many plugins exist that customize telescope and do you believe that there are no use cases or all use cases can and should be maintained in a central place?

My other questions to you: How many users that want a modal editor for longterm-efficiency care about initial investment as most decisive factor? Is the main problem of vim/neovim and kakoune not that they have too much hidden/annoying to search functionality? How would you ensure the code stays maintaineable and the interested people dont get distracted by stuff they dont work on?

kirawi commented 2 years ago

Just to clarify:

Personally I think a graphical debugger is what most people expect from a good IDE, but unfortunately gdb and lldb have only slow and maintained python frontends for with no C API for pretty printing (or better navigation like goto next fn and filter for symbol searches etc).

574

version control integration like gitsign or for folks doing frequent merge conflict resolving stuff like fugitive

Partially addressed by #467

midnightexigent commented 2 years ago

Some interesting points/questions have been raised, instead of addressing them one by one, here is a full detailed explanation

Concrete example

While my initial rant is mostly about how annoying it is to start getting productive with editors like Vim (in part) due to their plugins system, I am not advocating against a plugins system for helix. I still think that a plugins system is important; not everything can be builtin: not all use-cases can be covered, and as some comments pointed out, trying to do it would just become unmaintainable

However, the Vim approach where everything is a plugin can also make the code less maintainable. For example, imagine stripping tree-sitter from helix and making it into a plugin. You'd have to provide a custom syntax highlighting API that the tree-sitter plugin would use, and that the user could use if they didn't want syntax highlighting or if they wanted to do syntax highlighting with something other than tree-sitter.

Realistically tho, if such an API were to be provided, the overwhelming majority of users would just use the tree-sitter plugin. Furthermore, which users would such a change actually benefit ? IMO, not many; first of all, everyone that uses helix wants syntax highlighting, users that don't are already using vi and probably will not hear of helix; secondly, the 0.01% that want syntax highlighting and believe they can do better than tree-sitter (ie. tinkers) are probably already very happy Vimscripting

Also, if such functionality is provided as plugins, there will be competing implementations which would just create confusion for users. They would need to know a lot of internals to understand what are the actual differences to decide which one to pick. Something similar happened with Vim/Neovim with CoC vs vim-lsp for the implementation of an LSP client. Neovim eventually stepped in and provided that as a builtin module which was one of the best changes in that software yet

More details

More generally, IMO, extracting a feature/module and making it into a plugin doesn't make the code more maintainable - there is always a cost. Rather, it moves the maintainability problems elsewhere with the added costs of: inherent performance overhead of running a plugin ; plugins API complexity ; requiring the user to look at a 3rd party repo to install/configure the plugin. Again, in a worst case scenario (ie. above example with tree-sitter), all of this would be done just for the overwhelming majority of users to use the same plugin.

How to decide what should be a plugin or builtin module ?

IMO, in the aforementioned scenario, the feature should absolutely be provided as a builtin module. Luckily those scenarios aren't that numerous. More specifically, I think the answer to the question "Should feature X be a builtin module or offloaded to a plugin ?" should be guided by questions like "Who would actually benefit from feature X being a plugin rather than a builtin module ?" or "Would the overwhelming majority of users just add back this feature as a plugin if it were to be stripped from helix ?". Moreover, I believe that features that satisfy the following constraints should be builtin modules:

What about tinkers & how would this affect the plugins system ?

The good news here, is again, features that satisfy those constraints aren't that numerous and could realistically be maintained in a single repo. Now, does that make helix opinionated ? Absolutely. But which users does that NOT benefit ? Probably only tinkers. This brings me to my next point: why bother with tinkers ? They represent a very small percentage of potential users that are probably already very happy with something like Vim. Trying to satisfy them would make the rest of the users confused if not annoyed "this plugin is clearly an essential, why is it not builtin smh ?"; plus the cost of performance overhead and plugins API complexity. See example above for more details

This also has the added benefit of possibly reducing the complexity and scope of the plugins API. The more use cases that API needs to handle, the more complex it's going to be and the more convoluted its interactions with the "back-end" will be. Since some features are known to be builtin, the design of the public API can make opinionated decisions on what to expose or not and avoid being as permissive as Vim's "here is the API, do whatever" style

That being said, this still leaves a lot of room for all sorts of awesome plugins and features which is why I think a plugins system is still important. As a matter of fact, I think that the current features builtin to helix are starting to reach the ceiling of what features satisfy the aforementioned constraints.

Organizing the code base in a maintainable way

Since the builtin features shouldn't be that numerous, it's not unreasonable to maintain them under the same code base. Thanks to rust's infrastructure, it shouldn't be hard to organize everything in a logical and maintainable way. IMO, each builtin feature should live in its crate, in this model, the existing helix-lsp and helix-syntax crates would also be builtin features

This way, these "builtin modules" would be virtually as easily maintainable as if they were plugins on separate repos. Except that they'd have the added benefit of being tightly coupled with the helix internals (meaning that they get special support) and not pass by the plugins API which should result in higher overall quality and better performance. Additional benefits include: having a single implementation per important feature; contributors who would be split across competing implementations in the plugins world can offer enhancements/features in a single place and everyone would benefit; etc.

OOB & configuring everything

As for the OOB experience, in this model, the user has the essential functionality and can start getting productive with minimal configuration/setup. Moreover I think that there should be 3 configuration layers (kind of like VSCode)

Additional examples

Here are some features and my take on whether or not they should be builtin.

I believe that the current features in helix shouldn't be extracted as plugins (including lsp, tree-sitter, file-picker, etc.). They could however expose some options or even hooks to make them more configurable

Open questions

This obviously isn't the end of the story as this doesn't answer which features should actually be provided as builtin/plugins and how ? Some open questions that come to mind as I am writing this are

I believe each of those questions merit their own dedicated discussions and their answers should take into account community feedback

pianocomposer321 commented 2 years ago

@midnightexigent Thanks for that detailed and in-depth comment! It definitely cleared some things up for me. There are still a few points I'd like to address however...

I agree with the following points:

However, there are a few things about the comment that seemed a little strange to me.

First of all, treesitter syntax highlighting isn't the best example IMO. To be fair, it served your purpose well: it's an excellent example of something that should definitely not be extracted as a plugin. However, it doesn't really accomplish anything to point that out - nobody disputed that point in the first place. Of course treesitter should be builtin...it's literally in the list of features in the readme. The same goes for multiple selections and the builtin lsp support, and the same will probably go for dap once it arrives. Those are all things that are standard enough in modern editors to be expected, and complex enough to implement that there's a clear advantage in having them developed in-house rather than as plugins.

Secondly, the point it appears you are arguing against - the point that some features that are currently included in helix should be separated out so that they are no longer included by default - is not a point that anyone was making either. What I and others were saying was not that these features shouldn't be included but rather that they should be easily disabled, and, ideally, able to be swapped out for something else. One way of facilitating that would be to extract the features as plugins. However, that wouldn't be strictly necessary, it would just make developing replacements for them easier. And even if they were separated into plugins, there would be no reason not to include them by default in the editor.

pianocomposer321 commented 2 years ago

On another note, I was reading over the thread again, and this comment from @sudormrfbin caught my attention this time:

Wasm enables users to write plugins in any language that that can be compiled down to it:

This is arguably very important in nurturing a good plugin ecosystem. Let's take neovim as an example. There are plugins made in vimscript, yes, but lua lowered the entry barrier to writing plugins and experimenting so much that there has been a recent explosion in the number of plugins. If helix went a step further and allowed writing plugins in any language and still get the same perfomance (considering the wasm overhead of course), that'd be a game changer.

While being able to write a plugin in almost any language you want to is certainly an advantage for plugin authors and tinkerers, having 100 different languages that plugins could be written in is also a disadvantage for the end user. Particularly when there are competing implementations for a given feature (which, let's be honest, there's no way to avoid if plugins are possible at all), I want to be able to read the code and comprehend at least the gist of what's going on so I can decide which one is a better choice. With neovim, you have to be familiar with two, maybe three scripting languages to be able to do this for 99% of the plugins out there (those being vimscript, lua, and maybe python). It's worth noting that none of those languages are particularly difficult to learn, and most users will probably learn most of at least the first two by osmosis after using neovim for any length of time.

By going the route of making WASM the plugin/scripting interface, helix wouldn't get that same kind of ecosystem where almost any user of the editor can understand what's going on in all of the plugins that he/she has installed. By saying to plugin authors "You can write your plugin in whatever language you want!", you are saying to plugin users "To be able to understand what is going on in your config, you are going to have to learn all of these languages".

The same goes for plugin contributors. IMO one of the main reasons that lua plugins "exploded" as much as they did isn't just because plugin authors could write in something that wasn't vimscript, but also that lua was the new hot thing that everyone was learning, so almost anyone could contribute to plugin development as well. If plugins don't have one main language that they are being written in, it will be much more difficult for people to contribute to existing plugins.

To be clear, I'm not saying that I think using WASM is a terrible idea because of this. I just think that this is a huge potential issue, and one that should not be ignored.

There may be many possible solutions to this problem, but the one that I would propose is deciding on one WASM-compatible language to be the "official" plugin/scripting language for helix, and maybe even embedding the compiler for that one language in the binary for helix. Plugins written using this "official" language wouldn't need to be compiled to .wasm bytecode before they would be installable for the end-user, because they could be automatically compiled on first run using the included compiler. This would encourage plugin authors to use this language for most of the plugins, while still leaving the possibility of using other languages for people whose main focus would be on just getting something to work for themselves rather than on making it easy to install for other people.

What do you all think? Is this actually an issue, or am I missing something? Have you considered having one official language, or is there a better solution I haven't thought of?

CBenoit commented 2 years ago

What do you all think? Is this actually an issue, or am I missing something? Have you considered having one official language, or is there a better solution I haven't thought of?

Embedding an official language is indeed something that we considered since the beginning as noted in the original post:

Later we could investigate embedding a wasm-based scripting language such as Grain or AssemblyScript.

Emphasis on the "later" though. We can wait until we actually have a WASM-based plugin system before committing to anything, so we just do that.

pianocomposer321 commented 2 years ago

as noted in the original post

Lol, can't believe I missed that, thanks!

dead10ck commented 2 years ago

Having an embedded "officially recommended" language for plugins might be a good idea, if for no other reason than it makes distribution much easier to implement: similar to vim plugins, all you really have to do is git clone and git pull; GitHub becomes your distribution platform, as it has come to be for so many things.

However, I'm not sure I agree that having a polyglot ecosystem will significantly impact users. I personally have almost never looked at the source code of any plugin I've used, unless there was a bug. All I really look at is: what are the features, is it easy to configure, and how actively is it maintained? I like that you look at the source of your plugins, but I would be very surprised to learn that this is something that most people care about.

midnightexigent commented 2 years ago

@pianocomposer321

the point it appears you are arguing against - the point that some features that are currently included in helix should be separated out so that they are no longer included by default - is not a point that anyone was making either

As I mentioned in my initial post, I was mostly ranting on the Vim/Neovim approach and throwing out there my thoughts on why helix shouldn't follow in Vim/Neovim's footsteps once a plugins system lands. Even though no one suggested the changes I am arguing against, when initially reading through the thread, I don't think I saw any roadmap on what will happen once the plugins system lands - this is the only comment I recall seeing that kind of brought up the point.

At the very least, the points I brought up could be used in future discussions on the matter once the plugins system lands

For the other point you brought up. We should be very careful as to what to encourage the users to do (contributing to the builtin module VS developing a plugin that replaces the builtin module). If the builtin modules are extensible and highly customizable, there should be virtually no need to develop plugins that replaces them. For example, rather than providing (let's say) a bare-bones tabs system and telling the user "if you want a tabs system that shows the icons of the file types, just make (or use) a plugin", the builtin tabs system would be configurable in order to accommodate for that use case (and other reasonable use cases). In general, there should be nothing preventing the builtin modules to expose a set of configs/hooks that make sense to them (they even could expose their micro-plugins API but I am getting ahead of myself with this last point)

IMO, we should be saying to the users something along the lines of "while you could technically use the plugins API to re-implement X or Y feature, unless you're accommodating for a very niche use case, please most definitely contribute to the existing builtin module ".

With that being said, if there absolutely needs to be a way to have helix with none of the builtin modules, @adsick 's editions idea might be the way to go. rust's infrastructure should make it fairly easy to accomplish that, nushell have a project structure that accomplishes something similar. That could even allow the user to pick which of the builtin modules to include and create their own edition

pianocomposer321 commented 2 years ago

@midnightexigent I think I understand a bit better what you are communicating. Thanks for taking the time to explain again.

I mostly agree with your conclusion, I just want to stress something that you glossed over, which is the main reason IMO that the builtin modules should be swappable, and it mostly boils down to these two quotes from your previous message:

that use case (and other reasonable use cases)

that make sense to them

The question is, who decides what is reasonable? The authors and maintainers of helix. And they haven't hesitated to say that their vision for the project will likely differ from the users' visions (which will also undoubtedly differ from eachother's as well). A feature that seems "reasonable" to you or me may not be implemented by the core team, and, depending on other factors, they may not even accept a PR. And that's ok - that's completely up to them. But if we're stuck with the implementation that they provide for everything, rather than being able to create and contribute to third-party ones as well, we're stuck with an editor that's making design choices that may not work well with our workflow.

So while I agree completely that users should be encouraged to contribute to the main, builtin implementation, I strongly believe, and would like to stress, that they should have recourse if the maintainers don't accept the contribution (which, I'll say again, they have every right to do, and likely will).

devanlooches commented 2 years ago

Just out of curiosity what is the status of this feature? I would really like to see this feature in the near future. PS: I already love this editor!

kirawi commented 2 years ago

There hasn't been much progress on that front. I believe that the current work with DAP is taking up the forefront of most of the available time that @archseer has with Helix, along with code review (can't speak on his behalf though). I'm not sure if there's anyone currently working on it.

glyh commented 2 years ago

As to embeded scripting language(if there will ever be one):

I don't know why grain is recommended to embede here, but at this moment is pretty immature, running a single hello world application took me ~80 secs, and here's a relevant issue.

I personally recommend that we stick to mainstream scripting languages, assemblyscript, ruby, python, etc. The benefit of this choice would be:

  1. existing languages would be people's familiarity, and they are battle-tested.
  2. The embedding language is used to write personal configs/glues, there's not performance-critical, but it's important for users to feel it's easy to write(a bad example would be EMACS + elisp, though I personally love LISP).

  3. Ecosystem.

    It would also be nice if that compiler is written in rust so we can easily integrate it. I think the promising option for now would be ruby's artichoke. It's an implementation of dynamic scripting language that is written in rust, plus the author plans to have WASM as its compiling target. At the moment it only works as a interpreter that can run on WASM. The other choice would be rustpython, which is also good but it doesn't have AOT compilation on it's plan.

Nevermind, I'm talking nonsense here. Both rustpython and artichoke don't plan to be a compiler. The only choice that is robust at the moment, I guess, would be assemblyscript. But it's not design to be a good scripting language, but for speeding up javascript. It's closer to those static typed language, that happens to look like JavaScript...

kirawi commented 2 years ago

There is https://github.com/helix-editor/helix/issues/998 which could allow people to use the Wasm system for compiled languages, and the RPC method for dynamic languages. Personally, I'm not sure anymore if Wasm is mature enough for a plugin system with dynamic languages, but idk. I think the idea of Wasm is nice, and it would be a future-looking feature.

glyh commented 2 years ago

Great. but I don't know whether multiple plugin system will be implemented in the near future.

EDIT: Maybe we can somehow glue them together, so that a plugin can be split into 2 parts, one piece running on helix host and the other running in seperate processes?

BTW, the WASM system should be designed with parallel in mind, we don't want to block everything like it would be in EMACS.

kirawi commented 2 years ago

I think that should be possible. wasmtime seems to be purpose-built for parallelization and asynchronous execution.

glyh commented 2 years ago

@kirawi would you provide references? Here wasmer people say WASI didn't define multithreading.

Though we can run multiple WASM instances, I think allowing a single module to utilize multithreading can be helpful when it comes to plugins like autocomplete, file tree, etc.

I don't know how it would go but at the moment WASI is evolving(e.g. we don't have GC), and RPC seems like a more practical system.

kirawi commented 2 years ago

The standard itself hasn't defined multithreading yet, i.e. you can't spawn threads from within a Wasm module, but we on our end can execute modules in parallel (if it makes sense to). https://docs.rs/wasmtime/latest/wasmtime/#core-concepts https://github.com/bytecodealliance/wasmtime/issues/888

dzhou121 commented 2 years ago

It's good to see that more and more editors now choose WASI as plugin support. I'm wondering if it's a good idea to call for collaborations between editors to work towards a standardised API. Just to think about the benefit LSP brings to editors. We could potentially do something similar in plugin space.

For sure each editor has its own unique features that require unique APIs, but at the same time, there must be a bunch of things that can be shared across editors. It's win-win-win for editor authors, plugin writers, and users. New editors don't need to start from zero. Plugin writers write once to support all editors that are compatible with this API. Users have more plugins to choose from.

srenatus commented 2 years ago

collaborations between editors to work towards a standardised API.

It's a bit far-fetched, but there's something similar happening in the API gateway space: https://github.com/proxy-wasm/spec#implementations

kirawi commented 2 years ago

I'm wondering if it's a good idea to call for collaborations between editors to work towards a standardised API. Just to think about the benefit LSP brings to editors. We could potentially do something similar in plugin space.

I think that's a compelling idea for increasing the availability of plugins and accessibility for plugin developers. However, I think it might be a bit early to imagine what this would look like because Helix doesn't yet have a plugin system implemented. We could at least take some inspiration from the plugin systems of other popular and mature editors though, like VSCode, Emacs, and Neovim.

YJDoc2 commented 2 years ago

Hey, I have recently found helix, and really liking the experience of it. I was looking for the plug-in system for spell check, and saw this discussion. I have read through (most-of :sweat_smile: ) the above discussion and wanted share my thoughts on it.

I saw a lot of discussion on what should be built in and should helix be opinionated and how much. I feel helix should be opinionated, at least to certain basic functionalities. One of the reasons I found out helix, was that VS code was getting too slow for my use, so I changed to neovim, but was not able to configure it correctly, and ended up taking some default config from net, which felt would suffice my needs. That said, I have still not set-up coc correctly, and having that functionality out-of-box in helix feels amazing. I think many, if not most of helix users would be using it for code development (correct me if I'm wrong, I'm just guessing here) so providing functionalities which most would need such as bracket pairing (some kind of snippet support?) built-in rather than through plug-ins would be great for user experience. Apart from that, features which might be necessary, but not all might need, can be provided through default plug-ins. We can save the WASM files into a dir in the same way currently language config for lsp is getting stored, and provide a config toml to enable/disable plugins. This way if someone does not want default plugins, they can disable them through there.

I would like to ask is there any way I can help in this? I have experience in writing rust, but not in writing editors or plugin system, although would like to help in any way I can. I saw a plug-in MVP PR in above discussion, but I think currently maintainers are busy in other things as well, so would like to help if possible.

I also have an idea for potential plug-in system structure which I am mentioning here, but it might not be much good idea.

From the storage and user's perspective, the plug-in wasm files would be stored in a single dedicated location such as the one currently lsp config is stored. Here all the wasm files will be stored, along with corresponding (if-any) plugin_name.toml files which will provide config for the plug-in if needed, and a global config.toml file, which will provide plug-in setup config to helix. I feel we should use JSON as the intermediate language between plug-ins and helix, as it is widely used, most of the data that needs to be sent can be serialized as JSON, and many if not most languages have good support for ser/de-ser their structures to JSON. I don't have experience in plug-in system , so this question might not make any sense, but won't we need an event API for the plugins? otherwise, how are we going to decide when to run which plug-in? If there will be such event API, then we can split events into text-events, or lsp-events or gui events and such , and in config.toml the user will specify which type(s) of event the plug-in should be registered for. At such event , we can invoke that plug-in with a JSON-serialized context parameter, and it should return some standardized format result in JSON, which will then be used by helix to update the state. We'll need a way to resolve when multiple plug-ins return values on same event. Also we'll need to decide a standard interface for the plug-in to "get registered". I am not sure otherwise how can we decide the function to be called as plug-in , especially as it was discussed that potentially one might pack in interpreters in wasm and use that to develop plug-ins.

Please let me know your thoughts, and how I can help in this. Thanks for such a great editor!

adsick commented 2 years ago

I feel we should use JSON as the intermediate language between plug-ins and helix

call me a freak but I personally don't think that's fine

adsick commented 2 years ago

how are we going to decide when to run which plug-in?

I really encourage looking at what Bevy engine is doing with their 'systems', 'plugins' and scheduling

YJDoc2 commented 2 years ago

Hey

I feel we should use JSON as the intermediate language between plug-ins and helix

call me a freak but I personally don't think that's fine

I was suggesting JSON because of the reasons that I stated, but that does not mean it is definately the right choice. There might be some better alternative to JSON as well. One option I can think of is protobuf, but not sure how good it would be or how much it is supported by various languages. On the other hand, we might be able to make an interface which might not need such ser/de-ser data format, too. Can you suggest what other format can be used to pass data to-and-from between plugin and helix?

how are we going to decide when to run which plug-in?

I really encourage looking at what Bevy engine is doing with their 'systems', 'plugins' and scheduling

Thanks a lot for mentioning this, this was a great implementation to read about!

From what I understood, schedule is used to decide the ordering between various systems (essentially equivalent to plugin for this discussion), which we can also do in the TOML file using before/after/parallel-with kinds of params for each plug-in. But I do not think we can use the same structure for running the 'systems', as bevy uses rust type-system with concept of states , which might be difficult to cross across the WASM boundary.

I feel it would be better to interpret the states as events in our case, and associate some data with each, then a plugin can subscribe to event(s) and running of plug-ins for given event will be decided by scheduling as above.

adsick commented 2 years ago

On the other hand, we might be able to make an interface which might not need such ser/de-ser data format, too.

yeah, that seems to be what I'm thinking about.

Can you suggest what other format can be used to pass data to-and-from between plugin and helix?

sorry but no :smile: I'm not a pro in this area.

Anomalocaridid commented 2 years ago

Cap'n Proto might be worth considering if we need to use some sort of data interchange format for inter-process communication. It is very fast and same some interesting features like "time traveling" promise pipelines.

It has bindings in a handful of languages such as Rust, but some appear to be more ready to use than others and implementations for some languages only support serialization and not RPC.

glyh commented 2 years ago

@Anomalocaridid I guess this comment should better goto IPC plugin system thread? WASM have it's own proposal for data interchange with host though.

Aidan-Chelig commented 2 years ago

edit: I now see that I spoke too soon I like Zellij and think Helix would fit perfect into a Terminal Multiplexer provided there are facilities to navigate between the two seamlessly. Have you considered following their example on WASM plugins? I don't think its the best solution for helix specifically but could give some insight.

kirawi commented 2 years ago

It was mentioned as prior art here.

Himujjal commented 2 years ago

I would like to offer my two cents here.

I primarily work in the following programming languages as my default: Go, Zig and JavaScript/TypeScript. I like Rust. But it's too complex a language to write simple things such as plugins FOR ME! (NOTE: For ME!). For huge softwares? It's the best. I get it.

I love Helix editor. And the plugin system I want should be completely WASM based, where the exchange between the plugin and the main editor is in C Strings, integers, float and boolean. I would write the plugins in Zig or Go which are my preferred languages.

This is as simple as it gets. No Scripting language BS. I know lua is better than vimscript. But lua still is a bad language (in NeoVim).

As for when to call the plugins. The plugins should have an init function that should return a list of places the plugin's function should be called and which function specifically. In WebAssembly there is good support for strings, integers, floats and booleans. Basically primitive types (I know strings are not but!). This way we can simply call

call_func("function_name", string, int, bool, float): string;

and return the same primitive types.

And I agree with @midnightexigent that it should have Git integration by default.

In fact Helix should be: VS Code for the terminal in a Vim way.

emummel20 commented 2 years ago

I don't know why grain is recommended to embede here, but at this moment is pretty immature, running a single hello world application took me ~80 secs, and here's a relevant issue

@glyh fyi, that slowness issue for Grain is fixed in v0.5.0 (released in a few weeks)

Just my two cents but I would LOVE for Grain to be the embedded language of choice, because if we want a nice modern editor and tooling, then we should adopt the most modern paradigms for our language. I think using a typesafe functional language is the way to go for all the same reasons fp is great to begin with. Refactoring, preventing bugs, etc.

cameron1024 commented 2 years ago

Just stumbled across Helix and really enjoying it so far! It seems like it could be my perfect editor (opinionated and feature-rich like VSCode, but lightweight and fast like (neo)vim).

I largely use Neovim plugins to "replicate" VSCode features (telescope, various LSP helpers, a file tree, better terminal buffers, etc), so my personal need for plugins in Helix is low. But I think it's something that's likely to hinder adoption, even on principle.

With that said, I'd be keen to know if there's any thoughts/plans regarding sandboxing. This issue is often brought up with regard to the Rust ecosystem (particularly editors executing arbitrary code from the internet in proc-macros and build.rs). For example, would it make sense to have a permissions model, where plugins must declare which system resources they need access to (e.g. FS access to /foo/bar/**, clipboard, network, etc). I'm not super familiar with the available WASM runtimes that are being considered but this would be a feature that I'd quite like to see. Vim plugins having network access already makes me uneasy.

In an ideal world, the user would have to add these permissions explicitly to a plugins.toml (or equivalent) file, and plugin authors would include these in their install instructions. Perhaps even a TUI-based installer which asks for each required permission.

Other than that, this seems like a really exciting step! 🎉

bhougland18 commented 2 years ago

Just throwing it out there but would https://github.com/brundonsmith/rust_lisp be a possibility for a scripting language?

Small footprint (both code size and memory usage) No runtime dependencies [1] Easy, ergonomic interop with native Rust functions Small but practical set of Lisp functionality

glyh commented 2 years ago

I believe people here would prefer a WASM solution(they even reject Lua)... Maybe you can propose a lisp that compiles to WASM.

Just throwing it out there but would brundonsmith/rust_lisp be a possibility for a scripting language?

Small footprint (both code size and memory usage) No runtime dependencies [1] Easy, ergonomic interop with native Rust functions Small but practical set of Lisp functionality

jakenvac commented 2 years ago

im a little late to the party, but I had a quick skim and Ctrl + F and couldnt find any talk of this.

Im sure it has many downsides (size being one) but have we considered using deno as the wasm runtime? The reason I think to mention it is that it supports typescript, JavaScript and wasm and is fully compliant with web standards meaning plugin authors can leverage libraries made for the web, deno and in some cases, even nodejs.

Theres an official rust crate here that might offer some more insight.

The way I see it is we get the benefits of having a wasm runtime for any language that supports it as a target, and the benefits of the JS and TS ecosystem on top only take it further.

Long term the project could provide typescript types for the plugin api so people can get started quickly.