search

helix-editor / helix

A post-modern modal text editor.
https://helix-editor.com
Mozilla Public License 2.0
33.38k stars 2.47k forks source link

WebAssembly plugins system #122

Closed CBenoit closed 6 months ago

CBenoit commented 3 years ago

Basically load .wasm files.

Capabilities:

At first we could use a basic toml config file or CLI to feed .wasm files to the editor.

A way to configure permissions on a plugin basis could be investigated to use the sandboxing capabilities coming with WASM. Example with wasmtime:

$ wasmtime --dir=. --dir=/tmp demo.wasm [args…]

(reference)

I think the biggest challenge is to get well-defined interfaces down but let's not fear to break it during early stages.

Later we could investigate embedding a wasm-based scripting language such as Grain or AssemblyScript.

Here are some references:

I'm willing to experiment soon

pianocomposer321 commented 2 years ago

I believe people here would prefer a WASM solution(they even reject Lua)... Maybe you can propose a lisp that compiles to WASM.

Just throwing it out there but would brundonsmith/rust_lisp be a possibility for a scripting language? Small footprint (both code size and memory usage) No runtime dependencies [1] Easy, ergonomic interop with native Rust functions Small but practical set of Lisp functionality

This actually brings up a question I've had for a while...I don't know much about how plugin systems like this generally work, or what the plan is for the implementation of this one specifically, but I imagine it's something like writing the api in rust first, and secondly providing WASM bindings for the api. If this is the case, then I would think there will be an in-between stage where the api is ready (or at least base-line functional), but the bindings haven't been written yet. In this in-between phase, there would probably be some way to access the api through rust itself (though I'm not sure how complicated the process would be, e.g. whether you would have to recompile the entire source or not). So for that stage, something like rust_lisp could be useful for people who are eager to get started scripting things before it's exposed to WASM.

Am I right about any of this, or completely off base? ;-)

Himujjal commented 2 years ago

Why not QuickJS? Much better alternative to Deno just for writing plugins. Small, compact with okayish performance. I know JavaScript is not a very well designed languages. But it has one of the largest known library collection on planet earth. Maybe it would be a good fit for plugins. Another thing to note is that, the number of plugins that we can reduce and bring to the main Helix editor itself is a win-win in this situation.

WASM is a good fit. But development of plugins wise it will be a mess. If QuickJs is integrated with something like Wasmtime, it would be even better.

cameron1024 commented 2 years ago

I'm not convinced WASM plugin development is guaranteed to be a mess.

I can imagine a shiny future where I can type hx --plugin-dev --lang rust to create a project template for plugin development, with all the interface glue written for me, so I can start writing the code immediately.

In terms of language support, WASM has the advantage that it can attract people from various backgrounds. Many languages have excellent support for compiling to WASM, and JS developers have AssemblyScript.

Of course, such a smooth developer experience would require some work, but I don't think there's reason to believe it won't be possible. And even if the developer experience is slightly sub-par, I'm not sure it would be a major blocker. If people are willing to write VimScript to automate their workflow, I'm sure they'll be able to work around some slightly janky tooling at first :p

rishflab commented 2 years ago

I think the decision of whether to use WASM or not for plugins is a bit of a red herring.

I think the first thing to is figure out what you envision people building with the plugin system. Do you want people to be able to build tools like magit?

What sort of API would this require? Will simply exposing commands be enough or will it require exposing the UI and internal text datastructures?

What would the architecture in terms of concurrency be like. Does helix spawn a (green) thread to execute a plugin? Can you have long running background plugins? How do you deal with data races if you choose to go multithreaded. If you stay single threaded, how do you deal with UI hangs because of computationally expensive plugins?

Whether or not to use WASM and the runtime is secondary to all of this IMO.

eulerdisk commented 2 years ago

Just my 2 cents as a new helix user.

After many months using Neovim and many hours spent configuring it, I tried helix a couple of days ago and I think I will stay with it! Why? Because has no plugins!

I don't want to use anymore a text editor which relies for 95% on plugins to do the work. I think plugins, when overused like they are in Neovim world, are a total mess for the following reasons:

So I would really like to helix not to go down that path. I don't want to see important features (e.g. git integration) implemented as plugins. I really like the VS Code approach here. Plugins are necessary, but they should be 5%/10% of the experience not 95%. They should extend the editor, add specificity and customization, not implementing the big features. There should be different API based on what you want to extend. Example: generic+git VCS support is built-in, you can extend it with plugins to support specific VSC (e.g: mercurial?) (in other words, the same way LSP works)

To give plugins total freedom it's the easy path at first because you avoid the problem of designing a good set of APIs, but it's a road to hell.

eulerdisk commented 2 years ago

Just to expand a bit on what is my general idea for a plugin architecture.

For me there are roughly 2 ways plugins could work:

A- The editor provides the infrastructure, the plugin fills in the gaps. This is the LSP/DAP approach. (the "extension approach")

B- General scripting capabilities. Use the editor api to get infos/send commands to open windows, create buffers etc.. (the "script approach")

In Neovim basically everything is B (if you leave aside the LSP implementation..) which is where the mess come from.

Both are necessary but I want my ideal text editor to lean toward A as much as possibile. e.g: for a VCS implementation, UI/UX is provided by helix then you have N plugins filling the gaps for every VCS. I don't want to see VCS plugins inventing their own UI/UX.

You need B too for general scripting and automation, and it's easier to go that way first, but if you not limit the api the risk of getting out of control then is very high.

Badly written plugins will always be there faster than a proper designed built-in solution, so everyone will start to use a tons of plugins and you get the problems I listed in the previous message. (...and developers will lose motivation to develop a better integrated solution because a lot o people, even themselvers, are already using the plugin....)

Currently I do not know a way to limit the B approach, one solution could be that scripts could use only existing commands. Or maybe it's not necessary limit the API, maybe it's enough for helix to implement as much built-in stuff as it can then people will use that instead of plugins, then plugin authors will limit themselves.....I don't know.

(P.S: If I have to choose only one route I would go only with A)

rishflab commented 2 years ago

I don't want to use anymore a text editor which relies for 95% on plugins to do the work. I think plugins, when overused like they are in Neovim world, are a total mess for the following reasons:

* UI/UX inconsistencies (between themselves and/or the editor)

* conflicting/overlapping features (between themselves and/or the editor)

* distribution and versioning problems (some plugins require the nightly version, some don't)

* a lot of copy/paste configuration that you see one time in your life and forget or don't understand anymore

* often are unreliable and become unmaintained or broken over time

To give plugins total freedom it's the easy path at first because you avoid the problem of designing a good set of APIs, but it's a road to hell.

Agree with this.

I think it would be a good idea to ask/survey the community what functionality they want added to helix. A plugin system is a way to implement functionality so it would not be an acceptable answer. After gathering responses it might be easier for the devs to figure out what changes are needed to helix. Maybe a plugin system is not even required.

Lindenk commented 2 years ago

Just thought I'd throw my own two cents down the well

While I agree with the sentiment of avoiding plugin hell, I don't agree with the approach. Limiting the plugin api stifles creativity and the community's ability to experiment. It's like fixing a leaky pipe by removing the sink. Instead I believe the issue with many overly plugin-reliant ecosystems is the opinions and attitudes of the maintainers of their base projects, specifically:

Usually new features aren't just invented in their perfect state and added to a project. Having a plugin system allows the community to come up with and test new ideas, with as many test subjects as are interested. If enough support is gained, the plugin could be streamlined and integrated into Helix proper. In my opinion, attempting to handle this level of quick iteration and distributed testing in the core project alone is a pipe dream.

The main concerns with an extensive plugin system appear to be:

For issues 2 and 3, integrating core plugins with Helix and providing sane defaults completely alleviate the problems. For handing inconsistencies between plugins and/or the editor, one approach is to observe how these plugins are implementing their functionality and improving the api to provide a standardized way of interacting with it.

For example, let's say we wanted to write a plugin that provides type hints. The api could give the plugin full access to the screen buffer and have the plugin figure out how to render hints properly, which would have a high chance of breaking support with other plugins and updates to the editor. But it would work, even if it's a mess, which gives us information about how the api could be improved. Using what we've learned, we could then add virtual text support to the api, so the plugin can tell Helix it wants a chunk of text after a token instead and have Helix handle how it should be rendered

In any case, if Helix wants to compete with existing modal editors out there, I'd expect we'd require a plugin system to reach feature parity before the heat death of the universe.

It's only a road to hell if no one bothers keeping the signs updated

David-Else commented 2 years ago

@Lindenk I like your point about having plugins to iterate ideas. I would say that in this regard Helix is in a great position to simply clone the very best Neovim plugins and put them in core. They have been through many iterations, starting off back as Vimscript, evolving into Lua and then winning the battle between the competitors and becoming the most popular.

The main advantage of Helix over Neovim is having functionality built in, that is its USP. It would be cool to have Helix plugins for all the little niche things people want done, but there is already Neovim for that, there is no point in trying to compete. Why not write a Helix emulation in Neovim, it would be a lot easier :)

Anomalocaridid commented 2 years ago

@David-Else I see where you're coming from, but that seems to me like that approach would end up making Helix somewhat bloated, especially since I doubt everyone that uses it would necessarily need or want all of that functionality.

Instead of completely leaving out a plugin system, we should learn from the problems that plugin-reliant ecosystems tend to face and try to create an approach that address them.

coderedart commented 2 years ago

I would like to support Lua.

Wasm

just wanted to point out that WASI is still not standardized. you can see the status of proposals here. https://github.com/WebAssembly/WASI/blob/main/Proposals.md out of 0 - 5 steps, most of them are still within step 1 or 2.

the only scripting language that has even experimental support for wasm is python and we are talking about using wasm modules from python, not python itself being embedded or python modules being used by other (rust in this case) languages. essentially, you cannot actually use a scripting language yet in combination with wasm unless you embed the whole interpreter (wasmoon for lua or rustpython for python) in the rust app.

Witx-bindgen (mentioned at the start of this long discussion) is already deprecated in favor of the new wit-bindgen. there's a new thing called component models which allows us to define "C header" like wit files and generate bindings from those. the only good reference i can find is https://radu-matei.com/blog/intro-wasm-components/ and its sequel https://radu-matei.com/blog/wasm-components-host-implementations/ .

Unfortunately, this component model thingy is not yet implemented in wasmtime. again, component model is still an experiment and not a standard yet either.

Just trying to point out that we are years away from this utopia of "use wasm to support all languages". i have seen a lot of comments argue for wasm using "plugin authors can use any language they want" merit which is just a dream for now.

The actual progress is happening in two prs https://github.com/helix-editor/helix/pull/455 and https://github.com/helix-editor/helix/pull/2949 which have recent activity. neither of them have figured out a way to expose the interface to languages other than rust (and maybe C). there's no scripting possibility here AFAICT.

Alternatives

  1. Rustpython / PyO3 don't allow passing a rust object into the python runtime for scripts to use. both are still slow but nobody ever expected python to be fast i guess.
  2. Deno is not friendly to embed. there's a embedding deno blog post in making for months (the devs said so on discord), so this might not be true in the future.
  3. Rust langs. there's a bunch of languages made in rust like rhai, rune, gluon etc.. but ofcourse, they would be "new languages" for plugin authors to learn. and none of those languages are backed by companies or significantly sized community.

Lua

Actually embeddable right now with tealr crate which wraps mlua.

  1. Luajit is fast. if you need speed. but there's also luau (Roblox lua) which is reasonably fast and sandboxed (as Roblox needs user scripts to run on their servers, they take sandboxing seriously).
  2. Create native lua modules with rust. plugins which want to do some heavy computations or use threads or any rust stuff can simply push that functionality into a rust project, wrap it in a module api however they want and require it in their lua script. https://github.com/khvzak/lua-ryaml is a luarocks package made with rust / mlua. NOTE: native modules are not supported in luau (yet) because it throws all sandboxing out of the window.
  3. Tealr generates documentation for your bindings including function signatures and object fields. allows .help() method on your types in console / repl to display the docs for live experimentation. the generated docs can be turned into a website with themes etc... example: https://lenscas.github.io/tealsql/
  4. Async possible. idk if async is useful here, but its possible, so worth mentioning.
  5. Here's a killer feature :) mlua actually supports passing non-static data as arguments for lua functions inside a scope . so, &T or &mut T (wrapped in a newtype) can actually be passed to a lua function and it will be safe because when the scope ends, lua will invalidate any references stored by the scripts so that no use after free errors occur.

The obvious benefits of lua already being used in neovim, easy to learn etc.. also apply, but i wanted to speak mainly from a practical point. you can probably have two types of plugins: sandboxed (with luau) and unsandboxed (with luajit). this will allow people to make their choice based on their security needs / concerns of installing plugins. the decision on whether to mirror neovim Api to allow existing neovim plugins be used in helix is a separate discussion.

may not be relevant, bevy is also getting third party plugin system via https://github.com/makspll/bevy_mod_scripting which uses Tealr / Mlua and Rhai. but will need to wait for bevy 0.8 (maybe in august) to use it.

Conclusion

It is cool to debate about wasm, but its nowhere ready to be used here (or anywhere) with a non-trivial plugin system. if WASI really delivers in the coming years, there is always the option to have a second plugin system. this issue has been open for more than an year with no progress. and i consider that a good thing. if work had been started when witx was recommended, all that would have been a waste of energy because witx is abandoned. and wit is still not standardized (discussions ongoing whether it should be standardized with Component model or separately).

just to add. there's rust implementations of different languages like artichoke (ruby), boa (js), hematita (lua), rustpython (python), goscript (go, but as a scripting lang) but none of them are stable / ready yet.

there's also bindings for external languages like quickjs, v8 / deno, nodejs, PyO3, julia etc.. but none of them come close to the polished experience that mlua / tealr can provide. I can vouch for them from experience as i am using them for my bevy app and made egui bindings (which is NOT EASY) to enable plugins to add their gui. there's still some manual work to be done like README or docs etc.. but the bindings themselves were not hard to deal with.

all of this only makes sense if we are going for a "embedding" api / plugin system where plugins have direct access to data of the editor. if we instead use a json rpc sort of system like vscode, languages hardly matter and even wasm becomes a possibility (well, only rust, but still it counts xD).

EDIT: wit and witx are separate things. in case anyone got confused by both of them being so similar, almost appearing like a typo

kirawi commented 2 years ago

Lisp was considered as a possibility in this note.

coderedart commented 2 years ago

Lisp was considered as a possibility in this note.

never thought people would consider lisp seriously. then again, having plugins is better than not having them, so i have no complaints. i still think lua is better than lisp due to its popularity though.

archseer commented 2 years ago

Lisp is quite popular too, there's this text editor that uses it extensively. Scheme has really good meta-programming because of it's macro system, but it also has a very small spec so it would be easier to maintain if we have to.

sigmaSd commented 2 years ago

I spent this week messing with nvim and lua, my personal feedback having an untyped language is a lot of pain to debug and the ide experience is very weak. I would personally vote for any typed language.

bemyak commented 2 years ago

Adding to what @coderedart was saying:

Initially, I was really excited about WebAssembly plugins idea, but I'm not so sure now. For programmer-oriented editor it is very important to be approachable and hackable with some reasonable amount of effort and knowledge.

Now, imagine that there is a plugin that you like and use daily, but turns out there is a bug that you'd want to fix. So you check out the source code, and (if we choose the wasm route) it can be written in anything, from Assembly through JavaScript to Brainfuck. So, you'll have to learn an arbitrary language to fix the nasty bug.

IMO, this will fracture the plugin ecosystem and make it way less approachable. People may start picking plugins based on languages, so the community effort will be spread out and a lot of it will be wasted.

Emacs is a great example of the opposite approach: the editor (mostly) and all plugins are written in EmacsLisp and this enables developers contribute both to the editor, any plugin and their own config files.

Thus, I believe we should start and probably stick to simple Rust dynamic libraries and require plugins to be written in Rust, since it's already the baseline for hacking on Helix.

coderedart commented 2 years ago

I spent this week messing with nvim and lua, my personal feedback having an untyped language is a lot of pain to debug and the ide experience is very weak. I would personally vote for any typed language.

tealr crate actually is designed for use with teal language which is lua with types. its compiler can be embedded into lua, so that you can just require teal source files directly too.

coderedart commented 2 years ago

@bemyak if plugins are only allowed to be written in rust (with C ABI), using wasm might be better than raw dynamic libs. raw dylibs can crash the main application by using unsafe, raw dylibs need to be compiled separately for each OS you want to support. cannot be easily hot reloaded etc.. wasm will allow us to avoid these issues while also providing a security benefit of sandboxing.

although the main issue with Rust is the lack of scripting. for each tiny change, you have to recompile the plugin. Rust is much harder compared to lua (or any scripting language) and will be huge hurdle for a user wanting to extend his editor.

jakenvac commented 2 years ago

if a scripting language is the preferred route, id like to throw Deno out there one more time.

I personally hate untyped languages so typescript is a great option for me while allowing people who prefer untyped languages to use JavaScript.

ribosomerocker commented 2 years ago

I suggest Lua, or any Common Lisp or Scheme. They could have great type systems, especially with the many implementations that create a more strongly and statically typed Lua and Lisps. Furthermore, requiring the usage of something like JavaScript or a JavaScript runtime decreases the usability and extensibility, in my opinion. And if you were to head for a bytecode approach, it isn't too difficult to create your own, rather than using WASI or WASM which might not fit the task at hand. Personally, I'd like to use Rust, Lisp, and also languages like Haskell or Idris 2 to create plugins, but that's just a fantasy I suppose, unless you either go with wasi or create your own bytecode (cresting your own is my preferred outcome); but if that's deemed too complex, I suggest using more simple languages that were designed for the purpose at hand, e.g. Lua, and not JavaScript. This is just my opinion though.

coderedart commented 2 years ago

it is not maintained at the moment, but if you want a lisp and not start from scratch again, there's https://gamelisp.rs/ .

reading https://gamelisp.rs/reference/rust-bindings.html page would give a good overview of how easy it is to embed into rust.

not recommending it, but just wanted to make sure you were aware and avoid doing the same work again.

archseer commented 2 years ago

IMO, this will fracture the plugin ecosystem and make it way less approachable. People may start picking plugins based on languages, so the community effort will be spread out and a lot of it will be wasted.

I agree, it initially seems promising but I think it would shard the ecosystem too much.

Thus, I believe we should start and probably stick to simple Rust dynamic libraries and require plugins to be written in Rust, since it's already the baseline for hacking on Helix.

Rust is not a great language for this type of scripting. Imagine having to configure helix via rust: you'd need a full Rust compiler available and any config change would mean a recompile. End users would also need the whole toolchain just to compile plugins. Dynlibs are platform specific and it's also risky to just run a random dynlib from some 3rd party. WASM would help here since we could compile and sandbox the rust code, but at that point... why not just extend the helix source code directly?

and all plugins are written in EmacsLisp and this enables developers contribute both to the editor, any plugin and their own config files.

Well, the core itself and elisp are written in C of course. So rather than pure Rust, Rust + Scheme would be the closer analogy here.

it is not maintained at the moment, but if you want a lisp and not start from scratch again, there's https://gamelisp.rs/ .

Yeah I did some prior-art research and while GameLisp is the most widely known implementation, but https://github.com/mattwparas/steel/tree/blow-up seems the most impressive. If it was more stable I'd likely have just used that.

lazytanuki commented 2 years ago

If we embed a scripting language into Helix, will we be able to properly sandbox it? I really like the sandboxing provided by the WASM infrastructure, and I believe sandboxing plug-ins should be a requirement for security reasons. For instance, a basic neovim config might need tens of plug-ins to be as functional as helix, but no one can be expected to have a look at the code diff for each update of each plug-in in case an author account as been compromised. I don't really want to have the same issue with helix.

bjorn-ove commented 2 years ago

Unfortunately I don't see how sandboxing of any kind will help. It may stop exploitation, but for a plugin system to be useful it must have access to featured that bypass the sandbox.

For example, spawning a process, connecting to the internet, accessing the filesystem and so on.

archseer commented 2 years ago

Well the most crucial bit is WASI's capability based system. We can always implement something similar that limits plugin's access to let's say project_files but not /usr/lib, ~/.ssh/ etc. Same thing for network access.

https://crates.io/crates/cap-std/

lazytanuki commented 2 years ago

Unfortunately I don't see how sandboxing of any kind will help. It may stop exploitation, but for a plugin system to be useful it must have access to featured that bypass the sandbox.

For example, spawning a process, connecting to the internet, accessing the filesystem and so on.

Sandboxing capabilities will certainly help, but of course they have to be adapted to each plugin, with a permission system for instance. The goal would be to respect the principle of least-privilege. Does that plugin need access to the filesystem? Does it need access to anymore files than the current workspace ? Should it be allowed network access ? Which binaries should it be allowed to execute ?

IMO this is perfectly doable if the plugin system is designed with it in mind. A text editor is pretty much impossible to sandbox from the outside, unlike a web browser for instance, so this should be the best solution.

gavynriebau commented 2 years ago

I want to question whether the scripting language rhai has been seriously considered?

It has a clean and simple syntax and would be very easy to embed in helix, here's an example code snippet for calculating Fibonacci sequence:

const TARGET = 28;
const REPEAT = 5;
const ANSWER = 317_811;

fn fib(n) {
    if n < 2 {
        n
    } else {
        fib(n-1) + fib(n-2)
    }
}

print(`Running Fibonacci(${TARGET}) x ${REPEAT} times...`);
print("Ready... Go!");

let result;
let now = timestamp();

for n in 0..REPEAT {
    result = fib(TARGET);
}

print(`Finished. Run time = ${now.elapsed} seconds.`);

print(`Fibonacci number #${TARGET} = ${result}`);

if result != ANSWER {
    print(`The answer is WRONG! Should be ${ANSWER}!`);
}

I would be happy to create a spike PR showing how it might work if there is appetite to see that.

/Tag @archseer

jakenvac commented 2 years ago

@gavynriebau

I want to question whether the scripting language rhai has been seriously considered?

Could you provide some reasons why it would make a good language over more common languages? With something like lua/perl/js there are lots of community libraries that we'd be able to make use of when writing plugins. How does rhai compare?

@archseer

Well the most crucial bit is WASI's capability based system. We can always implement something similar that limits plugin's access to let's say project_files but not /usr/lib, ~/.ssh/ etc. Same thing for network access.

I've suggested Deno as the runtime for the plugin system, but even if that's not the way we go, it's worth looking at how it implements its security policies. By default, scripts are sandbox and permissions must be explicitly granted at execution.

Philipp-M commented 2 years ago

I think the discussion shows, that we should still consider wasm, everyone has a different taste in programming languages (I personally would prefer a strong typed functional language like ML or from the Lisp-world: Racket, or just Rust). wasm has the advantage that there's an increasingly growing ecosystem around it, a lot of work has already been done, it's a solid VM, and likely the future for various different domains.

I think a well thought ABI with defined (native) functions to access the underlying data (without much serialization) is probably the most flexible way forward. A single language may limit people writing plugins who don't want to invest the time to learn a new language (especially if it's a new paradigm). Emacs is an example, it's widely used in the functional world, but outside of that it's not used that much (probably mostly because configuration is done in it).

I don't share the fear of a fractured ecosystem. I'll take the best plugin, I don't care in which language it's written in, as long as it gets the job done...

As for the approachability, it's easy to provide various templates for popular languages, to quickly start hacking plugins.

I think something like Lisp would only make sense, if pretty much everything is done with it in helix (i.e. configuration) like in emacs. But as stated above we would probably limit ourselves a bit.

p-e-w commented 2 years ago

Almost every comment in this thread is about how Helix should implement plugins, as opposed to whether it should do so. I urge people to reconsider that basic assumption.

I believe the main reason why we tend to think "a text editor must have a plugin system" is because most existing editors are unusable for serious work without half a dozen plugins. Editors like Vim, with decades of development behind them, treat Git integration, EditorConfig, even autocompletion(!), as optional nice-to-haves that should be provided by plugins. But those things aren't "nice-to-haves", they're an essential part of most professional developers' workflows.

VSCode was the first editor to get this right. It provides the entire standard workflow out of the box, without turning into a language-specific IDE. The only extensions I find myself installing when I use VSCode are those for individual languages, which are mostly wrappers around LSP servers. If Helix has good LSP+DAP support, and lots of configuration options, I need plugins like I need a hole in my head.

And this isn't just a "if you don't want plugins, don't use them" thing. A plugin system is a massive liability. Embedding a full WASM (or whatever) runtime, defining an API (probably versioned) which becomes a roadblock whenever you want to refactor internals, a plugin repository and auto-updater (I really want an editor that doesn't connect to some server every time I start it), possibly sandboxing, metadata, plugin dependency management...

None of the concrete use cases described in this thread seem like they are remotely worth this undertaking. Should Helix really become an operating system just so people can have a custom clipboard manager? Is there any "must-have" capability that can only be achieved by an integrated plugin system, as opposed to shelling out to an external program, or a configuration option, or an LSP feature?

archseer commented 2 years ago

Is there any "must-have" capability that can only be achieved by an integrated plugin system, as opposed to shelling out to an external program, or a configuration option, or an LSP feature?

Of course there is. For a pragmatic, unix-y approach where there is no plugin system and you shell out for additional functionality there's always kakoune. But there's a whole class of LSP features that are unofficial extensions specific to a single language server that wouldn't be possible to support without tighter integration.

I'd like to claim that plugins shouldn't be as necessary for helix as for other editors though, the experience out of the box should be enough for most users. Everyone's so used to terminal-based editors being tough to configure and require lots of settings, there was recently a user that was looking for a Helix config to download, (in vein of LunarVim or Spacemacs) -- that really shouldn't be necessary!

glyh commented 2 years ago

@p-e-w Could you list some editors without plugin system but it's still widely used?

nrabulinski commented 2 years ago

I agree that there is a need for a plugin system and that helix shouldn't have to concern itself with every possible LSP implementation besides what the standard defines (and maybe the more common extensions or implementations for more popular languages). Forcing everything to be upstreamed will put too much responsibility on helix maintainers while limiting customizability for the users and even if some of them are smaller features (like for example rainbow brackets) each come with their own set of edge-cases and they add up fast. And I say that despite being very satisfied with helix experience so far, especially out of the box compared to other editors and having used it as my main (and only) editor for months now.

bjorn-ove commented 2 years ago

One question to ask is why Helix is such a great editor. I worry that the answer includes the fact that there is no plugin system. Not directly, but indirectly. Once a plugin system is added there is no advantage left and you may as well go back to vim/emacs.

jakenvac commented 2 years ago

Better shell integration, hooks and a way to extend the LSP features would be a great compromise between no plugin system and a full runtime in my opinion.

I think not having a way to extend the LSP features would be detremental to Helix's success. There are many languages/toolchains that require slightly different functionality than the base LSP spec. You could argue that its an issue for those implementations but the matter of the fact is the LSP spec isn't perfect and a lot of projects are doing the best they can with it. The Deno lsp is a great example of this.

As for plugins to extend the editor in other ways, tighter shell integration would overcome many of the things people want. For example, I find the regex functionality in Helix leaves a lot to be desired. I find myself often using shell tools to account for this but it means quitting helix and relaunching.

If helix standardised the way it expects input and output from third party tools, we could wrap all kinds of tools in simple shell scripts to integrate them.

combine shell integration with hooks and it only gets more powerful with little effort.

This is very similar to Kakounes approach, except to me, kak feels like it cant decide if it wants a full plugin system or not so just uses the shell as a stop gap.

bugeats commented 2 years ago

Would it help frame the discussion if instead of referring to plugins, we use the word modules instead? Hear me out.

Architect the editor such that most interactivity is isolated into distinct modules that refer to a core API. Discover what the surface area of that API is. Keep it tidy and documented. Maybe use WASM & Rust, or maybe don't. Collect and curate these modules carefully.

Now here's the important part: if someone wants to add XYZ feature/support they do so by forking the project and submitting a PR with a new module added. The project's benevolent dictators and tastemakers keep the noise and riffraff out, while everyone else is able to easily and freely experiment.

bemyak commented 2 years ago

@bugeats that's an interesting idea. We have a quite successful monolithic kernel out there, so maybe a monolithic text editor is worth a shot as well?

theherk commented 2 years ago

We have a quite successful monolithic kernel out there, so maybe a monolithic text editor is worth a shot as well?

We have successful monolithic kernels precisely because modules can be loaded dynamically. Can you imagine how restrained we would be if we could not load drivers? Not that I'm saying there isn't merit to this idea regarding the editor, but if the analogy is extrapolated it actually supports that each person may need or prefer slightly different configuration. Managing and pulling patches from forks to get those benefits does not sound pleasant.

bemyak commented 2 years ago

@theherk dynamic modules load was introduced in 2.2. It's an important but not crucial feature. Linux is monolithic because all drivers and modules live in the main repo, they have responsible maintainers. If something is not maintained, it is deleted.

I can imagine Helex could take a similar approach: all plugins modules are contributed upstream, and then can be enabled through the config file if needed.

Pros:

Cons:

P.S. I use Zola static site generator, which has also chose the monolithic approach. It's a bit annoying that I can't have my obscure features mainlined, but so far it has really nice "all you need and ready to go" feeling

ym-han commented 2 years ago

Re the need for some sort of plugin or scripting system / API: It can be very helpful, indeed probably indispensable, for accessibility reasons. For instance, someone with RSI might want to be able to do both Helix/Kakoune style keyboard-driven editing along with Cursorless-with-Talon-style voice coding (see, e.g., https://www.youtube.com/watch?v=5mAzHGM2M0k for an example). But it would be impossible to implement something like Cursorless for Helix without a plugin system that allowed us to draw cursor hats and so forth.

jakenvac commented 2 years ago

That's a really good point. Accessibility should play a huge factor in this conversation.

I once worked with someone who used a screen reader and used a plugin for his editor that interpreted certain aspects of source code in a screen reader friendly way. For example, if I recall correctly, it figured out when whitespace was intended to be used as indentation so it didn't just shout 'space' in his ear eight times every row. Instead it would tell him what level of indentation he was in.

Obviously it would be impossible to cater for every impairment people may have to deal with, so an extensible editor helps in this situation.

On that front, however, one might argue that some accessibility features should be built in.

EpocSquadron commented 2 years ago

Does the arrival of wasmflow add to this conversation? It seems like it might make a neat ready to go packaging format for the plugins, although it is immature.

dcompoze commented 2 years ago

I don't share the fear of a fractured ecosystem. I'll take the best plugin, I don't care in which language it's written in, as long as it gets the job done...

I agree, I think generally the point of plugins is to use a ready-made solution and not have to write/modify the code yourself. And If there's a bug or an issue with the plugin, the best thing to do is usually to talk to the maintainers of the plugin.

That said, I think plugins are generally more useful for large applications - e.g. a browser, an IDE environment or a game engine - where the cost of editing the source code and contributing a new module or a feature is a lot higher compared with just writing an isolated plugin.

Personally, when it comes to code editors with a plugin system, having a significant number of plugins installed would always end up creating a messy experience for me, hence I would generally stick to just a couple of essential plugins and remove them once I had the time to patch the functionality directly into the editor.

Though I do agree that there needs to be a solution for code that is useful to have in general, but is too awkward to include in the core implementation of the editor (LSP extensions, accessibility features, custom tooling integrations etc.).

In my opinion, this is best solved at the level of static modules and conditional compilation instead of a Wasm plugin system, an embedded scripting language or even dynamically loaded modules (as these would have to either target multiple architectures or depend on the Rust compiler - more on that below).

The static module solution would provide:

This would avoid having to bundle a specific runtime for a scripting language or web assembly together with the editor (often larger in terms of LoC than Helix itself).

And would avoid the problems associated with loading and distributing dynamic Rust modules:

Disadvantages of static modules:

Package managers that don't support building from source would have to provide multiple packages with different features enabled e.g. helix-lsp-extra, or users would have to install Helix via cargo if they want to enable different modules.

Advantages of dynamic modules:

An argument could be made that Helix already bundles a C compiler dependency to compile the grammars with hx --grammar fetch and hx --grammar build, effectively fetching C source code and building it on the users' machine. Which could technically be re-used for dynamic module building and loading - though I'm not sure if anyone wants to write Helix modules in C.

Alternatively, it might not be a terrible idea to introduce a similar dependency on a local Rust compiler and add a corresponding hx --module fetch and hx --module build commands and store them under runtime/modules.

Though I think users are more likely to want to add/remove grammars dynamically at runtime than add/remove modules, hence the static approach might be better for modules.

I've suggested Deno as the runtime for the plugin system, but even if that's not the way we go, it's worth looking at how it implements its security policies. By default, scripts are sandbox and permissions must be explicitly granted at execution.

Using Deno as an embedded scripting language would require bundling a relatively large amounts of code with the editor:

Where Helix itself is around 36K LoC.

Also, TypeScript is a complete programming language which seems overkill if the goal is to embed a simple scripting language.

glyh commented 2 years ago

@dcompoze One more disadvantage is that if helix use local rust compiler to compile every plugin, it would be slow to update plugins.

aecepoglu commented 2 years ago

I use kakoune, an editor helix takes inspiration from. In kakoune a plugin may be written bash or in any language. I agree with both sides of the argument but I heavily favour the "choose your own language" approach.

I am a bit of a minimalist so I do prefer to omit plugins if they require dependencies that I don't want on my system. I'd hate to have my system polluted by infinitely deep node_modules directories, for example. Or if a simple plugin is written with python that I think I don't want a python dependency for, I will go out of my way to rewrite the plugin.

But it also means if I have any idea for what I want to do within my text/code editor, I have the means to be able to achieve it. My setup is wacky at times and I like it! I'll whip up something together in OCaml, in GuileScheme, in whatever I like to use. I'm more inclined to write something for it if I get to use my favourite programming language.

eulerdisk commented 2 years ago

Re the need for some sort of plugin or scripting system / API: It can be very helpful, indeed probably indispensable, for accessibility reasons. For instance, someone with RSI might want to be able to do both Helix/Kakoune style keyboard-driven editing along with Cursorless-with-Talon-style voice coding (see, e.g., https://www.youtube.com/watch?v=5mAzHGM2M0k for an example). But it would be impossible to implement something like Cursorless for Helix without a plugin system that allowed us to draw cursor hats and so forth.

I don't see with this should be an argument in favor of a generic plugin system. A builtin solution would always be better, expecially for something that needs a deep integration like I think accessibility is. On top of that, if some external tool needs to be integrated (e.g screenreader), I would go for a more LSP/DAP approach.

KotUW commented 2 years ago

I would like to point out the way micro does plugins. They are blobs (of lua, but that's not the point) that are downloaded from an official maintained repo. So they are highly curated and well tested.

But it also open the gates for anybody who wants to use there own plugin by simply dropping in a specific folder.

While giving maintainers a way to weigh on the experience.

stappersg commented 2 years ago

And another specification addition:

3339 and https://github.com/fregante/GhostText/blob/main/PROTOCOL.md explain why.

noor-tg commented 2 years ago

any news about helix plugin system ?

noor-tg commented 2 years ago

@adamcstephens @nrabulinski @christianfosli I know my comment was not productive . but the issue was opened from more than year ago. and there is not clear solution about the plugins yet. I checked some associated pull request about wasm plugins . but in the pull request comments one of the members wrote they are abandaning the wasm plugins. so what the alternative for wasm to be used for plugins system ? or what the current status for making plugin system for helix ?