add support for webworkers

[prabirshrestha]

useful for plugins to implement heavy processing such as fuzzy search.

Also need to implement TransferrableObjects for perf if possible. https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers#Passing_data_by_transferring_ownership_(transferable_objects)

[bobpepin]

Current trend seems to be to offload the heavy processing to external processes, e.g. LSP. Which makes sense as well when thinking of mobile platforms. But maybe WebWorkers API can still come in handy, I will have a closer look.

Especially since the world model of Vim seems to be single-threaded, and the Vim OS interface does not expose threads either. Also, the good old select() loop has finally got a nice API with await :)

[prabirshrestha]

Problem with external process is it gets chatty and using single thread like node also doesn't scale well when you need perform heavy calculations.

Here are few examples:

1. For autocomplete plugins. I want to add fuzzy search and more complex ranking algorithm. Doing this in the UI thread would cause vim to hang. Doing it in worker thread would make it more responsive. If I implement sources such as utilsnips in vim, I don't need to transfer the entire items to a different process. I can always treat is as immutable and use concept of ownership and only use for readonly allowing me to share the data across different threads.
2. In vim-lsp I need to compute diff of old and new. Doing this or other bigger computation would be cheaper than starting an entire process.
3. On smaller devices you might not be able to start too many process since it is heavy.

I still think it could be possible to have both worker and process and let the plugin authors decide what they want.

[bobpepin]

I agree, I have also been thinking about having simple LSP servers running inside Vim, for example something that replicates C-x C-n. I can see the use case for worker threads, and they should be available on most modern architectures and implementation should not be too hard if the interactions between worker threads and the main thread are sufficiently restricted (e.g. via sockets/pipes or something similar). Duktape easily supports multiple execution contexts, and even the same context active on multiple threads as long as two threads don't trigger automatic memory management at the same time. Of course this probably wouldn't work on Amiga or VMS ;)

[prabirshrestha]

Plugin authors can easily use globalThis.WebWorker to check if web worker exists or not.

[bobpepin]

• [x] Interface to socket listen() and accept() for the worker, using callbacks -> Promises -> async
• [x] Spawn a thread with a new Duktape context (pthreads)
• [x] Spawn a thread with a new Duktape context (Windows)
• [x] PoC with listen() in the worker thread and incoming connection from the main Vim thread
• [x] WebWorker API on top of the above
• [ ] Transferrable Objects

[bobpepin]

Basic functionality is there on Linux/MacOS, give it a spin if you are on one of those platforms.

[bobpepin]

... and Windows, Win64 binaries at https://github.com/bobpepin/vim/releases/tag/duk-win64-20190419T182100. To try (after putting Babel files into runtimepath)

:so lsp.vimrc
:dukfile workertest.js

wait until connected, then

:duk worker.postMessage("FOO"); Promise.runQueue()

[prabirshrestha]

This is awesome. I will be out for weeks on vacation. Would love to try this soon.

[prabirshrestha]

@bobpepin any plans to send an initial path to the upstream vim? I also filed an issue for wasm support in vim https://github.com/vim/vim/issues/4604. Wondering if that would be better since that would allows us to write in any language.

[bobpepin]

No immediate plans to submit to the upstream Vim, since I am quite busy with other things these days. The thing with only wasm is that it would either mean not having a REPL/Notebook interface or Vim would have to ship a compiler from some language to wasm. Also as far as I am aware the GC interface for wasm is still work in progress.