Decide on language server binary distribution

[edusporto]

The language server (LS) for Bend is currently written in Rust using the relevant libraries tower-lsp, tokio, and tree-sitter. We have now reached a stage we would like to publish the language server as an extension to VSCode, but this raises a technical challenge - how are we going to distribute the language server binary to users of the extension?

We have thought of four solutions, ranked by how desirable they would be to the team.

1. Compile the LS to WebAssembly (WASM) and embed it into the VSCode extension
   • Pros:
     1. Distribution is extremely easy
     2. Updating the LS is easy as simply an update to the extension
     3. Does not require users to install the Rust toolchain
   • Cons:
     1. Compiling to WASM is not straight-forward. tokio, which we currently use and could change, does have support for WASM, but it is a bit bare-bones and is missing a lot of features. The real problem though is tree-sitter, which is currently fundamental to our syntax highlighting and diagnostic reporting, and is not easy to compile into WASM.
2. Check on startup if the language server is installed, and if not, pull it and compile it automatically
   • Pros:
     1. Easy implementation
     2. Easy distribution
     3. Updating the LS is somewhat easy (just check for updates on every startup)
   • Cons:
     1. Requires the user to have the Rust toolchain installed
     2. Takes a while (not much) to compile, diminishes user experience
3. Check on startup if the language server is installed, and if not, pull a precompiled binary and install it automatically
   • Pros:
     1. Easy distribution
     2. Easy updates
     3. Does not require users to have the Rust toolchain installed
   • Cons:
     1. Hard to implement
        • Requires us to publish binaries to MacOS, Linux, Windows on arm64 and x86-64, with a total of 6 binaries
        • We will need to develop a few build and testing systems to get this working
4. Tell users to run cargo install bend-language-server
   • Pros:
     1. Very easy to implement (we don't implement anything!)
     2. Allows users to use any version of the LS they want
        • This is not really important at this stage of Bend's development, since it still is very early
   • Cons:
     1. Terrible user experience
     2. No way to guarantee updates

I am currently trying to get option 1 working.

[edusporto]

Updates on option 1 below. This is mostly me describing the most promising paths I took so I can try it again in the future when the ecosystem is more mature; to see the conclusion, skip to the next section.

What I tried

After attempting multiple ways to compile to WASM with tree-sitter support described in this thread, this amazing blog post, and this comment, we got the project compiling successfully to WebAssembly by disabling tokio's rt-multi-thread, installing wasi-sdk-24 into /opt/wasi-sdk-24.0, and finally, running the following command:

RUSTFLAGS='-L /opt/wasi-sdk-24.0/share/wasi-sysroot/lib/wasm32-wasip1 -lstatic=c++ -lstatic=c++abi' CXXSTDLIB=c++ CC=/opt/wasi-sdk-24.0/bin/clang CXX=/opt/wasi-sdk-24.0/bin/clang++ CXXFLAGS="-fno-exceptions" CFLAGS="-Wno-everything" cargo build --release --target=wasm32-wasip1

Unfortunately, this isn't enough with how the project currently works. We use the tower_lsp crate to handle the language server asynchronously, and tokio to boot the server. To boot up the server, we do this:

#[tokio::main(flavor = "current_thread")]
async fn main() {
    let stdin = tokio::io::stdin();
    let stdout = tokio::io::stdout();

    let (service, socket) = LspService::new(Backend::new);
    Server::new(stdin, stdout, socket).serve(service).await;
}

As tracked in tokio-rs/tokio#6516, the library does not currently have support for stdin/stdout in WASI, even though its standard allows it. One option to deal with this would be to have the server receive messages through TCP sockets, although I'm unsure on how to connect those to VSCode.

I've tried poking around tokio to get io-std working through rt-multi-thread based on discussions in the aformentioned tokio issue, and I found out our language server refuses to compile in wasm-wasip1-threads. Relevant discussions on this can be found in this issue and this chapter of the rustc book. Based on these, I created the following .cargo/config.toml file:

[unstable]
build-std = ['std', 'panic_abort', 'core', 'alloc']

[build]
target = "wasm32-wasip1-threads"
rustflags = '-C target-feature=+atomics,+bulk-memory,+mutable-globals,-crt-static'

and ran the following command:

RUSTFLAGS='-L /opt/wasi-sdk-24.0/share/wasi-sysroot/lib/wasm32-wasip1-threads -lstatic=c++ -lstatic=c++abi' CXXSTDLIB=c++ CC=/opt/wasi-sdk-24.0/bin/clang CXX=/opt/wasi-sdk-24.0/bin/clang++ CXXFLAGS="-fno-exceptions" CFLAGS="-Wno-everything" cargo +nightly build --release --target=wasm32-wasip1-threads

It fails during linking with the following error message:

  = note: rust-lld: error: --shared-memory is disallowed by 8df27ce4335e3b1c-lib.o because it was not compiled with 'atomics' or 'bulk-memory' features.

This should have been fixed by the config.toml file, as suggested in the issue mentioned, but I couldn't get it working.

---

Current attempt

Not all hope is lost. We could switch from using tower-lsp into using the lsp-server crate, which is not async and should definitely both work as WebAssembly and integrate well into VSCode, as we can see in this blog post from the VSCode team.

This will be my last attempt on the proposed solution 1, if it doesn't work, we will be trying solution 2.

[edusporto]

I followed the blog post from the VSCode team to try to reproduce their build, but unfortunately, that lead us to two problems:

1. If we try compiling to target wasm-wasip1-threads, we get the following error when VSCode starts the WASM process:

   starting generic LSP server
   thread 'main' panicked at /rustc/8e86c9567154dc5a9ada15ab196d23eae2bd7d89/library/std/src/thread/mod.rs:680:29:
   failed to spawn thread: Os { code: 6, kind: WouldBlock, message: "Resource temporarily unavailable" }
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

   This is possibly a fixable error, but the real problem is below.

2. If we try compiling to wasm-wasip1, we get:

   starting generic LSP server
   thread 'main' panicked at /rustc/8e86c9567154dc5a9ada15ab196d23eae2bd7d89/library/std/src/thread/mod.rs:680:29:
   failed to spawn thread: Error { kind: Unsupported, message: "operation not supported on this platform" }
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

   That's because lsp-server works with threads on its concurrency model. That means we're forced to target wasm-wasip1-threads if we use lsp-server. So, even if we fix error 1, we'll still have the problem I described in the previous comment in this issue when compiling to this target.

Unfortunately, the WASM/WASI technology is still not good and reliable enough for us to work with proposed solution 1. Maybe one day... For now, we'll work with solution 2.

[edusporto]

Just to be clear, if tokio-rs/tokio#6516 was solved, we would probably be able to go with solution 1.

[kings177]

Update

As explained above by edusporto, Solution 1 didn't work, so we have decided to proceed with Solution 2:

2. Check on startup if the language server is installed, and if not, pull it and compile it automatically
   • Pros:
   • Easy implementation
   • Easy distribution
   • Updating the LS is somewhat easy (just check for updates on every startup)
   • Cons:
   • Requires the user to have the Rust toolchain installed
   • Takes a while (not much) to compile, diminishes user experience

Regarding the Visual Studio Code extension, the plan is to release an MVP focusing on Debian-based and Darwin-based (MacOS) operating systems. full support for windows will be added at later dates as we refine the cross-platform functionality.

[edusporto]

Finished implementing solution 2 on commit 0f4a173. It works well on my MacOS machine and should probably work on Windows (at least with WSL enabled). Eventually we should add an automated test suite.