RFC: Plugin system

[milesj]

For proto to become the best toolchain manager in the ecosystem, it needs to support plugins. Plugins will allow consumers to implement custom languages (no need for it to be in proto itself), possibly implement non-language tools (like CLI binaries), hook into life-cycles, and more.

Enabling a plugin

Plugins can be enabled on a per-directory basis through the .prototools configuration file. This allows companies and projects to pin the plugins required for their repository to function correctly.

Plugins can also be enabled in ~/.proto/config.toml for the current user.

Since both of these configuration files are TOML, the plugins will be defined in a [plugins] table, which requires a mapping of tool name to plugin locator. A plugin locator is a string that combines a unique protocol to a location in which to fetch the plugin.

[plugins]
kotlin = "protocol:location"

The following protocols are supported:

schema: protocol

The schema: protocol defines the location of a TOML plugin schema. The location can be a file path (relative from the parent config), or an https URL.

[plugins]
# By path
kotlin = "schema:../../tools/kotlin.toml"
# By url
kotlin = "schema:https://raw.githubusercontent.com/kotlin/kotlin/master/proto-schema.toml"
# By url using a specific version
kotlin = "schema:https://raw.githubusercontent.com/kotlin/kotlin/9ec2503ce3324d637065f8f657b6efea0bbb81b7/proto-schema.toml"

source: protocol

The source: protocol defines the location of the compiled WASM plugin file. The location can be a file path (relative from the parent config), or an https URL.

[plugins]
# By path
kotlin = "source:../../tools/kotlin.wasm"
# By url
kotlin = "source:https://github.com/kotlin/kotlin/releases/latest/download/proto-plugin.wasm"
# By url using a specific version
kotlin = "source:https://github.com/kotlin/kotlin/releases/download/v0.5.0/proto-plugin.wasm"

github: protocol

The github: protocol defines the GitHub repository to download the WASM plugin file from. This protocol requires that the WASM file is added as an asset to each release, using a specific naming scheme.

[plugins]
# Default to latest
kotlin = "github:kotlin/kotlin"
# Using a specific version
kotlin = "github:kotlin/kotlin@v0.5.0"

Implementing a plugin

Because proto is written in Rust, supporting a runtime based plugin system is **difficult**. The only native solution to this problem is WASM plugins, which have the following concerns:

• Tooling to compile and build the WASM plugins must be stable.
• Plugins should be able to be written in any language that supports WASM, not just Rust.
• WASM is a foreign/unknown concept to many. Will most likely have a learning curve.

To mitigate these concerns and to ease the plugin implementation process, we’ll actually be supporting 2 types of plugins. The standard WASM plugin, and a TOML configuration schema plugin.

Configuration schema plugin

The schema plugin is a TOML file that defines all the necessary information for downloading, installing, and executing a tool. Since this is a configuration file, and all settings are defined literally, it only provides the bare minimum of functionality, and only supports pre-built tools and not build from source.

If more advanced functionality is required, then a WASM plugin will need to be used instead.

Here’s an example of what this schema may look like, using Node.js:

# PROOF OF CONCEPT, NOT FINAL!
name = "Node.js"
bin-name = "node"

[platform.os]
linux = "linux"
macos = "darwin"
windows = "win"

[platform.arch]
arm = "armv7l"
aarch64 = "arm64"
x86_64 = "x64"
x86 = "x86"

[resolve]
versions-url = "https://nodejs.org/dist/index.json"
version-selector = ".version"

[install]
archive-prefix = "node-v{version}-{os}-{arch}"
download-url = "https://nodejs.org/dist/v{version}/node-v{version}-{os}-{arch}{ext}"
checksum-url = "https://nodejs.org/dist/v{version}/SHASUMS256.txt"

[install.ext]
linux = ".tar.xz"
macos = ".zip"
windows = ".tar.xz"

[detect]
version-files = [".nvmrc", ".node-version"]

[execute.bin-path]
linux = "bin/node"
macos = "bin/node"
windows = "node.exe"

[commands]
install-global = "npm install -g {dependency}"

WASM plugin

For WASM plugins, we’ll be using Extism, a universal plugin system for multiple languages. On the host side (proto), we import the .wasm plugin file and execute functions provided by the plugin. On the guest side (the plugin), plugins can be written in multiple languages, like Rust, JavaScript, and Go, all of which define and export functions that compile to .wasm.

An example of what this may look like in Rust:

use extism_pdk::*;
use serde::Deserialize;
use std::env::consts;

#[derive(Deserialize)]
struct DownloadParams {
  version: String,
}

#[plugin_fn]
pub fn get_download_url(input: String) -> String {
    let params: DownloadParams = serde_json::from_str(&input).unwrap();

    format!(
        "https://nodejs.org/dist/v{version}/node-v{version}-{os}-{arch}{ext}", 
        version = params.version, 
        os = consts::OS,
        arch = consts::ARCH,
        ext = ".tar.xz"
    )
}

And again in JavaScript (not tested):

export function get_download_url() {
    const params = JSON.parse(Host.inputString());

    Host.outputString(
        `https://nodejs.org/dist/v${params.version}/node-v${params.version}-${process.platform}-${process.arch}.tar.xz`
    );
}

Extism is a good choice for the following reasons:

• It provides libraries/SDKs for both plugin authors and for plugin loader (proto).
• It supports multiple languages and doesn’t lock authors into Rust.
• We’re not reinventing the wheel.
• It’s maintained by a company and supports a community.
• Supports async functions.

[IgnisDa]

Really awesome proposal!

Concerning "Configuration schema plugin", I do not think it would be powerful enough. Writing one for nodejs might work, but will it be able to support other languages? How would you write one for Zig, for example?

Just exploring options here.

[milesj]

@IgnisDa As stated, the schema isn't meant to be powerful, just enough to support very common setups. I feel like this would be used more for non-language based tools.

FWIW, Node.js, Bun, Deno, and Go could all be supported with the schema format.

[elmpp]

Quick check here: would there be any common ground between this plugin system and Moon's (if this becomes a thing)

It would be super nice if the same semantics for Proto could somehow appear (in terms of DX) to be a subset of Moon's. In my understanding, Proto is just fulfilling a project's runtime requirements which I think of as a particular lifecycle.

If I could define other lifecycle in the same way it'd be much smoother imo.

[milesj]

@elmpp The outcome of this will definitely decide how moon plugins work, specifically for tiered language support.

Based on my understanding of WASM, the same WASM plugin can be used for both moon and proto, as long as the function names don't collide. This would be pretty great in supporting tier 2 and 3 rather easily.

[milesj]

The schema config turned out pretty great: https://github.com/moonrepo/proto/pull/74

[milesj]

TOML plugins have landed: https://moonrepo.dev/docs/proto/plugins#toml-schema-plugin