Atomic Plugins, Apps, Store

[joepio]

Being able to extend an application at runtime is something that only few systems can do. Operating Systems and Browsers are some of the few, but there are also plenty of apps that allow plugins (Raycast, VSCode, Wordpress). This is part of what makes these systems incredibly powerful and versatile. I'd love an Atomic Server to be similar to these, in some regards: it should be able to run apps, which should be able to do pretty much whatever a developer might want.

Atomic Server provides a really cool extra bonus: It runs on the web! You can access your atomic server and all its data and apps from any device.

In order to make this plugin abstraction well designed, we should extensively use it internally, during development of (core) features.

Naming: apps or plugins?

• Apps sound more powerful, Plugins are more technical
• Apps feel like they take over the entire screen
• Plugins add functionality to something that already exists

I'll mostly use plugin from now on.

Plugin or Core?

One of the hardest questions when I'm trying to extend Atomic-Server, is whether it's a Plugin, or whether it's Core functionality.

• Core is available on any atomic server. This means that a Plugin can always use Core features.

Some usecases

Let's describe some apps and try to identify which kind of abstractions that would be required to realize this.

Versioning / history #42

• Host endpoint for queries (fetch version x of resource y, show all commits for resource y, etc.)
• Extend existing resources (not sure if this is necessary, but it might come in handy for adding a versions property to existing resources, which increases discoverability)
• Custom views (e.g. a version browser, like hackMD or Google Docs has)
• Might be part of the core API instead of plugin, but trying to design this as a plugin basically helps us to design a powerful abstraction layer.

Auditing #95

• Add properties to resources that relate to when and by whom they have been edited.
• Has a dependency on Commits, and could interoperate or overlap with the Versioning plugin (e.g. audit log = history log)

Calendar app

• Host a front-end app (for handling browser requests)
• Read iCal from external calendar, perhaps periodically
• Convert internal calendar to .ical format (custom endpoint?)

Full text search #40

• Store and access an index optimized for full-text search (k/v access? Disk access?)
• Re-build / modify index on state changes in main store (access to Commits, as they happen?)
• Host an endpoint for queries
• Host a front-end app
• Should maybe be core. I've already built it, anyway.

IPFS storage #66

• Command for 'freezing' resources on request. Needs authorization.
• Resolve urls that start with ifps
• Run IPFS conversion logic
• Store to disk (K/V api, or maybe provide access to disk? rust-ipfs uses sled, so that might make things easier in this case)
• Network access / connect to swarm

Sys monitor

• HTML page that refreshes every second
• Shows hardware system stats like memory usage, CPU, disk size
• Possibly shows running plugins, option to kill processes.

SPARQL endpoint

• Use something like Oxigraph as a library. It features both a MemoryStore as well as two persistent options, one using Sled (which this project uses!) and one using RocksDB
• User should be able to have some form of authorization
• Maybe serve an HTML + JS browser app
• Create commits from SPARQL UPDATE queries. This means that the SPARQL endpoint must be able to sign commits. Maybe because it has a Private Key, or because it has some token that is publicly verifiable and only usable in this scope?

Simple CRUD models endpoints

• an HTTP endpoint for CRUD operations for one specific class, which feels like a very common JSON endpoint.
• POST plain JSON objects (with shortnames as keys) to some endpoint, the plugin creates Commits.
• Perhaps some authorization token to include?

TODO MVC App

• A simple HTML + JS webapp with some client side logic that uses an Atomic Server, possibly the CRUD example from above?

Runtime options

WASM

I'm very enthousiastic about WASM runtimes, such as Wasmer and WasmEdge. WASM is a compilation target for many languages, which means that many developers could write for it. It can be very performant, just a little short of full native. WASM executables can be shared as byte arrays. A wasm runtime can be sandboxed, and might be given access to certain system abscrations (e.g. filesystem / networking). The WASI spec aims to standardize these types of system interfaces.

Some projects for inspiration, that have implemented a WASI / WASM plugin system:

• Feather which has a wasmer plugin system called quill. It stores system data using wasmer::ctx.
• Zellij (plugin docs, examples)
• Tremor, blog
• proxy-wasm
• Lapce (code editor) plugin code
• Fiberplane, with fp-bindgen
• Spin is a framework for web servers that uses WASM modules. Check out how HTTP modules work. (cool talk!)
• Ambient (new 3d game engine), see crate. Using wit-bindgen

WASM + fp-bindgen

• Nice API
• Backed by a company that's making it an essential part of their stack
• Similar goals (plugin system in rust context, used for importers)
• Has some serialisation macros, they might support the atomic_lib::Resource type, as it supports HashMap and most low-level types. Cool stuff!
• See PR #361
• Doesn't support calling host-functions from plugin.
• Believes that a .wit based approach is the future, but fp-bindgen is nice for the short term.

wasmtime + wit

• .wit files are type definitions for modules that can be used in the wasmtime runtime to call typed WASM functions using a host-plugin system.
• https://github.com/bytecodealliance/wit-bindgen
• Should, on the long run, support many languages (contrary to fp-bingen which is rust only)
• Spin example
• witgen can be used to generate the .wit files from rust, although it's currently outdated.

WASMER + WAI

https://wasmer.io/posts/wasmer-takes-webassembly-libraries-manistream-with-wai

Fork of wit-bindgen. See issue.

A key difference between WAI and wit-bindgen is the focus on stability - people should be able to start using WAI right now.

EXTISM

https://extism.org/

Interesting project with nice docs, but it doesn't seem to support calling host functions from within a plugin. Update: just read in discord that calling host functions is now possible in one branch. Uses unsafe, though.

JS

JS is very popular and a ton of modules can be used.

I've explored QuickJS-rs, but kind of got stuck because I don't think it's possible to limit the runtime (e.g. no network access). There is no fetch.

WasmEdge also has JS execution by porting QuickJS to WASM, but does provide ways to limit access of the runtime to networking, for example. It also does have fetch support.

I've also experimented with GreenCopperRuntime, which is really amazing, but also a bit too young to use.

Plugin API

Installation / registration

Adding a plugin should be as simple as possible. Ideally, it can be done using a web API without rebooting the server.

• Owner posts a commit to the server, describing a Installation. This commit contains a (ipfs link to the) WASM byte array containing the logic. It optionally contains a link to a Config instance, which should be applied in another Commit.
• Server checks commit, applies it
• Server checks code for correct version. If
• Server checks code for registration hooks, so the server knowns when the plugin needs to be called. (e.g. periodic register, specific endpoint, specific class). These registrations are probably saved in memory, but they also need to be constructed when the server boots. Maybe store registrations on disk first?
• WASM code is loaded into memory, so it can be called when necessary
• Functions are called when needed

Configuring a plugin

• An Installation resource has an optional configuration field, which links to a Configuration Resource.
• These can be edited like any other resource.
• When a configuration is updated, the Plugin should re-initialize / re-load. Not sure how this should work, but I'll probably need some sort of callback handler thing that listens to changes to specific resources.

Removing a plugin

• Another Commit is made, which removes the plugin
• The registrations are removed.
• The WASM code is loaded out of memory.

Updating a plugin

• Another Commit is made which

When the code can be called

• Periodically: similar to cron jobs. Maybe the plugin should be able to determine which handlers (e.g. every 10 minuten). Perhaps reserve one thread for this - or find a more elegant solution with actix actors.
• On specific endpoints: e.g. /ipfs
• Before returning instances of Classes: e.g. all Invites, which uses query params similar to Endpoints
• Before applying a Commit: e.g. before saving an Invite, you need to check the Rights of the one making the Commit.
• When certain resource conditions are met: (e.g. if a certain class or property is present)
• By other plugins: See below.

Inter-plugin dependencies

• Plugins might call methods from other plugins, and therefore have dependencies to other plugins. For example, the Calendar plugin might use the versions plugin to undo changes to calendar items.
• Not sure how this will work, but it seems like a really powerful thing.

Front-end

Users are very likely to extend the views, perhaps even more than the back-end. Should a Plugin host it's own JS + HTML? Should it link to some URL of a component? Does it depend on Atomic-Data-Browser, or not per say?

[joepio]

The Mosaic rust project has an interesting WASM powered plugin system.

[joepio]

I'd like to make start on running some external code by creating a mock plugin, compiling it to wasm, registering it to an Atomic-Server and running the plugin. I'd like this to be a really minimal plugin, without any store interactions. Perhaps have an endpoint that accepts one argument, which is passed to the WASM function.

[joepio]

The minimal example worked fine, but now I'd really like to do things with the Store (e.g. get some data, calculate some numbers, return a new resource with these numbers), and that poses a few challenges:

• Crossing the Wasmer-Rust memory boundary seems necessary for providing a nice API for developers. Ideally, I'd pass the entire Store, but doing so is not easy
• If I'm going to use function from Atomic Lib (e.g. Store methods), I'm going to have to add WASM as a compilation target. #76.

Maybe I should consider a scripting language such as Rhai?

[joepio]

I'm currently working on implementing Search, which I wanted to do as a plugin from the start. However, my existing plugin abstraction was surely not powerful enough. For search, I need:

• Async runtime (index in the background)
• Access to commits on the fly
• Disk access (for custom index)
• Custom HTTP endpoints (but this could be done using the existing abstraction, though)

[joepio]

I should also consider using wasmedge, which also features a JS runtime. This is interesting, as JS is such a familiar language for many devs, probably more so than any WASM compilable language.

But how would I use Atomic Data in a JS runtime?

Use wasmedge's HTTP library

https://github.com/second-state/wasmedge-quickjs/blob/main/example_js/http_demo.js

It's possible, but it will take a long time to develop, because I don't think I can use @tomic/lib here. Maybe if I add a few methods for using custom fetch functions, maybe things will work out.

See https://github.com/second-state/wasmedge-quickjs/issues/30

WASI interface

The wasm / wasi interface only allows for passing Vec<u8>, so I can't do things like pass the Store or a Resource from rust. I can, however, pass a UTF-8 string as bytes representing a JSON-AD string, for example. I think that would be pretty quick.

[joepio]

WasmEdge now supports the fetch API, which should make things easier. Definitely should try this.

Read https://www.secondstate.io/articles/embed-javascript-in-rust/

[joepio]

comment moved to OP

[joepio]

I've been playing with fp-bindgen for a couple of hours now, and I think this feels like the most realistic approach for me so far. It works, I can move Rust strucs across the app-plugin boundary, I can export and import functions... Pretty sweet stuff.

Now I have to think about some other questions:

Where to store runtimes

A Runtime in fp-bindgen contains one WASM module. This means that we need multiple. I'd like to be able to instantiate these at runtime, for example by posting a Commit to some specific collection.

Options:

• Use sled! We like sled. It's pretty fast, it's thread-safe. Persists changes, even if the store is turned off. Requires parsing / deserializing when reading. May be a bit slower than memory-only solution.
• Use Arc<RWLock<HashMap>>! Also fast, thread-safe. No persistence. No parsing. Really really fast.

Do we want the Runtimes to persist on a database? I think that may lead to corrupt Runtimes if we update things. And it's slower. Hmm. No sled.

So it's part of the Store, I think? We clone it between threads when initializing the server. We sometimes need a lock for when we install a new plugin, but not when reading.

We need some sort of install function. And an uninstall function. Should it be on the Store struct? Maybe it will make Store too big, it already has (too) many methods.

What to expose to the Runtime

I can define a bunch of structs and functions that will be made available to the runtime. However, I won't be able to move references across the boundary. This means that I can't pass Store, for example, and make all functions available from there. I think every thing that I make available in the runtime, requires some manual work and possibly some code duplication.

Things we are very likely to need:

• get_resource, returns one resource
• queryreturns a set of resources
• save stores a bunch of JSON-AD

Use atomic_lib in plugins

One radically different approach is to import atomic_lib itself. This means that the runtime will have an in-memory Store. It will provide ways to create Commits, for example, and use all of Resource's methods. But it will come at a size cost, of course.

[joepio]

https://wasmer.io/posts/wasmer-takes-webassembly-libraries-manistream-with-wai

[joepio]

The Zed editor now supports extensions.

• There's a central repo for all extensions which has a lot of git submodules
• Extensions use the API which means implementing the extension Trait. Mostly focused on language servers.