WASI Key-Value Store

A proposed WebAssembly System Interface API.

Current Phase

WASI Key-Value is currently in Phase 2.

Champions

• Dan Chiarlone
• David Justice
• Jiaxiao Zhou
• Taylor Thomas

Portability Criteria

wasi:keyvalue must have at least two implementations in terms of relevant mainstream key-value stores in each of the following categories:

1. Open-source key-value stores like Redis, Memcached, Etcd etc.
2. Proprietary key-value stores like AWS DynamoDB, Azure CosmosDB, Google Cloud Firestore etc.

The implementations must be able to run on Linux, MacOS, and Windows.

A testsuite that passes on the platforms and implementations mentioned above.

Table of Contents

• WASI Key-Value Store
  • Current Phase
  • Champions
  • Portability Criteria
  • Table of Contents
  • Introduction
  • Goals
  • Non-goals
  • API walk-through
  • Working with the WIT
  • References \& acknowledgements
  • Change log

Introduction

This wasi:keyvalue proposal defines a collection of interfaces for interacting with key-value stores. It additionally defines a world, wasi:keyvalue/imports, that groups together common interfaces including

1. Store operations on single key-value pairs
2. Atomic operations on single key-value pairs
3. Batch operations on multiple key-value pairs

There is also an additional world called wasi:keyvalue/handle-watch, which includes the full imports world as well as a set of interfaces for watching for changes to a key-value store.

The API is designed to hide data-plane differences between different key-value stores. The control-plane behavior/functionality (e.g., including cluster management, data consistency, replication, and sharding) are not specified, and are provider specific.

[Interfaces]: https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md#wit-interfaces

Goals

The primary goal of this API is to allow WASI programs to access a wide variety of key-value stores. This means the wasi:keyvalue interfaces should be implementable by a wide variety of key-value stores, including but not limited to: in-memory key-value stores, on-disk key-value stores, document databases, relational databases, and either single-node or distributed key-value stores.

The second goal of wasi:keyvalue interfaces is to abstract away the network stack, allowing applications that need to access key-value stores to not worry about what network protocol is used to access the key-value store.

A third design goal of wasi:keyvalue interfaces is to allow performance optimizations without compromising the portability of the API. This includes minimizing the number of copies of data, minimizing the number of requests and round-trips, and support for asynchronous operations.

Non-goals

• The wasi:keyvalue interfaces only focus on the most common use cases for key-value stores. It does not attempt to cover all possible use cases.
• The wasi:keyvalue interfaces do not guarantee data consistency except as defined in the interface documentation. Data consistency models (eventual, strong, casual etc.) are diverse and are specific to each key-value store's architecture. The implication is that components importing wasi:keyvalue interfaces would need to be aware of the data consistency model of the key-value store they are using.
• The wasi:keyvalue interfaces do not handle data replication and sharding. These are control-plane behaviors that are specific to each key-value store's architecture. Replication needs and sharding management are left to the outside of the wasi:keyvalue interfaces.
• No cluster management is provided. Operational aspects like scaling, node management, and cluster health are to be managed separately from the wasi:keyvalue interface wasi:keyvalue interfaces.
• No built-in monitoring. Monitoring of key-value store performance and health should be done using external tools and not be expected as part of the wasi:keyvalue interfaces.

API walk-through

The proposal can be understood by first reading the comments of world.wit, store.wit, atomic.wit, and batch.wit.

Working with the WIT

Bindings can be generated from the wit directory via:

wit-bindgen c wit/ --world imports

and can be validated and otherwise manipulated via:

wasm-tools component wit wit/ ...

References & acknowledgements

This proposal was inspired by Dapr's [State API]

State API

Change log

• 2024-03-29:
  • Simplified interfaces to 3 key pieces of functionality: store, atomics, and batch. Also removed the use of streams to simplify the starting point of the API
• 2024-01-16:
  • Changed the readwrite and batch interface names to eventual and eventual-batch. See more details here.
• 2023-12-19:
  • Changed the size to incoming-value-size and it's signature
• 2023-11-30:
  • Changed the get and get-many and keys signatures
  • Updated comments in all the interfaces.
  • Renamed wasi-cloud-error to wasi-keyvalue-error
• 2023-05-17: Updated batch example to use one interface instead of 2
• 2023-05-25: Change the WITs to the newest syntax.
• 2023-02-13: The following changes were made to the API:
  • Added bucket type to the types interface.
  • Uses pseudo-stream and pseudo-resource types inspired from wasi-io
  • Added *.wit files for the interfaces and worlds, which are verified by the wasm-tools
  • Added a inbound-keyvalue interface that handles the watch operation.
• 2023-01-28: The following changes were made to the API:
  • Changed stream<T> type to list<T> type because stream<T> is not supported by the current version of *.wit.
  • Removed the worlds section because the star import syntax is not supported by the current version of *.wit
• 2022-11-29: Renamed bulk-get to get-multi and bulk-set to set-multi to be consistent with the naming convention of the other interfaces.
• 2022-10-31: Initial draft