Irmin is based on distributed version-control systems (DVCs), extensively used in software development to enable developers to keep track of change provenance and expose modifications in the source code. Irmin applies DVC's principles to large-scale distributed data and exposes similar functions to Git (clone, push, pull, branch, rebase). It is highly customizable: users can define their types to store application-specific values and define custom storage layers (in memory, on disk, in a remote Redis database, in the browser, etc.). The Git workflow was initially designed for humans to manage changes within source code. Irmin scales this to handle automatic programs performing a very high number of operations per second, with a fully automated handling of update conflicts. Finally, Irmin exposes an event-driven API to define programmable dynamic behaviours and to program distributed dataflow pipelines.
Irmin was created at the University of Cambridge in 2013 to be the default storage layer for MirageOS applications (both to store and orchestrate unikernel binaries and the data that these unikernels are using). As such, Irmin is not, strictly speaking, a complete database engine. Instead, similarly to other MirageOS components, it is a collection of libraries designed to solve different flavours of the challenges raised by the CAP Theorem. Each application can select the right combination of libraries to solve its particular distributed problem.
Irmin consists of a core of well-defined low-level data structures that specify how data should be persisted and be shared across nodes. It defines algorithms for efficient synchronization of those distributed low-level constructs. It also builds a collection of higher-level data structures that developers can use without knowing precisely how Irmin works underneath. Some of these components even have a formal semantics, including [Conflict-free Replicated Data-Types (CRDT)][]. Since it's a part of MirageOS, Irmin does not make strong assumptions about the OS environment that it runs in. This makes the system very portable: it works well for in-memory databases and slower persistent serialization such as SSDs, hard drives, web browser local storage, or even the Git file format.
Irmin is primarily developed and maintained by Tarides, with contributions from many contributors from various organizations. External maintainers and contributors are welcome.
ppx_irmin
irmin-git
uses an on-disk format that can be
inspected and modified using GitAPI documentation can be found online at https://mirage.github.io/irmin
Please ensure to install the minimum opam
and ocaml
versions. Find the latest
version and install instructions on ocaml.org.
To install Irmin with the command-line tool and all unix backends using opam
:
opam install irmin-cli
A minimal installation containing the reference in-memory backend can be installed by running:
opam install irmin
The following packages have are available on opam
:
irmin
- the base package, plus an in-memory storage implementationirmin-chunk
- chunked storageirmin-cli
- a simple command-line toolirmin-fs
- filesystem-based storage using bin_prot
irmin-git
- Git compatible storageirmin-graphql
- GraphQL serverirmin-mirage
- mirage compatibilityirmin-mirage-git
- Git compatible storage for mirageirmin-mirage-graphql
- mirage compatible GraphQL serverirmin-pack
- compressed, on-disk, posix backendppx_irmin
- PPX deriver for Irmin content types (see README_PPX.md)irmin-containers
- collection of simple, ready-to-use mergeable data structuresTo install a specific package, simply run:
opam install <package-name>
To install the development version of Irmin in your current opam switch
, clone
this repository and opam install
the packages inside:
git clone https://github.com/mirage/irmin
cd irmin/
opam install .
Below is a simple example of setting a key and getting the value out of a Git-based, filesystem-backed store.
open Lwt.Syntax
(* Irmin store with string contents *)
module Store = Irmin_git_unix.FS.KV (Irmin.Contents.String)
(* Database configuration *)
let config = Irmin_git.config ~bare:true "/tmp/irmin/test"
(* Commit author *)
let author = "Example <example@example.com>"
(* Commit information *)
let info fmt = Irmin_git_unix.info ~author fmt
let main =
(* Open the repo *)
let* repo = Store.Repo.v config in
(* Load the main branch *)
let* t = Store.main repo in
(* Set key "foo/bar" to "testing 123" *)
let* () =
Store.set_exn t ~info:(info "Updating foo/bar") [ "foo"; "bar" ]
"testing 123"
in
(* Get key "foo/bar" and print it to stdout *)
let+ x = Store.get t [ "foo"; "bar" ] in
Printf.printf "foo/bar => '%s'\n" x
(* Run the program *)
let () = Lwt_main.run main
The example is contained in examples/readme.ml It can be compiled and executed with dune:
$ dune build examples/readme.exe
$ dune exec examples/readme.exe
foo/bar => 'testing 123'
The examples directory also contains more advanced examples, which can be executed in the same way.
The same thing can also be accomplished using irmin
, the command-line
application installed with irmin-cli
, by running:
$ echo "root: ." > irmin.yml
$ irmin init
$ irmin set foo/bar "testing 123"
$ irmin get foo/bar
testing 123
irmin.yml
allows for irmin
flags to be set on a per-directory basis. You
can also set flags globally using $HOME/.irmin/config.yml
. Run
irmin help irmin.yml
for further details.
Also see irmin --help
for list of all commands and either
irmin <command> --help
or irmin help <command>
for more help with a
specific command.
Irmin's initial desing is directly inspired from XenStore, with:
In 2014, the first release of Irmin was announced part of the MirageOS 2.0 release here. Since then, several projects started using and improving Irmin. These can roughly be split into 3 categories: (i) use Irmin as a portable, structured key-value store (with expressive, mergeable types); (ii) use Irmin as distributed database (with a customizable consistency semantics) and (iii) an event-driven dataflow engine.
Feel free to report any issues using the GitHub bugtracker.
See the LICENSE file.
Development of Irmin was supported in part by the EU FP7 User-Centric Networking project, Grant No. 611001.