[[https://clojurians.slack.com/archives/CB7GJAN0L][https://img.shields.io/badge/slack-join_chat-brightgreen.svg]] [[https://clojars.org/io.replikativ/konserve][https://img.shields.io/clojars/v/io.replikativ/konserve.svg]] [[https://circleci.com/gh/replikativ/konserve][https://circleci.com/gh/replikativ/konserve.svg?style=shield]] [[https://github.com/replikativ/konserve/tree/development][https://img.shields.io/github/last-commit/replikativ/konserve/main.svg]]
[[https://whilo.github.io/old/articles/16/unified-storage-io][Simple durability, made flexible.]]
A simple document store protocol defined with synchronous and [[https://github.com/clojure/core.async][core.async]] semantics to allow Clojuresque collection operations on associative key-value stores, both from Clojure and ClojureScript for different backends. Data is generally serialized with [[https://github.com/edn-format/edn][edn]] semantics or, if supported, as native binary blobs and can be accessed similarly to =clojure.core= functions =get-in=, =assoc-in= and =update-in=. =update-in= especially allows to run functions atomically and returns old and new value. Each operation is run atomically and must be consistent (in fact ACID), but further consistency is not supported (Riak, CouchDB and many scalable solutions don't have transactions over keys for that reason). This is meant to be a building block for more sophisticated storage solutions (Datomic also builds on kv-stores). A simple append-log for fast write operations is also implemented.
** Features :PROPERTIES: :CUSTOM_ID: h:115591f9-90d2-4c25-8499-6f53a8ae4bc6 :END:
*** Garbage Collector :PROPERTIES: :CUSTOM_ID: h:5529aa34-11b1-4499-bf62-7fc7be2b8a12 :END:
Konserve has a garbage collector that can be called manually when the store gets too crowded. For that, the function =konserve.gc/sweep!= allows you to provide a cut-off date to evict old keys and a whitelist for keys that should be kept.
*** Error handling :PROPERTIES: :CUSTOM_ID: h:10edb2cf-b2fc-4cc5-8854-77e6e8a1d82d :END:
For synchronous execution normal exceptions will be thrown. For asynchronous error handling we follow the semantics of =go-try= and =<?= introduced [[https://swannodette.github.io/2013/08/31/asynchronous-error-handling][here]]. We have the [[https://github.com/replikativ/superv.async/][superv.async]] library around the error handling in core.async, but since there is no need to push it onto the users of konserve, you just need these two macros that properly handle the errors. =<?= needs to check for an exception and rethrow and =go-try= needs to catch and pass it along as a return value such that it does not get lost.
** Usage :PROPERTIES: :CUSTOM_ID: h:07b8872b-1b84-412b-8133-4dbb9d2a7430 :END:
Add to your dependencies: [[http://clojars.org/io.replikativ/konserve][http://clojars.org/io.replikativ/konserve/latest-version.svg]]
*** Synchronous Execution :PROPERTIES: :CUSTOM_ID: h:e290028c-78d8-4af6-8742-18b6d46680e3 :END:
Run the following synchronous code if you are not using core.async in your scope:
(ns test-db (:require [konserve.filestore :refer [connect-fs-store]] [konserve.core :as k]))
(def store (connect-fs-store "/tmp/store" :opts {:sync? true}))
(k/assoc-in store ["foo" :bar] {:foo "baz"} {:sync? true}) (k/get-in store ["foo"] nil {:sync? true}) (k/exists? store "foo" {:sync? true})
(k/assoc-in store [:bar] 42 {:sync? true}) (k/update-in store [:bar] inc {:sync? true}) (k/get-in store [:bar] nil {:sync? true}) (k/dissoc store :bar {:sync? true})
(k/append store :error-log {:type :horrible} {:sync? true}) (k/log store :error-log {:sync? true})
(let [ba (byte-array (* 10 1024 1024) (byte 42))] (time (k/bassoc store "banana" ba {:sync? true})))
(k/bget store "banana" (fn [{is :input-stream}] (your-read-does-all-work-here is)) {:sync? true})
*** Asynchronous Execution :PROPERTIES: :CUSTOM_ID: h:929c501d-2a31-4f05-b231-132f79ee6cb5 :END:
In a ClojureScript REPL you can evaluate the expressions from the REPL each wrapped in a go-block.
(ns test-db (:require [konserve.memory :refer [new-mem-store]] [clojure.core.async :refer [go <!]]))
(go (def my-db (<! (new-mem-store)))) ;; or (go (def my-db (<!
From a Clojure REPL run the following functions for the core.async variants of the code.
(ns test-db (:require [konserve.filestore :refer [connect-fs-store]] [konserve.core :as k] [clojure.core.async :refer [go <!]]))
(go (def store (<! (connect-fs-store "/tmp/store")))
(<! (k/assoc-in store ["foo" :bar] {:foo "baz"}))
(<! (k/get-in store ["foo"]))
(<! (k/exists? store "foo"))
(<! (k/assoc-in store [:bar] 42))
(<! (k/update-in store [:bar] inc))
(<! (k/get-in store [:bar]))
(<! (k/dissoc store :bar))
(<! (k/append store :error-log {:type :horrible}))
(<! (k/log store :error-log))
(let [ba (byte-array (* 10 1024 1024) (byte 42))]
(time (<! (k/bassoc store "banana" ba)))))** Supported Backends :PROPERTIES: :CUSTOM_ID: h:387ed727-24da-41df-b0f6-cfa03f95bbdd :END:
*** In-Memory Store :PROPERTIES: :CUSTOM_ID: h:63d979c0-4c4b-41fd-b1e2-e447adee3908 :END:
For simple purposes a memory store wrapping an Atom is implemented for Clojure and ClojureScript.
Usage:
(ns test-db (:require [konserve.memory :refer [new-mem-store]] [konserve.core :as k]))
(def my-db (new-mem-store))
*** fs-store :PROPERTIES: :CUSTOM_ID: h:c88f8eb7-27b1-46ff-bc64-918dd1eb30bc :END:
A file-system store in Clojure and for Node are provided as elementary reference implementations for the two most important platforms. No setup and no additional dependencies are needed.
The file-system store currently uses [[https://github.com/clojure/data.fressian][fressian]] in Clojure and [[https://github.com/pkpkpk/fress][fress]] in ClojureScript and is quite efficient. Both implementations use the same on-disk format and can load the same store (but not concurrently). It also allows to access values as a normal file-system file, e.g. to open it with a native database like HDF5 in Java. You can decide not to fsync on every write by a configuration of ={:sync-blob? false}=, if a potential, but unlikely data loss is not critical for you (e.g. for a session store). Note that the database will not be corrupted in this case, you can just lose some write operations before the crash.
Usage:
(ns test-db (:require [#?(:clj konserve.filestore :cljs konserve.node-filestore) :refer [connect-fs-store]] [konserve.core :as k]))
(def my-folder "path/to/folder") (def my-db (connect-fs-store my-folder))
*** IndexedDB :PROPERTIES: :CUSTOM_ID: h:ccbb272e-24b1-4f1e-b525-dd07c4e0e9b4 :END:
[[https://developer.mozilla.org/en-US/docs/IndexedDB][IndexedDB]] is provided as reference implementation for ClojureScript browser backends. The IndexedDB store is restricted to the async api only.
Usage:
(ns test-db (:require [clojure.core.async :refer [go]] [konserve.indexeddb :refer [connect-idb-store]] [konserve.core :as k]))
(go (def my-idb-store (<! (connect-idb-store "example-db"))))
*** External Backends :PROPERTIES: :CUSTOM_ID: h:a8505bd7-5e7a-4e1c-a851-20f11ca9affe :END:
We recently updated konserve not only to provide the option to choose between synchronous and asynchronous execution, but also to provide protocols that simplify the implementation of external backends. Unfortunately most of the external backends are deprecated now because of this.
**** Supported backends
Please let us know if you are interested in other backends or if you need help with implementing one.
**** Unofficial backends
**** Outdated backends
The following projects are incompatible with the latest konserve release, but describe the usage of the underlying store API and could still be helpful to implement new backends for the underlying store:
** Serialization formats :PROPERTIES: :CUSTOM_ID: h:a4cf3b14-1275-42d4-88f2-89fefb5c6085 :END:
Different formats for =edn= serialization like [[https://github.com/clojure/data.fressian][fressian]], [[http://blog.cognitect.com/blog/2014/7/22/transit][transit]] or a simple =pr-str= version are supported and can be combined with different stores. Stores have a reasonable default setting. You can also extend the serialization protocol to other formats if you need it. You can provide [[https://github.com/replikativ/incognito][incognito]] support for records, if you need them.
*** Tagged Literals :PROPERTIES: :CUSTOM_ID: h:1beb2a17-ca92-42b1-b909-1d043e3d81f6 :END:
You can read and write custom records according to [[https://github.com/replikativ/incognito][incognito]].
** Compression and encryption :PROPERTIES: :CUSTOM_ID: h:98bf90fd-4778-49da-80d7-58f89f00aec5 :END:
Compression and encryption are supported by the default store implementation that is used by all current backends. They can be activated in the store configuration as follows:
{:encryptor {:type :aes :key "s3cr3t"} :compressor {:type :lz4}}
LZ4 compression is currently only supported on the JVM. AES encryption is supported on both JVM and JS targets with the same cold storage format, i.e. the same store can be read and written from Clojure and ClojureScript runtimes. We use AES/CBC/PKCS{5/7}Padding with 256 bit and a different salt for each written value.
** Backend implementation guide :PROPERTIES: :CUSTOM_ID: h:7582b1c9-e305-4d51-a808-c10eb447f3de :END:
We provide a [[file:doc/backend.org][backend implementation guide]].
** Projects building on konserve :PROPERTIES: :CUSTOM_ID: h:79876ac1-414b-4180-8d65-63737cb3bc53 :END:
** Combined usage with other writers :PROPERTIES: :CUSTOM_ID: h:8a1b4a06-4b9f-496b-9eb2-52ac953a8e35 :END:
konserve assumes currently that it accesses its keyspace in the store exclusively. It uses [[https://github.com/replikativ/hasch][hasch]] to support arbitrary edn keys and hence does not normally clash with outside usage even when the same keys are used. To support multiple konserve clients in the store the backend has to support locking and proper transactions on keys internally, which is the case for backends like CouchDB, Redis and Riak.
** Changelog :PROPERTIES: :CUSTOM_ID: h:db9710e5-93b2-45db-ab9c-38e2d7ef6765 :END:
*** 0.7.274 :PROPERTIES: :CUSTOM_ID: h:433a14fe-229b-4944-8beb-fd268917705c :END:
*** 0.6.0-alpha1 :PROPERTIES: :CUSTOM_ID: h:c5fec032-a11d-4e4c-a367-9b8990168a75 :END:
- introduce common storage layouts and store serialization context with each
key value pair, this will facilitate migration code in the future
- implementation for the filestore (thanks to @FerdiKuehne)
- introduce metadata to track edit timestamps
- add garbage collector
- introduce superv.async error handling
- extend API to be more like Clojure's (thanks to @MrEbbinghaus)
- add logging
- update on ClojureScript support still pending*** 0.5.1 :PROPERTIES: :CUSTOM_ID: h:067c43cf-f940-4afa-87ea-730afc9bd5b4 :END:
*** 0.5 :PROPERTIES: :CUSTOM_ID: h:044ec59d-7487-437c-8068-d7e0d927ad46 :END:
*** 0.5-beta3 :PROPERTIES: :CUSTOM_ID: h:5f3907ee-c8de-4d9e-b5ff-beef6d5bf21b :END:
*** 0.5-beta1 :PROPERTIES: :CUSTOM_ID: h:53cb7995-3421-4223-8af2-e26a704db27f :END:
*** 0.4.12 :PROPERTIES: :CUSTOM_ID: h:e826b646-e350-4fa5-832e-3f7d84491c25 :END:
*** 0.4.11 :PROPERTIES: :CUSTOM_ID: h:d6bc4403-163c-4f31-8622-5fc02d1d65f4 :END:
*** 0.4.9 :PROPERTIES: :CUSTOM_ID: h:21be969e-b459-477e-bac3-a258bc04303c :END:
*** 0.4.7 :PROPERTIES: :CUSTOM_ID: h:eb0526f5-00cc-43eb-bb91-1d95132b6716 :END:
*** 0.4.5 :PROPERTIES: :CUSTOM_ID: h:5e770a0e-41b5-4003-a9c3-911fd7af94b7 :END:
*** 0.4.4 :PROPERTIES: :CUSTOM_ID: h:901e16eb-9e4f-445c-9f0c-749353b041b0 :END:
*** 0.4.3 :PROPERTIES: :CUSTOM_ID: h:ab372258-4800-4c07-b752-974bc5ea14ae :END:
*** 0.4.2 :PROPERTIES: :CUSTOM_ID: h:d72f44e0-9b54-4278-b8ff-7451b0e1bb45 :END:
*** 0.4.1 :PROPERTIES: :CUSTOM_ID: h:762a1693-f9ac-4086-890d-f68cb2e7dd33 :END:
*** 0.4.0 :PROPERTIES: :CUSTOM_ID: h:3a664ab1-1451-45dd-8d75-5eb1303f0214 :END:
*** 0.3.6 :PROPERTIES: :CUSTOM_ID: h:fbb5cae9-d70a-4423-80c1-847f638adca4 :END:
*** 0.3.4 :PROPERTIES: :CUSTOM_ID: h:0cda9a27-5b55-4916-a149-2361c068832a :END:
*** 0.3.0 - 0.3.2 :PROPERTIES: :CUSTOM_ID: h:40ff4f34-a46d-48fb-9989-da44b42ba050 :END:
*** 0.3.0-beta3 :PROPERTIES: :CUSTOM_ID: h:8be02dba-fad0-4184-8c33-0bffc6c3b667 :END:
*** 0.3.0-beta1 :PROPERTIES: :CUSTOM_ID: h:96af0a03-1f58-4636-9e18-49b260552e8b :END:
*** 0.2.3 :PROPERTIES: :CUSTOM_ID: h:7ef9fc3f-2372-4e7e-a2cb-3924ee3d65a4 :END:
*** 0.2.2 :PROPERTIES: :CUSTOM_ID: h:57386ea1-e952-45fa-9bbe-8c6cdb1d5bdc :END:
*** 0.2.1 :PROPERTIES: :CUSTOM_ID: h:9a3e49e9-9dd0-474d-949e-eb8eb0a15b80 :END:
*** 0.2.0 :PROPERTIES: :CUSTOM_ID: h:757b5af0-3262-4bb4-82ea-85aee87d77e1 :END:
** Contributors :PROPERTIES: :CUSTOM_ID: h:dd1ebb1a-2748-4f04-86f1-c2a5347ec9f8 :END:
** License :PROPERTIES: :CUSTOM_ID: h:8153b6f6-d253-4863-86b4-038dd383b6fe :END:
Copyright © 2014-2023 Christian Weilbach and contributors
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.