Deno KV browser implementation

[mfulton26]

I want to write code that can be reused between clients and servers. As Deno KV does not implement a standard Web API available in web browsers, I think it would be great to publish a module for kind of polyfilling Deno KV in browsers but backed by IndexedDB. This way apps can be built to store and sync data using a consistent API, etc.

[mfulton26]

An IndexedDB backing implementation would require no 3rd party libraries but a possible alternative is to use SQLite Wasm in the browser backed by the Origin Private File System.

If I understand correctly, Deno's current implementation of KV outside of Deno Deploy is backed by SQLite. Some or all of that might be done in Rust at the moment but once #17833 lands then Deno KV outside of Deno Deploy could be updated to be built on top of that using SQLite WASM and then most, if not all, of that code becomes reusable in the browser and easily maintained going forward (because no separate implementation will be needed for the browser).

[exside]

@mfulton26 well, don't blame me if it goes completely wrong 😉 ... wrote this last night, did my best in terms of trying to figure out how Deno KV serializes the keys etc.

It's basically a browser compatible Deno KV mock that uses localStorage, it of course simply can't do everything Deno KV can, but I think it should be good enough to implement something in the browser that should (hopefullly) translate realtively easy to a real Deno KV environment:

/**
/*  Creates a Deno KV-like key-value store based on the localStorage with optional
/*  (custom) versioning and migration support.
/*  This has passed basic testing, although not how well it aligns with what native Deno KV does itself!
/*  The API is the same minus atomic operations and some options like consistency level, Kv64 etc.
/*  that don't really make sense or are insanely complex or impossible to do with a synchronous
/*  API like localStorage is. The KV's methods are all fake async (because Deno KV's api is async)
/*  so code written and used in the browser with this implementation should (hopefully) work
/*  with Deno KV on the server side as well (muuuch more testing needed to confirm that)!
/*  Aside from `open()` and `close()` this implementation should cover the entire Deno KV API.
/*
/*  There are of course differences in behavior you should be aware of if you build any logic around it:
/*  - The most important limitation with this mock is that if you use non-serializable values in
/*    in your keys (your values too!), it will not behave like you expect! Stick to types that can be
/*    serialized to JSON... localStorage can only handle strings, that's the reason.
/*  - With Deno KV you can pass an `expireIn` option, but the values you receive from `get()`
/*    will never return that expiration date. This implementation returns that information
/*    when querying a key, e.g. besides value and versionstamp you get back `expires`
/*    if a TTL was provided when setting the value, it's going to be an ISO timestamp otherwise `null`.
/*  - `list()` is an AsyncIterator like it is with Deno KV, but it does NOT support cursor, consistency
/*    or batchSize options, you can pass them to the method, but nothing will happen.
/*    Deno KV `list()` requires a selector, this implementation does not, it will simply list all
/*    KV entries if you don't provide a selector.
/*    Furthermore this `list()` implementation supports an optional 3rd argument which is a function
/*    passed to `Array.filter()` that allows filtering the returned entries further after selector(s)
/*    and options have been applied.
/*  - This implementation does support some features Deno KV does not have:
/*      1. You can optionally provide a `version` function that will replace how the versionstamp is created
/*         and thus provide your own versioning implementation.
/*      2. You can also provide a `migrate` function that will be applied if an outdated versionstamp
/*         is encountered, you can for example extend the old value with some new ones keeping what was stored.
/*      3. There is a `toJSON()` method you can use to serialize either the entire KV store or a portion of it.
/*         It supports the same arguments as `list()` (because it internally calls it) and has options to
/*         serialize prettified JSON or streamable (individual store entries separated by newlines) JSON.
/*
/*  @param {Function} [version] - Optional function to generate a versionstamp for stored values.
/*  @param {Function} [migrate] - Optional function to migrate outdated values. It is called with the key and outdated value and should return the migrated value.
/*  @param {String} [prefix='kv:'] - Prefix to be used for keys in localStorage, helping in namespacing and avoiding key conflicts.
/*
/*  @returns {Object} - An object providing the mocked Deno KV API methods to interact with the store.
/*/ 
const kv = function(version, migrate, prefix = 'kv:') {
    const $hash = typeof version === 'function' ? version : ( window?.$hash || ((v) => ((v) => 1) );
    const _queueListeners = new Set();
    const _key = {
        inrange: (key, start, end) => {
            const orderedKey = _key.order(key);
            const orderedStart = start ? _key.order(start) : null;
            const orderedEnd = end ? _key.order(end) : null;
            if (orderedStart && orderedKey < orderedStart) { return false; }
            if (orderedEnd && orderedKey >= orderedEnd) { return false; }
            return true;
        },
        order: (key) => key.slice().sort((a, b) => {
            const order = ['Uint8Array', 'string', 'number', 'bigint', 'boolean'];
            const typeA = typeof a, typeB = typeof b;
            return typeA === typeB ? (typeA === 'number' ? a - b : String(a).localeCompare(String(b))) : order.indexOf(typeA) - order.indexOf(typeB);
        }),
        // Check how Deno KV actually does it here:
        // https://github.com/denoland/deno/blob/main/ext/kv/codec.rs
        serialize: (key) => prefix + JSON.stringify(key),
        deserialize: (serializedKey) => JSON.parse(serializedKey.substring(prefix.length))
    };

    return {
        /**
        /*  Retrieves a value from the store by its key(s).
        /*
        /*  @param {Array} key - The keys array to look up the value.
        /*
        /*  @returns {Promise<Object>} - An object containing the key, its associated value, versionstamp and expiry.
        /*/
        get: async (key) => {
            const serializedKey = _key.serialize(key);
            const data = JSON.parse(localStorage.getItem(serializedKey));

            if ( !data || (data.expires && new Date(data.expires) < new Date()) ) {
                data && data.expires && localStorage.removeItem(serializedKey);
                return { key, value: null, versionstamp: null };
            }

            if ( data.versionstamp !== $hash(data.value) ) {
                if (typeof migrate === 'function') {
                    data.value = await migrate(key, data.value);
                    data.versionstamp = $hash(data.value);
                    localStorage.setItem(serializedKey, JSON.stringify(data));
                } else {
                    console.warn('Outdated value found but no migration function defined!', key, data);
                }
            }

            return { key, ...data };
        },

        /**
        /*  Retrieves multiple values from the store based on the provided keys.
        /*
        /*  @param {Array<Array>} keys - An array of key arrays to look up the values.
        /*
        /*  @returns {Promise<Array<Object>>} - An array of objects.
        /*/
        getMany: async function(keys) { return Promise.all(keys.map(key => this.get(key))) },

        /**
        /*  Stores a value in the store with the given key.
        /*
        /*  @param {Array} key - The key array to associate with the value.
        /*  @param {*} value - The value to be stored.
        /*  @param {Object} [options] - Optional parameters for storing the value. `expireIn` sets the expiration time in milliseconds.
        /*
        /*  @returns {Promise<Object>} - An object indicating the success and the versionstamp of the stored value.
        /*/
        set: async (key, value, options = {}) => {
            const data = {
                value,
                versionstamp: $hash(value),
                expires: options.expireIn ? new Date(Date.now() + options.expireIn).toISOString() : null
            };
            localStorage.setItem(_key.serialize(key), JSON.stringify(data));
            return { ok: true, versionstamp: data.versionstamp };
        },

        /**
        /*  Removes a value from the store by its key.
        /*
        /*  @param {Array} key - The key array of the value to be removed.
        /*/
        delete: async (key) => localStorage.removeItem(_key.serialize(key)),

        /**
        /*  Iterates over values in the store based on the provided selector prefix, range, or limited by options.
        /*  This method provides an AsyncIterator to be used with `for await...of` loops. 
        /*  This `list()` implementation supports an additional third argument which is NOT STANDARD for Deno KV.
        /*  It's a function that allows for further filtering the results before yielding them within the iterator.
        /*
        /*  The selector can either be a prefix selector or a range selector:
        /*  - A prefix selector selects all keys that start with the given prefix (optionally starting at a given key).
        /*  - A range selector selects all keys that are lexicographically between the given start and end keys.
        /*
        /*  @param {Object} [selector] - The selection criteria. NOT STANDARD: Can be undefined here, not with Deno KV!
        /*      @property {Array} [selector.prefix] - Defines the prefix for filtering the results.
        /*      @property {Array} [selector.start] - The starting key for range selection.
        /*      @property {Array} [selector.end] - The ending key for range selection.
        /*  @param {Object} [options] - Optional parameters for the listing.
        /*      @property {number} [options.limit] - Limits the number of results.
        /*      @property {boolean} [options.reverse] - If true, reverses the order of results.
        /*  @param {Function} [fn] - NON STANDARD: Optional filter function to filter entries before yielding.
        /*
        /*  @returns {AsyncIterator} - An AsyncIterator yielding store entries.
        /*/
        list: async function*(selector, options = {}, fn) {
            const length = localStorage.length;
            for (let i = (options?.reverse ? length-1 : 0); (options?.reverse ? i > -1 : i < length); i+= (options?.reverse ? -1 : 1)) {
                // Limit results based on `options.limit`
                if ( options?.limit !== undefined && options.limit <= 0 ) { break; }

                const serializedKey = localStorage.key(i);
                if ( !serializedKey.startsWith(prefix) ) { continue; };

                const deserializedKey = _key.deserialize(serializedKey);
                if ( selector?.prefix && !deserializedKey.slice(0, selector.prefix.length).every((part, index) => part === selector.prefix[index]) ) {
                    continue;
                }
                if ( selector?.start && !_key.inrange(deserializedKey, selector?.start, selector?.end) ) { continue; }

                const entry = await this.get(deserializedKey);
                if ( entry.value !== null ) {
                    if ( !fn || (typeof fn === 'function' && fn(entry)) ) {
                        if ( options?.limit !== undefined ) { options.limit--; }
                        yield entry;
                    }
                }
            }
        },      

        /**
        /*  Adds a value into a mock database queue to be delivered to queue listeners.
        /*  This method simulates the behavior of the Deno KV's enqueue().
        /*  `keysIfUndelivered` option is not supported, you can pass it, but won't have an effect.
        /*
        /*  @param {*} value - The value to be enqueued.
        /*  @param {Object} [options] - Optional settings for the enqueue operation.
        /*  @param {number} [options.delay] - Delays the delivery of the value by the specified number of milliseconds.
        /*  @returns {Promise<Object>} - An object indicating the success of the enqueue operation.
        /*/
        enqueue: async (value, options) => {
            for (const fn of _queueListeners) {
                options?.delay && await (new Promise(res => setTimeout(res, options.delay)));
                await fn(value);
            };
        },

        /**
        /*  Listens for queue values to be delivered from the mock database queue.
        /*  This method simulates the behavior of the Deno KV's listenQueue().
        /*
        /*  @param {Function} handler - A callback function that gets when a new value is dequeued.
        /*
        /*  @returns {Promise<void>}
        /*/
        listenQueue: async (handler) => {
            if ( !_queueListeners.has(handler) ) {
                _queueListeners.add(handler)
            }
        },

        /**
        /*  Provides a mock for Deno Kv's AtomicOperation API with the same chainable methods, but
        /*  they do nothing...you can provide a function to each of the mock methods that receives
        /*  `this`, just return it again, otherwise you break the chaining! Something like this:
        /*
        /*  @example
        /*  ```
        /*  (await $core.kv().atomic()).check((t) => (console.log('check'), t)).min((t) => (console.log('min'), t)).commit();
        /*  // you can pass an arbitrary number of additional args to those mock functions, like
        /*  (await $core.kv().atomic()).check((t, arg1, arg2) => (console.log('check', arg1, arg2), t), 'foo', 'var')
        /*  ```
        /*
        /*  @returns {Promise<*>} - A mocked API of Deno KV's atomic().
        /*/
        atomic: async () => {
            console.warn('Atomic ops are not supported!');
            return [
                'check',
                'commit',
                'delete',
                'enqueue',
                'max',
                'min',
                'mutate',
                'set',
                'sum'
            ].reduce((r, m) => (r[m] = function(fn, ...args) { return typeof fn === 'function' ? fn(this, ...args) : this }, r), Object.create(null));
        },

        /**
        /*  NON STANDARD: Serializes the store to a JSON string.
        /*  The methods first 3 arguments match the ones from list() (see there for details).
        /*  
        /*  @param {Object} selector - The selection criteria for list().
        /*  @param {Object} [options] - Optional parameters for list().
        /*  @param {Function} [fn] - Optional filter function list().
        /*  @param {Number|String} [pretty] - JSON.stringify() space param to prettify output. Defatuls to tab indent.
        /*  @param {Boolean} [streamable] - If the output should be stringified entries separated by newlilnes.
        /*  
        /*  @returns {String} - A JSON string representation of the KV store.
        /*/
        toJSON: async function(selector, options, fn, pretty = '\t', streamable = false) {
            let data = [];
            for await (const entry of this.list(selector, options, fn)) {
                data.push(entry);
            }

            data = data.reduce((r, entry) => {
                if ( streamable ) { r.push(JSON.stringify({ key: entry.key, value: entry.value })); return r; }
                return r[JSON.stringify(entry.key)] = entry.value, r;
            }, streamable ? [] : {});

            return streamable ? data.join('\n') : JSON.stringify(data, null, pretty);
        },

    };
};

feel free to try it out and let me know what breaks or doesn't work as expected, going to use this as well for my local storage needs and hopefully improve it over time...

[nhrones]

@exside I'm working on something similar.
I have the kvKey-codec completed if you need that.

https://github.com/nhrones/kv-key-codec

I'll complete the kvValue codec soon. With these codec, you could use any persistence layer. I'm using SQLite wasm in the browser.

[exside]

awesome, gonna have a look 👍 !

[exside]

@nhrones that looks very good, so this is actually how Deno KV does it, with byte Buffers? I couldn't find the source within the deno repo, all I found was core.serialize() (well, for string keys at least) but then got stuck... I do very dumb lexicological ordering above, but then I tested and it didn't work as expected 🤣 , so i changed the logic of the ordering... still need to run comparison tests with actual Deno KV to see how badly it deviates... in the end I was already using a localStorage wrapper already that had versioning, expiration, migrating etc., so I thought, why not re-write that closer to what Deno KV's API looks like, so it's going to be easy going from a pretty dumb client side localstorage wrapper to server side and use the same API with Deno KV... but as I wrote, it's very fresh, basic stuff seems to work, haven't gotten into any complex stuff with mixed key types or very large collections (not a good idea with localStorage anyways)... So yes people, if you look for a proper way to actually do this, watch @nhrones repo!

[nhrones]

The Deno-codec is at https://github.com/denoland/deno/blob/main/ext/kv/codec.rs Lets take this conversation to the Discord server. https://discord.com/channels/684898665143206084/1138852815725395978

[exside]

not able to join?

Regarding

The Deno-codec is at https://github.com/denoland/deno/blob/main/ext/kv/codec.rs

that is way over my head 🤣 ! Glad you are implementing it in JS!

[secext2022]

I want to store large amount of data in browser with deno-kv, so the backend should be IndexedDB, not localStorage.