IndexedDB persister

[segments-tobias]

Is your feature request related to a problem? Please describe. localStorage is limited to 5MB in size.

Describe the solution you'd like IndexedDB is a browser API for client-side storage of significant amounts of structured data. Since it's a database, I would think it could be a nice complement to TinyBase. Ideally, every change to TinyBase would be incrementally pushed to IndexedDB (maybe even in a Web Worker to unblock the main thread).

Do you know if anyone has explored this option? Are there any reasons this might not be a good idea?

[yan-yanishevsky]

I made a custom persister using idb and broadcast-channel packages. It probably needs some improvements, but it works well enough

import { BroadcastChannel } from 'broadcast-channel'
import { DBSchema, openDB } from 'idb'
import {
  Store,
  Table,
  Tables,
  Value,
  Values,
  createCustomPersister,
} from 'tinybase'

const DB_NAME = 'tiny-db'
const STORE_TABLES_NAME = 'tiny-tables-store'
const STORE_VALUES_NAME = 'tiny-values-store'

interface IDB extends DBSchema {
  [STORE_TABLES_NAME]: {
    value: Table
    key: string | number
  }
  [STORE_VALUES_NAME]: {
    value: Value
    key: string | number
  }
}

export function createIdbPersister(store: Store) {
  const broadcastHandler = async (event: string) => {
    if (event === 'reSync') {
      persister.load()
    }
  }
  const broadcast = new BroadcastChannel(DB_NAME)
  const idbPromise = openDB<IDB>(DB_NAME, 1, {
    upgrade: (db) => {
      db.createObjectStore(STORE_TABLES_NAME)
      db.createObjectStore(STORE_VALUES_NAME)
    },
  })
  const persister = createCustomPersister(
    store,
    async () => {
      try {
        const [tables, values] = await Promise.all([
          getPersistedStore(STORE_TABLES_NAME),
          getPersistedStore(STORE_VALUES_NAME),
        ])

        return [tables, values]
      } catch (e) {
        console.error('e:', e)
      }
    },
    async (getContent) => {
      const [tables, values] = getContent()

      await Promise.all([
        setPersistedStore(STORE_TABLES_NAME, tables),
        setPersistedStore(STORE_VALUES_NAME, values),
      ])
      broadcast.postMessage('reSync')
    },
    () => broadcast.addEventListener('message', broadcastHandler),
    () => broadcast.removeEventListener('message', broadcastHandler),
  )

  async function setPersistedStore(
    storeName: typeof STORE_TABLES_NAME | typeof STORE_VALUES_NAME,
    content: Tables | Values,
  ) {
    const idb = await idbPromise

    return Promise.all(
      Object.entries(content).map(async ([key, value]) => {
        return idb.put(storeName, value, key)
      }),
    )
  }

  async function getPersistedStore(
    storeName: typeof STORE_TABLES_NAME | typeof STORE_VALUES_NAME,
  ) {
    const idb = await idbPromise
    const keys = await idb.getAllKeys(STORE_TABLES_NAME)
    const output = {}

    await Promise.all(
      keys.map(async (key) => {
        const value = await idb.get(storeName, key)

        output[key] = value
      }),
    )

    return output
  }

  return persister
}

[jamesgpearce]

Wow that is really awesome. I held back from doing this because I couldn't see a way to make it reactive to idb changes outside of TinyBase. Looks like you just stubbed that out, so this requires explicit saves and loads?

[yan-yanishevsky]

In my code above I made more code transformations so that:

• Each Table is stored as a row in idb (Ideally I wanted to store each Table as idb table, but that's harder to do, because of the way idb versioning works).
• Each Value is stored as a row in idb.

It was easier to debug for me. If you don't need that, here is a much simpler example with idb-keyval.

import { BroadcastChannel } from 'broadcast-channel'
import { get, set } from 'idb-keyval'
import { Store, createCustomPersister } from 'tinybase/cjs'

const STORE_NAME = 'tiny-store'

export function createIdbPersister(store: Store, storeName = STORE_NAME) {
  const broadcast = new BroadcastChannel(storeName)
  const broadcastHandler = (event: string) => {
    if (event === 'reSync') {
      persister.load()
    }
  }
  const persister = createCustomPersister(
    store,
    async () => get(storeName),
    async (getContent) => {
      await set(storeName, getContent())
      broadcast.postMessage('reSync')
    },
    () => broadcast.addEventListener('message', broadcastHandler),
    () => broadcast.removeEventListener('message', broadcastHandler),
  )

  return persister
}

I also noticed just now, that I have a mistake in my first example... broadcast.postMessage('reSync') should be in the setter function, not in the getter (updated the comment)

[jamesgpearce]

Would you consider packaging this up as a pull request? Or are you OK if I implement this into the persister framework?

[yan-yanishevsky]

Yes, you can grab a version you like more

[jamesgpearce]

It's coming! https://github.com/tinyplex/tinybase/releases/tag/v4.2.0-beta.1 - please kick the tires and check it works.

Thank you so much @yan-yanishevsky for getting the ball rolling here 🙏

[jamesgpearce]

OK, released 4.2 with this in. Have fun and good luck!