Multiple concurrent transactions slow

[mfbx9da4]

I am building a react app which has multiple views. Each view fetches the data it needs from IDB to render. Sometimes it fetches from the same table in different views. Also concurrently I am fetching data from the network and inserting data into IDB. I am experiencing queries taking up to 400ms. Presumably the issue is this (from the readme)

Do not await other things between the start and end of your transaction

Am I using IDB totally wrong? I am not supposed to use IDB as the primary store in the app and instead use it only on initial boot? I don't see how I can avoid doing other concurrent stuff while fetching.

[jakearchibald]

I can't debug this without a minimal reproducable example https://stackoverflow.com/help/minimal-reproducible-example. It seems unlikely that the speed issues are due to this library, but I'll happily reopen the issue and a look if you can provide an example.

[mfbx9da4]

import { openDB } from "idb/with-async-ittr";
import _ from "lodash";

const initDB = async () => {
  const instance = await openDB("db-name", 1, {
    upgrade(db, oldVersion, newVersion, transaction) {
      db.createObjectStore("messages", {
        keyPath: "id",
        autoIncrement: false
      });
    }
  });
  return instance;
};

const execute = async (db) => {
  const start = performance.now();
  await db.getAll("messages");
  return performance.now() - start;
};

function printTimings(timings: number[]) {
  console.log(
    "Count",
    timings.length,
    "avg",
    _.mean(timings),
    "max",
    _.max(timings),
    "min",
    _.min(timings)
  );
}

const TABLE_SIZE = 10000;
const N = 10;

async function main() {
  const db = await initDB();

  // populate
  const trx = db.transaction("messages", "readwrite");
  const items = [];
  for (let i = 0; i < TABLE_SIZE; i++) {
    const item = { id: "uuid-" + i, attr1: 1, attr2: 2, attr3: "attr3" };
    items.push(item);
  }
  await Promise.all([...items.map((x) => trx.store.put(x)), trx.done]);

  const inSeries = [];
  for (let i = 0; i < N; i++) {
    inSeries.push(await execute(db));
  }
  console.log("inSeries");
  printTimings(inSeries);

  const inParallel = await Promise.all(
    new Array(N).fill(1).map(() => execute(db))
  );
  console.log("inParallel");
  printTimings(inParallel);
  document.body.innerHTML = inParallel.join("<br />");
}

main();

https://codesandbox.io/s/gifted-wilbur-520eh?file=/src/index.ts

It looks like IDB serializes the inParallel requests. This is surprising to me as I would have thought that since theses are readonly transactions, IDB should be able to serve them more or less in parallel?

In most web applications it's likely that reading and writing from the store will be happening concurrently from many places. Is it therefore not suitable to use IDB as the primary location for holding state? Should it instead be used only to restore state on initial load?

[mfbx9da4]

db.getAll seems to produce results of about 3 seconds in the above experiment consistently - but when I use async iterators via import('idb/with-async-ittr') it can go up to 15 or 30 seconds for the query to resolve

[jakearchibald]

This isn't a reproducible example. window.db is undefined.

[mfbx9da4]

Yeah fair enough, didn't expect you to be quite so attentive. Appreciate your time 😊 I have updated the code snippet above and provided a codesandbox URL @jakearchibald

https://codesandbox.io/s/gifted-wilbur-520eh?file=/src/index.ts

[jakearchibald]

Thank you, I'll take a look

[jakearchibald]

db.getAll seems to produce results of about 3 seconds in the above experiment consistently - but when I use async iterators via import('idb/with-async-ittr') it can go up to 15 or 30 seconds for the query to resolve

The reduction doesn't include any async iteration at all, so I'll ignore this for now.

It looks like IDB serializes the inParallel requests. This is surprising to me as I would have thought that since theses are readonly transactions, IDB should be able to serve them more or less in parallel?

Right, but the creation of objects from their serialised form still needs to happen on the main thread. In your demo I'm seeing 100,000 objects being fetched and deserialised in 400ms in Chrome. Is that bad? Compared to what?

Doing it in series looks around 150ms slower than in parallel in your test. That seems expected.

https://codesandbox.io/s/quizzical-dijkstra-qh1xg?file=/src/index.ts – Doing it without the library is around 30ms quicker. So the overhead of the library is around 3ms per transaction on an object store with 10000 items. Maybe that's something to look into, but it seems pretty small, and the library makes it easy to drop down to raw IDB for the edge cases.

The overhead seems much less in Safari. In fact it's generally quicker in Safari, which might be worth filing a crbug.com, although you'll need to state why this demo is representative.

Results in Firefox are similar to Chrome.

If I'm missing something, or if you were actually talking about async iterators, please make a reproducible example that demonstrates the things you're talking about.

[mfbx9da4]

You are absolutely correct. Thank you for looking into this. I was mislead by my timing function. The timings I observed have nothing to do with IDB. The query looked like it was taking ages to finish because the async function was taking ages to resolve. The async function was taking ages to resolve because the main thread was busy rendering the list returned from IDB. I was memoizing each item in the list based on a weak reference to the db row which obviously changed each time it was serialized out of the database.

The root cause of the performance issues I was seeing was my shoddy code and not your library.

Thanks again @jakearchibald 😊 👍

[jakearchibald]

No problem!