Speeds of loading multiple items into db

[mikaelwills]

Hey Alexandre, At work I've been assigned a research project to find a good flutter database solution. Ive impemented ObjectBox and now have been looking into Sembast as ObjectBox wasn't that map friendly.

I've implemented sembast and am loving how flexible it is with maps and I want to put it forward for the one that we'll use but I'm hoping you might be able to help me speed up the time it takes to load items in.

In ObjectBox i'm loading about 71,000 records 0.32s In Sembast, I've tried using two approaches (found in issues on here),

The first way uses addAll and takes roughly 4.8s:

    List<Map<String, dynamic>> itemsForDb = [];
    var itemStopwatch = Stopwatch()..start();
    model.items.forEach((key, value) {
      itemsForDb.add(value.toJson());
    });
    await itemStore.addAll(db, itemsForDb).then((value) async {
      itemStopwatch.stop();
      var records = await itemStore.find(db, finder: Finder());
      print(
          "Loading ${records.length} items took ${itemStopwatch.elapsedMilliseconds / 1000}s");
    });

The second way using a db.transation takes up to 16s but interestingly the itemStopwatch which awaits the itemFutures finishes in 0.7s which im pretty happy with but the transaction stopwatch doesn't stop for another 15s or so..

      var transationStopwatch = Stopwatch()..start();
      await db.transaction((txn) async {
      var itemStopwatch = Stopwatch()..start();
      List<Future> itemFutures = [];
      model.items.forEach((key, value) async {
        itemFutures.add(itemStore.add(txn, value.toJson()));
      });
      await Future.wait(itemFutures).then((value) async {
        itemStopwatch.stop();
        print(
            "Loading items took ${itemStopwatch.elapsedMilliseconds / 1000}s");
      });
    }).then((value) {
      transationStopwatch.stop();
      print(
          "The db transation took ${transationStopwatch.elapsedMilliseconds / 1000}s");
    });

If you could point out where i'm going wrong to speed these up that would be amazing!

[alextekartik]

Hi and thanks for doing performance testing that I never did. sembast is not designed to be super fast but rather easy to use (no setup, no schema) or to setup mocking for other databases. However it is ok for some apps (small to medium database, and I use it myself in some apps) and some features like onSnapshots are well optimized to work well within flutter applications.

See https://github.com/tekartik/sembast.dart/blob/master/sembast/doc/usage_recommendations.md for what I mean by small to medium database.

If you want to do performance testing, you should call disableSembastCooperator() mentionned in the same document.

You should then see roughly similar performance between addAll (your first test) and the transaction (your second test), since addAll creates a transaction to add them all.

[mikaelwills]

Thanks for your reply! Just put in the disable cooperator code and yea they have become similar now, but I think our app is just outside of the small to medium range. Great work on Sembast though!

[alextekartik]

I indeed don't recommend sembast for application with lots of data. isar, objectdb, hive, realm, sqflite should be considered (and maybe others).