isar / hive

Lightweight and blazing fast key-value database written in pure Dart.
Apache License 2.0
4.02k stars 399 forks source link

[Question] Performance single big list vs key per item #482

Open Normynator opened 3 years ago

Normynator commented 3 years ago

Question I have a question about performance. I have a list of up to 50000 items. The list is read a lot but nearly never written. My question is: Is there a performance difference if I save the list as one big list with one key and load it into memory before accessing it vs storing each item with a key and use the „get“ function? I need to access the items by key only.

TLDR: how much slower are read operations if the box has a lot of keys compared to accessing the list from memory? Is it negligible? What impacts performance of box reads in general?

Version

firatagdas commented 3 years ago

I 'm wondering the same thing. So 50-60 mb data can be stored per key like in 30-40 key? which means ~2400 MB?

themisir commented 3 years ago

I write a simple benchmark for testing single key / multiple key read / writes. Read/writing with single key preforms better than multiple keys.

Benchmark code

```dart import 'dart:async'; import 'dart:math'; import 'integration.dart'; Future benchmark(String what, FutureOr Function() cb) async { var start = DateTime.now(); await cb(); var delta = DateTime.now().difference(start); print('$what took ${delta.inMilliseconds}ms'); return delta.inMilliseconds; } void main() async { var box1 = await openBox(false); var box2 = await openBox>(false); var random = Random.secure(); print('Generating items...'); var count = 500000; var list = List.generate(count, (index) => random.nextDouble()); var bench1 = 0, bench2 = 0; var total1 = 0.0, total2 = 0.0; bench1 += await benchmark('Writing multiple keys', () => box1.addAll(list)); bench2 += await benchmark('Writing single key', () => box2.put(1, list)); await box1.close(); await box2.close(); bench1 += await benchmark('Opening w/ multiple keys', () async { box1 = await box1.reopen(); }); bench2 += await benchmark('Opening w/ single key', () async { box2 = await box2.reopen(); }); var randomKeys = List.generate( count ~/ 1000, (index) => random.nextInt(count - 1), ); bench1 += await benchmark('Randomly reading multiple keys', () async { for (var key in randomKeys) { total1 += (await box1.get(key)) as double; } }); bench2 += await benchmark('Randomly reading single key', () async { var list = await box2.get(1); for (var key in randomKeys) { total1 += list[key] as double; } }); bench1 += await benchmark('Reading multiple keys', () async { for (var i = 0; i < count; i++) { total1 += (await box1.get(i)) as double; } }); bench2 += await benchmark('Reading single key', () async { var list = await box2.get(1); for (var i = 0; i < count; i++) { total2 += list[i] as double; } }); print('Equals: ${total1 == total2}'); print('Multiple keys: ${bench1}ms'); print('Single key: ${bench2}ms'); } ```

Results:

Generating items...
Writing multiple keys took 1773ms
Writing single key took 30ms
Opening w/ multiple keys took 1810ms
Opening w/ single key took 29ms
Randomly reading multiple keys took 4ms
Randomly reading single key took 0ms
Reading multiple keys took 1196ms
Reading single key took 6ms
Equals: false
Multiple keys: 4783ms
Single key: 65ms