search

objectbox / objectbox-dart

Flutter database for super-fast Dart object persistence
https://docs.objectbox.io/getting-started
Apache License 2.0
927 stars 115 forks source link

Flatbuffers marshalling performance #68

Closed vaind closed 3 years ago

vaind commented 4 years ago

The way we currently marshal and unmarshal flatbuffers into objects isn't optimal. We should get rid of the intermediary propVals map<string, dynamic>. Ideally creating objects (or flatbuffers in the other direction) directly in the generated code.

greenrobot commented 4 years ago

Yes, this should use code generation for both ways.

Once this is done and reviewed, please give me the final review.

Buggaboo commented 4 years ago

Can we borrow anything from google's flatc?

vaind commented 4 years ago

I think we're already calling the right methods from flatbuffers. Just not being 100 % efficient for the whole roundtrip.

greenrobot commented 4 years ago

We should have a minimal CRUD perf app before doing that?

Buggaboo commented 4 years ago

What is the bottleneck here precisely? Runtime property selection using the Map? Can we use the offsets to get/put the data from a stream of bytes? I have a feeling we might need ByteData. N vs. NlogN?

greenrobot commented 4 years ago

Look at how Go does it, e.g. the generated code here:

Go does the FlatBuffers handling directly, while Java FlatBuffers handling is to slow and defers that to JNI. I hope compiled Dart is closer to Go performance, so it's at least worth a try.

Buggaboo commented 4 years ago

It could be even faster with caching probably e.g. maintain _lastwrite and _lastread timestamps in atomic transactions.

greenrobot commented 4 years ago

ObjectBox already benefits from OS cached memory mapping (see FAQ for some details). We did not bother about object-level caches so far, as ObjectBox gets like 1 million objects per second from the DB on regular notebooks.

Buggaboo commented 4 years ago

I'm looking into benchmark harness.

How does one stress test the library? N number of writes to a random (sub)set of fields, and reads? Then print the CPU processor type, number of cores, and memory?

vaind commented 3 years ago

This (future) change in vm/ffi will allow us to avoid copying the data from FlatBuffers builder to OBX_bytes_wrapper (or other c-backed memory) https://github.com/dart-lang/sdk/issues/44589 :eyes: