vlcn-io / cr-sqlite

Convergent, Replicated SQLite. Multi-writer and CRDT support for SQLite
https://vlcn.io
MIT License
2.77k stars 76 forks source link

lookaside for primary keys. Replaces #344 #373

Closed tantaman closed 1 year ago

tantaman commented 1 year ago

Replaces #344

TODO:

Test key creation:

tantaman commented 1 year ago

There's a regression to insert performance (https://github.com/vlcn-io/cr-sqlite/blob/c26d2f8a088fadc1cdca7b592f8cb73101e8feff/py/perf/perf.ipynb). Apparently writing an extra table is a heavy (~20%) hit. Update performance is improved by ~15% though. Makes sense since updates do not have to write to the lookaside table -- the row is already there.

This is probably worth it given the storage savings. Users will most likely use uuids for primary keys in CRRs so we can save ~12-14 bytes per column with this. For users with compound primary keys, the savings are greater.


Still a number of failing integration tests.

tantaman commented 1 year ago

@jeromegn - everything looks good if you want to give it a spin.

I'm going to write a few more tests still before merging.

jeromegn commented 1 year ago

Nice!

What about the remaining tasks? I fear they'd require a new migration.

tantaman commented 1 year ago

The rowid change will not since virtual tables don't actually save a rowid.

I don't think I'm going to move the sentinel information out of the clock tables since it will require ordering another set of tables when reading changes out.