Consistency / Transactions / Integrity

KinanBab commented 2 years ago

We currently have several sources of potential inconsistency:

No synchronization between tables and views ---> inconsistency when indices are updated during a concurrent table update (e.g. #118)
~~No synchronization between copies of the same record being moved/deleted/modified.~~
~~No synchronization between tables and Rocksdb Indices~~

Potential solution

~~Table-level locks in rocksdb connection or similar, which are acquired even when reading from views that are based on the tables.~~ We use rocksdb transactions!

Transactions/atomicity:

~~Failure of sub-operations of a single logical operations (e.g. cascading variable ownership delete, updating mutliple copies of the same shared row) does not rollback.~~

Integrity:

~~Uniqueness of PK is not enforced and results in replace: consistency issue with views that do not see the replace (#134)~~
Integrity of FK is not enforced.

KinanBab commented 1 year ago

Cleanups for after the deadline:

Integrity (Consistency as in ACID):

Transactions:

[ ] Implement multi-statement transactions with REPEATED READS: Batch writes to dataflow across transaction. Read from batch and dataflow within transaction. Version numbers?

Consistency (as in linearizability vs eventual):

[ ] Revisit deletes of dependent tables with changes to index.
[ ] Optimization: improve locking duration of in memory index when used to handle insert.

Isolation:

[ ] Change SK format: Sk -> PK (but not shard), then PK index gives us shard (allows us to lock PK index record for writes)
[ ] Unlock unmatching records as soon as they are discarded during scans (optimization to match myrocks)
[ ] Separate out reading with snapshot vs without snapshot into different functions in RocksdbTransaction, and RocksdbIndex.
[ ] Read most recent version of a record after iterating (for both records and indices) (optimization to reduce write skew and match myrocks)

KinanBab commented 1 year ago

brownsys / K9db