Closed gruuya closed 1 year ago
How do information schema queries work under this proposed model? It looks like we use the catalog DB to satisfy them instead of crawling the object store for all tables and their schemas?
Yes, that is correct, we pre-load the schema information from our catalog.
This is the initial effort of switching our custom MVCC/Lakehouse protocol to use the Delta lake protocol instead.
A couple of general points:
DELETE
&UPDATE
VACUUM
still not implemented for new tablesdropped_table
table and related methods)As for the implementation details:
inner
ininternal_object_store
), and then "scoping" it for each DeltaTable accordingly. This means wrapping it up with some other object stores so that/
points to the table root uri (see below). The other alternative that delta-rs offers is providing the store_options, out of which it will then build the store for a given uri, but I didn't like this mostly because the memory store implementation then doesn't work (it get's re-created from scratch for each create/write/table builder instantiation).<inner_store_root>/<table_uuid>/
(that is the place where delta-rs puts all the table-related stuff as per the protocol). For instances3://my_bucket/e62c26e2-b6e8-414a-a37a-66dd77430b5d/
orfile:///data/dir/e62c26e2-b6e8-414a-a37a-66dd77430b5d/
. Thetable_uuid
is kept in thetable
catalog table and created randomly for each new table.VACUUM
s in chunked batches) because table uri is unique to a given table and independent of db/schema/table name. Likewise, renaming a table is only a catalog operation.