The following text was present in a retired "lake design" document (see #3803). It has been established that this was really a pending to-do, so this issue tracks its ultimate implementation and corresponding updates to docs.
Since a Zed lake can provide strong consistency, workflows that manipulate
data in a lake can utilize a model where updates are made to the data
in place. Such updates involve creating new commits from the old data
where the new data is a modified form of the old data. This provides
emulation of row-based updates and deletes.
If the pool-key is chosen to be "this" for such a use case, then unique
rows can be maintained by trivially detected duplicates (because any
duplicate row will be adjacent when sorted by "this") so that duplicates are
trivially detected.
Efficient upserts can be accomplished because each data object is sorted by the
pool key. Thus, an upsert can be sorted then merge-joined with each
overlapping object. Where data objects produce changes and additions, they can
be forwarded to a streaming add operator and the list of modified objects
accumulated. At the end of the operation, then new commit(s) along with
the to-be-deleted objects can be added to the journal in a single atomic
operation. A write conflict occurs if there are any other deletes added to
the list of to-be-deleted objects. When this occurs, the transaction can
simply be restarted. To avoid inefficiency of many restarts, an upsert can
be partitioned into smaller objects if the use case allows for it.
> TBD: Finish working through this use case, its requirements, and the
> mechanisms needed to implement it. Write conflicts will need to be
> managed at a layer above the journal or the journal extended with the
> needed functionality.
philrz
commented
2 years ago
The following text was present in a retired "lake design" document (see #3803). It has been established that this was really a pending to-do, so this issue tracks its ultimate implementation and corresponding updates to docs.