danthegoodman1 / icedb

An in-process Parquet merge engine for better data warehousing in S3
https://blog.danthegoodman.com/icedb-v3--third-times-the-charm
Other
125 stars 5 forks source link

Merge/Tombstone clean coordination example #56

Open danthegoodman1 opened 1 year ago

danthegoodman1 commented 1 year ago

Support coordination for concurrent merges to prevent duplicate data.

Maybe we make a class to implement that will handle the locking and such, with examples?

The class would need:

Example options:

Merging locks should be by table

danthegoodman1 commented 1 year ago

Should we just have examples of how to do this?

danthegoodman1 commented 41 minutes ago

with the new s3 conditional writes, this can even be done in pure S3!