splitgraph / seafowl

Analytical database for data-driven Web applications 🪶
https://seafowl.io
Apache License 2.0
397 stars 9 forks source link

Vanilla S3 object store doesn't support `copy_if_not_exists` #330

Closed gruuya closed 1 year ago

gruuya commented 1 year ago

This is needed by delta-rs when committing changes to the table via the default implementation of ObjectStore::rename_if_not_exists, and will consequently result in Object Store error: Operation not yet implemented. atm.

Since we don't expect a great write frequency it is ok to override the ObjectStore::rename_if_not_exists with ObjectStore::rename for now (which is unsafe in principle), and down the line look into using delta-rs specific S3 store which has support for dynamo DB lock to achieve the rename atomically.