Is your feature request related to a problem? Please describe
When we designed shallow copy feature initially, remote store shard level metadata file was per commit point and was being updated with each refresh.
Also existing full copy snapshots supports shard snapshots at commit level. So we have gone ahead with commit level shallow copy snapshots.
however, since remote store shard level metadata file is immutable now, which means we create new metadata at every refresh. we can also directly use this for snapshot operation.
After that, we do not have to flush on every shard snapshot to create a new commit point. we can just rely on remote store feature to push the latest data and we can lock the latest metadata from remote store. which will make shard snapshot operation light weight as well.
dblock
commented
1 week ago
Is your feature request related to a problem? Please describe
When we designed shallow copy feature initially, remote store shard level metadata file was per commit point and was being updated with each refresh. Also existing full copy snapshots supports shard snapshots at commit level. So we have gone ahead with commit level shallow copy snapshots. however, since remote store shard level metadata file is immutable now, which means we create new metadata at every refresh. we can also directly use this for snapshot operation. After that, we do not have to flush on every shard snapshot to create a new commit point. we can just rely on remote store feature to push the latest data and we can lock the latest metadata from remote store. which will make shard snapshot operation light weight as well.
Describe the solution you'd like
Allow snapshots to trigger at refresh level.
Related component
Storage:Snapshots
Describe alternatives you've considered
No response
Additional context
No response