search

man-group / ArcticDB

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.
http://arcticdb.io
Other
1.23k stars 79 forks source link

Make modification of snapshot atomic and corresponding tombstone handling changes #1650

Open phoebusm opened 1 week ago

phoebusm commented 1 week ago

Reference Issues/PRs

Originally only for forward porting https://mangit.maninvestments.com/projects/DATA/repos/arcticc/pull-requests/1095/overview But corresponding test requires https://github.com/man-group/arcticdb-enterprise/issues/32 so part of changes is included too.

What does this implement or fix?

  1. Modification of snapshot will not longer delete the original SNAPSHOT_REF key. It will be simply overwritten so atomicity can be achieved
  2. As the changes will affect op logs as well and the test cannot be made without https://github.com/man-group/arcticdb-enterprise/issues/32 so some corresponding changes for that ticket as well. More details can be found there.

Any other comments?

Checklist

Checklist for code changes... - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?
phoebusm commented 1 week ago

Depended by https://github.com/man-group/arcticdb-enterprise/pull/133

poodlewars commented 4 days ago

This change requires a test in this repo. I think that I did something similar a while ago for vref atomicity - https://github.com/man-group/ArcticDB/blob/master/python/tests/stress/arcticdb/version_store/test_concurrent_read_and_write.py