man-group / ArcticDB

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.
http://arcticdb.io
Other
1.53k stars 94 forks source link

Use a better `ReliableStorageLock` based on If-Match S3 atomic operations #2028

Open IvoDD opened 2 days ago

IvoDD commented 2 days ago

Is your feature request related to a problem? Please describe. The current ReliableStorageLock was being implemented before If-Match on put objects was announced by aws and hence uses a more complex structure than it is needed. This ticket lists all minor things we need to address when doing this simplification.

This is blocked on aws-sdk-cpp not having 1.11.455 or later which has the new If-Match on put requests support.

Describe the solution you'd like

IvoDD commented 1 day ago

Also we need to think about:

IvoDD commented 1 day ago

Rework C++ ReliableStorageLock tests to use a MockS3Client instead of the InMemoryStore. https://github.com/man-group/ArcticDB/pull/2014#discussion_r1865390437

Attempting to write such python tests turned out futile because of the GIL handling. It would probably still be better to have the C++ tests use MockS3Client instead of InMemoryStore