man-group / ArcticDB

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.
http://arcticdb.io
Other
1.52k stars 93 forks source link

Do not read/write the `APPEND_REF` key when calling `write/append` with `incomplete=True` on the V1 API #1937

Closed alexowens90 closed 2 weeks ago

alexowens90 commented 1 month ago

Calling write/append with incomplete=True eventually calls version_store.append_incomplete. This method reads the APPEND_REF key, modifies the next_key field of the TimeSeriesDescriptor, and then writes it back to storage again. This is pointless, as compact_incompletes as accessible to users ignores the linked-list structure, and collects APPEND_DATA keys via iteration. In addition, in a truly parallel write/append, the linked list would not be in the correct order anyway. Calling write with parallel=True does the correct thing. Namely, writing APPEND_DATA keys without touching the linked-list structure. Both write and append with incomplete=True should be modified to do the same thing. The functionality to write linked-list structures of APPEND_DATA keys is useful for testing though, so this should be moved to the LibraryTool.