Finalize staged data memory use improvements - Githubissues

man-group / ArcticDB

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.

http://arcticdb.io

Other

1.52k stars 93 forks source link

Finalize staged data memory use improvements #2013

Open poodlewars opened 3 days ago

poodlewars commented 3 days ago

Discussion: https://chat-man.slack.com/archives/C02NEG3V38X/p1732181354814409

Remaining work for this PR:

Tidy up the error code handling
Tests that data keys are removed upon failure
Final memory and performance measurements

Future work:

Backpressure - https://github.com/man-group/ArcticDB/pull/2013/files#diff-07407837d69849c0f3084af0394134f44646daae2e34ba042122d7bdc8a44d12R151 . There are still theoretical ways where memory use could grow, I think (if we have too many incomplete "write" futures live at once, since each holds a segment, and the IO executor has an unbounded queue size).