search

man-group / ArcticDB

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.
http://arcticdb.io
Other
1.52k stars 94 forks source link

Add `lib.delete_batch` similar to `read_batch` and `write_batch` #1387

Open IvoDD opened 9 months ago

IvoDD commented 9 months ago

Is your feature request related to a problem? Please describe. Batch deletes can be quite a lot faster than doing the deletes sequentially.

Describe the solution you'd like A lib.delete_batch which can delete a batch of symbols concurrently. Must also be able to support specifying a version to delete for each symbol, probably through a DeleteRequest class similar to ReadRequest etc. Non-batch delete supports passing a list of versions, but then just iterates these in Python. If we make delete_batch support having multiple DeleteRequests for the same symbol, then we can also replace this iteration with an efficient implementation in C++.

Describe alternatives you've considered delayed_deletes can make individual deletes faster but have to think about applying them later.

delete_library can delete an entire library with all it's symbols.

IvoDD commented 8 months ago

It would also be helpful if lib.update_batch allows updating multiple date ranges inside a single symbol.

zeddit commented 8 months ago

looking forward to this feature! It really helps the efficiency of daily data management.

alexowens90 commented 6 months ago

Batch update is covered here https://github.com/man-group/ArcticDB/issues/700 Updating multiple date ranges in a single symbol would be a change to update, rather than part of a batch update API. Changing this ticket to just cover batch deletion