[Enhancement]: Refine Delegator L0 segment & delete buffer

[congqixia]

Is there an existing issue for this?

• [X] I have searched the existing issues

What would you like to be added?

Reduce the memory usage for delegator with large number of l0 segments or delete buffer

Why is this needed?

Before streaming node implementation, delegator cannot control the size of l0 segments and delete buffer. In some cases the memory pressure could be too large to handle. We would like to change the l0 "loading" and split process in delegator and do some refinement for delete buffer purging

Anything else?

No response

[congqixia]

• [ ] Load deltalog in worker instead of delegator
  • [x] Store l0 deltalog meta only and dispatch logs when loading segments #36189
  • [x] Utilize PK index when load deltalog files to discard delete entries which does not hit #36207
  • [ ] Add deltalog cache for querynode worker to reduce duplicated object store IO
• [ ] Purge delete buffer asap when all l0 segments for current target is ready #35304
• [ ] Split delete data when streaming delete size is too large
• [ ] (optional) Flush delete buffer into oss files when delete buffer is too large, shall be easier to implemented with streaming node

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.