rosedblabs / wal

Write Ahead Log for LSM or bitcask storage(or any append-only write).
Apache License 2.0
234 stars 38 forks source link

Batch writes #23

Closed amityahav closed 1 year ago

amityahav commented 1 year ago

it would be nice to support batch writes. i want to be able to buffer all of my DB modifications that are appended to the WAL during a batch insert and only when the batch insert is completed flush it to the underlying file.

roseduan commented 1 year ago

I have thought about this before, but I need more time to design how to implement it.

amityahav commented 1 year ago

this can also optimize batch inserts in rosedb

roseduan commented 1 year ago

Here is my thought:

  1. Add a function named PendingWrites(data []byte)
    • all new data will be maintained in WAL pendingWrites [][]byte
  2. Add a function named WriteAll
    • write all data in pendingWrites
    • update blockNumber and blockSize
    • reset the pendingWrites in wal
    • return all ChunkPosition