cubefs / cubefs

cloud-native distributed storage
https://cubefs.io
Apache License 2.0
4.57k stars 651 forks source link

[Enhancement]: append-only write performance optimization #3459

Open bladehliu opened 1 month ago

bladehliu commented 1 month ago

Contact Details

No response

Is there an existing issue for this?

What would you like to be added?

For application scenarios of separated storage and computing (such as databases and retrieval systems), the performance of WAL (Write-Ahead Logging) append writes is extremely important. Append writes require two steps: replicated writes to datanodes + replicated writes to metanodes.

We can further optimize it from two-step operations to one-step operation to reduce latency.

Why is this needed?

No response

Anything else?

No response

leonrayang commented 3 weeks ago

I have an idea: how about we treat WAL as a special case and set a flag on its inode when a new extent is created? The client can then asynchronously update the inode’s extent details. Even if the update fails, it can still proceed by fetching data from the basic extent information.