lieuwex / LirsFS

MIT License
0 stars 0 forks source link

Decide on splitting big writes into multiple Raft `Write` operations or not #45

Open sholtrop opened 1 year ago

sholtrop commented 1 year ago

This leads to consistency problems. If we split a big write into n writes, a client should not be able to read the file while < n writes have been applied.

lieuwex commented 1 year ago

Is there a disadvantage of not splitting them? And maybe webdav already splits them in chunks?

sholtrop commented 1 year ago

Not splitting them means a huge write will sit in our raftlog until the log gets compacted. I am speculating that raft compacts the log at a certain amount of log entries rather than time or accumulated log size in bytes. In that case, the earlier write parts could get compacted out before we apply the newer ones, meaning we don't need x bytes of storage to handle a read of x bytes.

Writing this out, I'm realizing how small the gain would be compared to the complexity, so let's skip it for now.

If webdav splits the writes, we'd have the same consistency problems as before. But we'll see about that then.