search

canonical / dqlite

Embeddable, replicated and fault-tolerant SQL engine.
https://dqlite.io
Other
3.89k stars 218 forks source link

posix_fallocate extremely slow on ext3 file system #653

Open LakshK98 opened 6 months ago

LakshK98 commented 6 months ago

posix_fallocate is extremely slow on ext3 file system (~12s for 8mb file) when file is opened with O_DSYNC flag. This causes overall slowdown in dqlite execution when log/snapshot files are to be written. For instance, the average response time per request increases from 5ms for 1000 requests to a sluggish 25ms for 10,000 requests on our 3 node cluster setup.

I tried manualling setting fallocate to false in uv_fs.c here so that the fallocate emulation logic is executed. This reduces avg response time for 10k requests to around 5-6 ms.

Would it possible to set fallocate to false on ext3 file systems in probeFallocate?

cole-miller commented 6 months ago

Sorry for the delayed response. This is unfortunate, but I'm hesitant to add (more) carveouts for specific filesystems to our code---especially for ext3, which (aiui) has been superseded by ext4. Maybe we should just add a general mechanism, like an environment variable, for dynamically overriding dqlite's decision to use fallocate, etc. (But this has security implications: if you can control this environment variable you can control dqlite's performance.)