Closed abudnik closed 9 years ago
Is this pull request ready to be merged? If not what should be done?
I think it is ready to be merged. Is there anything to be fixed?
@bioothod can you merge it?
I will review it today or tomorrow and either merge it or let you know if something can be changed
Fixes: https://github.com/abudnik/eblob/commit/aa0bfbfa9e47d951adc3d61d449cb9f9d27d4526
Finally, it is ready to be merged (also with https://github.com/reverbrain/elliptics/pull/575)
1) Problem description If error happens at block i/o device level, then access to memory mapped file, which is stored in this device, may lead to program termination with SIGBUS signal. When elliptics uses multiple backends, then problems of some backend may lead to malfunction of all backends. Elliptics (dnet_ioserv) nodes used to work with one backend only (some time ago), therefore increasing network traffic (route list updates), but isolated problems of a particular backend.
2) Problem solution This PR provides eblob version without usage of memory-mapped files. Now elliptics/eblob use syscalls for read/write operations (pread, pwrite), which returns error code in case of any i/o errors.
3) Estimation of i/o operation cost (pread vs. mmap) Using pread/pwrite for i/o operations leads to additional overhead and resources consumption comparing to using of memory-mapped files. Every i/o operation consists of following steps: system call, data copying between user space buffers and kernel space, memory allocacation and dealocation for data buffers.
4) Affected operations after elimination of mmap
5) Performance comparison on synthetic load tests (pread vs. mmap) We compared performance of pread vs. mmap version of eblob on following load tests:
All read and write load tests where aimed at single elliptics node with 16 backends. Write tests lasts short time (few minutes) and long time (30 minutes) at high rps.
Those tests (accessing small keys) has shown some performance divergence between mmap and pread versions. Tests with heavy-weight keys didn't lead to any performances divergence. Analyzing performance tests leads to next conclusion: in some tests response times increased a little bit, in other tests response times even decreased a little bit; if average and median rps differs, then it differs at a gap of 4-5 % in both sides (faster or slower). Also pread version did not lead to changes in system load (cpu usage, context switches, interrupts, memory consumption and iowaits).