mmap elimination

[abudnik]

1) Problem description If error happens at block i/o device level, then access to memory mapped file, which is stored in this device, may lead to program termination with SIGBUS signal. When elliptics uses multiple backends, then problems of some backend may lead to malfunction of all backends. Elliptics (dnet_ioserv) nodes used to work with one backend only (some time ago), therefore increasing network traffic (route list updates), but isolated problems of a particular backend.

2) Problem solution This PR provides eblob version without usage of memory-mapped files. Now elliptics/eblob use syscalls for read/write operations (pread, pwrite), which returns error code in case of any i/o errors.

3) Estimation of i/o operation cost (pread vs. mmap) Using pread/pwrite for i/o operations leads to additional overhead and resources consumption comparing to using of memory-mapped files. Every i/o operation consists of following steps: system call, data copying between user space buffers and kernel space, memory allocacation and dealocation for data buffers.

4) Affected operations after elimination of mmap

• Index element lookup in index file => all operation of read and write to eblob backend
• Sorting of index files
• Record iteration
• Checksumming

5) Performance comparison on synthetic load tests (pread vs. mmap) We compared performance of pread vs. mmap version of eblob on following load tests:

• read at 10k rps: records 0.5-2.5 Mb
• read at 12k rps: records 100 bytes - 10 Kb
• write from 4 writing nodes (elliptics client): records 100-500 bytes
• write from 1 writing node: records 100-500 bytes
• iterate all keys

All read and write load tests where aimed at single elliptics node with 16 backends. Write tests lasts short time (few minutes) and long time (30 minutes) at high rps.

Those tests (accessing small keys) has shown some performance divergence between mmap and pread versions. Tests with heavy-weight keys didn't lead to any performances divergence. Analyzing performance tests leads to next conclusion: in some tests response times increased a little bit, in other tests response times even decreased a little bit; if average and median rps differs, then it differs at a gap of 4-5 % in both sides (faster or slower). Also pread version did not lead to changes in system load (cpu usage, context switches, interrupts, memory consumption and iowaits).

[shaitan]

Is this pull request ready to be merged? If not what should be done?

[abudnik]

I think it is ready to be merged. Is there anything to be fixed?

[shaitan]

@bioothod can you merge it?

[bioothod]

I will review it today or tomorrow and either merge it or let you know if something can be changed

[abudnik]

Fixes: https://github.com/abudnik/eblob/commit/aa0bfbfa9e47d951adc3d61d449cb9f9d27d4526

Finally, it is ready to be merged (also with https://github.com/reverbrain/elliptics/pull/575)