fs-integration

[fengggli]

TODO

• File operations: remove file append to file

• Capture system call

  • open/close
  • read/write

• systemutils

• fio test

  • simple implementation
  • dual-channel version.

• Intercept

  • read/write can be interepted.
  • experiment 1 show overhead of fuse by tracing. (why set max_write as 128KB? is the default lower than this? see this issue 161 in libfuse )
  • experiment 2 can be comparing performance of dual path or all direct.
  • Explain differences of this with direct-fuse and fusionfs. (zero-copy in userspace?)
  • dd/ perf

[fengggli]

with fio

1. capture all the systemcalls of fio(sync/psync engin at https://github.com/fengggli/fio/blob/v3.1/engines/sync.c) , so that i can intercept them.
2. Currently client send request to server to allocate iomem and map to local address.

3. call stack

   • backend.c: init_io_u: (https://github.com/fengggli/fio/blob/b6b662b64c70bf043bbafd654f9bf98513ea5dc9/backend.c#L1226-L1239)
   • memory.c : allocate_io_memory: (https://github.com/fengggli/fio/blob/b6b662b64c70bf043bbafd654f9bf98513ea5dc9/memory.c#L289-L336) (This actually checks MMAPHUGE/SHMHUGE, etc).
   • ioengines.h gives a hook where different engines can specify there iomem allocation/free, but I didn't see engions does so, Instead, there are different options avaialble as specified in memory.c
   • I could do that: (https://fio.readthedocs.io/en/latest/fio_man.html#cmdoption-arg-iomem)

     ./options.c:                        .oval = MEM_MALLOC,
     Binary file ./client.o matches
     Binary file ./backend.o matches
     Binary file ./libfio.o matches
     Binary file ./steadystate.o matches
     Binary file ./rate-submit.o matches
     Binary file ./blktrace.o matches
     Binary file ./time.o matches
     ./thread_options.h:     MEM_MALLOC = 0, /* ordinary malloc */

   • get_io_u: https://github.com/fengggli/fio/blob/b6b662b64c70bf043bbafd654f9bf98513ea5dc9/io_u.c#L1714-L1715
   • static void fio_init fio_syncio_register(void) (At this point io mem is already allocated.)

4 attribute(Constructor can be useful)

[fengggli]

intercept:

https://stackoverflow.com/a/4586534/6261848 use --wrap in gcc

https://www.apriorit.com/dev-blog/537-using-constructor-attribute-with-ld-preload

https://www.tldp.org/HOWTO/pdf/C++-dlopen.pdf

https://stackoverflow.com/a/43005999

(py36) lifen@sievert(:):~/Workspace/vagrantvm/vagrant-ubuntu18-spdk1810/comanche/build$ltrace ./src/fuse/ustack/unit_test/test-preload
__libc_start_main(0x4005a0, 1, 0x7ffe6d64cce8, 0x400730 <unfinished ...>
open64("foobar.dat", 65, 0700)                                                                                                                     = 3
close(3)                                                                                                                                           = 0
__fprintf_chk(0x7f4ad696e540, 1, 0x4007d4, 0x4007ce[LOG]:done!
)                                                                                               = 21
+++ exited (status 0) +++

I checked the client side system call traces, all mmap are called without huge_tlb, I could intercept the mmap call, if it contains huge_tlb, use special memory, other wise..

with fio

1. the thread_main is started as a forked process.
2. flags in mmap is 0x40022 (mem=mmaphuge)

Note

get performance of simple workload first

2. Some guideline about POSIX file operations: https://www.classes.cs.uchicago.edu/archive/2017/winter/51081-1/LabFAQ/lab2/fileio.html

[fengggli]

whole-file write-throughput

Performance

• ustack-naive also uses O_SYNC|O_DIRECT, otherwise writes will be split to 4k writes/ 128k writes(if -o max_write=131072 -o big_writes is specified in server)
• also a manually flush after each iterations.
• I only run 1000 iterations for initial testing.

[fengggli]

compare between kvfs-ustack and kvfs-naive

1. Since currently both of them use kvstore backend with cache(lock/unlock), they are actually fast.
2. Problems are that intercepted write doesn't know its an O_SYNCED open or not. I shall check that in the server-side, save that information in filemeta Now kvfs-daemon can read the ioflag