search

cloudius-systems / osv

OSv, a new operating system for the cloud.
osv.io
Other
4.1k stars 603 forks source link

Support for AIO, for Seastar #656

Open nyh opened 9 years ago

nyh commented 9 years ago

Filesystem APIs involving blocking - both the classic read()/write() calls and the newer file-mapping-via-mmap() - are bad for performance because the fact your thread can block forces you to have multiple threads on the same CPU - and having multiple threads inevitably cause inefficiencies like context switches, need for locks, extra wasteful stacks, etc.

For this reason, applications using the Seastar library use AIO (asynchronous I/O), where disk I/O does not block, and one thread per core is enough. Linux's filesystems support for AIO is imperfect: support in ext4 has numerous bugs (i.e., cases which do cause blocking, so are not really asynchronous) and xfs has much fewer such bugs, but still non-zero. But AIO support in OSv is even worse - it is stubbed out :-(

I don't think we need OSv support in ZFS - I think most of ZFS's features (especially its sophisticated caching) are not interesting for AIO and O_DIRECT as used by Seastar. It would probably make more sense to write a new simple file system for this purpose, perhaps making some simplifying assumptions on the use case (e.g., a few large files, modifications are rare except append, etc.) and on the other hand improve parallelism like XFS does (with SSDs with zero seek costs, it is important we can have multiple cores proceed with multiple writes to the same disk in parallel, without getting locked on common data structures in memory or on disk).

Instead of implementing a new filesystem in OSv, another alternative is to have a raw boot device in OSv, and implement the new filesystem in Seastar. This would mean we will just need to implement in OSv asynchronous I/O to a block device.

vinayrmrp commented 8 years ago

+1 for aio support in osv.

raphaelsc commented 8 years ago

@nyh and of course, if anyone is interested in implementing a new file system with AIO capabilities, that would be very welcome :-)

DerangedMonkeyNinja commented 8 years ago

I have a basic version of POSIX AIO in my OSv fork. It's mainly for block I/O devices. If interested, I can submit it to the mailing list for further review.

posix-aio.cc