Platform differences: kqueue, epoll

[imrn]

For linux users (I'm one) default selector is epoll, for mac/bsd folks it is kqueue. epoll works with sockets. But it gives error for regular files. If you fall back to poll or select it 'seems' they work but in fact these selectors always report readiness for files. If you go ahead, you'll block. The blocking is expected to be 'short' but it may not be the case. So Curio uses threads to avoid blocking. That's the current situation.

I wonder if curio can take advantage of kqueue facilities on supported platforms. I read that one can give anything with a file descriptor to kqueue. Even timers are natively supported. Does taking advantage of such facilities mean a special fork of Curio kernel? Currently I'm not a user of those platforms -but thinking about it.

So this issue is created for discussing platform specific selectors Curio may or may not use.

[dabeaz]

As far as I can tell "async" for regular files is a gigantic wasteland of fail on Unix machines. Basically it doesn't work. You could try using some POSIX aio_* functions for it, but those apparently don't really work either. Maybe that's overstating the problem, but punting file I/O out to threads seems to be about the only sane approach. As far as I can tell, other libraries like libevent basically do the same thing with regular files.

Maintaining separate forks of the kernel is not something I'd probably want to do. Although, I suspect at some point there will be an IOCP implementation that would operate quite differently. If Curio has done its job with the higher-level APIs though, that might not effect user code though.

[njsmith]

Kqueue has a way to watch regular files, but it's the equivalent of Linux inotify, not aio.

Windows iocp does support async file io though.

[imrn]

It is ideally right thing to expect file watching and timer capabilities from the platform. In the non-ideal world, Curio embeds clocking infrastructure in the kernel and provides a threaded file class to cover all platforms. My proposal is this:

1) Files: Base Curio should not treat files differently and send them to the selector along with sockets. But depending on the platform a specific File implementation or current threaded File would be used. I think, this is a low hanging fruit.

2) Timers: Do not implement clocking in the kernel and again expect it from selector. This would be tricky (or may be not) because current clocking code diffused into the kernel and need substantial refactoring to make it optional depending on the platform. There may not be a quick fix. But I suggest keeping this in mind as a future roadmap.

These are analogous to making Base Curio only working with kqueue. But it will bring in an important modularity for providing infrastructure for platform ports, which will enable Curio to perform its best everywhere. And it can make Base Curio much simpler.

David: I think you are Mac user. Would you like to experiment with (1)?

http://stackoverflow.com/questions/14299928/kqueue-on-regular-files

[imrn]

Let's think about a caching http proxy: Using a curio file means submitting 'very single file call' to the thread pool and getting back the result. This is a serious overhead and can not scale for highload configurations.

To be honest it is not a Curio created problem but using the threaded file on all platforms, cripples the ones which infact already have support for native file selectors.

[dabeaz]

I'm not planning to pursue any different support for non-blocking I/O on regular files because it doesn't actually seem to work regardless of whether or not you can pass a file descriptor to an underlying polling mechanism. Every package I've ever seen for supporting regular files in an event loop has involved thread pools. For example, see https://www.remlab.net/op/nonblock.shtml

[imrn]

I know that page. Note that, his tone leads anyone to the impression 'it can not solved'.

But if you read it carefully he is infact talking about 'poll()' and 'select()' with known deficiencies which I already mentioned above: They always report files as ready.

kqueue is expected to work correctly and it is already under your hand. I'd recommend having a try.

[imrn]

http://people.eecs.berkeley.edu/~sangjin/2012/12/21/epoll-vs-kqueue.html

Last Section: This is something inverted though. disk file support In this model, you simply issue I/O operations on the disk files, and get notified when they are done. kqueue supports this approach with the EVFILT_AIO filter type

[dabeaz]

I'll keep it in mind, but I'm stretched pretty thin right now so this isn't something I can really go exploring right this moment. I'm also not convinced that "asynchronous" I/O on regular files actually works on OSX any better than it does elsewhere--which is to say that it probably doesn't.

[imrn]

Any creative ideas despite those? At least we now have some explanation..

This probably means file is ready. However, we'll still need to issue read() and we'll still wait for sometime if it is not already in os buffers:

• EVFILT_READ, Vnodes Returns when the file pointer is not at the end of file. data contains the offset from current position to end of file, and may be negative. This behavior is different from poll(2), where read events are triggered for regular files unconditionally."

Another blow.. It sounds good, until you read the last sentence.. All act as non-blocking while there is space at buffers and you know what that buffer is. EXCEPT FILES.. That is you'll write blindly and pray that os has enough buffers. And you will never know when blocking is coming.

• EVFILT_WRITE Returns whenever it is possible to write to the descriptor. For sockets, pipes and fifos, data will contain the amount of space remaining in the write buffer. Note that this filter is not supported for vnodes.

For AIO, we still need to issue the read/write call in an other thread and our selector in Curio would detect the end of operation in the home thread and continue the tasks accordingly. So we still can not eliminate threads..

• EVFILT_AIO When the aio() system call is made, the event will be registered with the specified kqueue, and the ident argument set to the struct aiocb returned by the aio() system call.

https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2

[dabeaz]

Closing for now. Open to revisiting this topic later should it be of interest.