nyoci_plat_wait() does not exit if a timer is started

[snej]

I'm using nyoci in a multithreaded environment. I'm careful to wrap a mutex around all calls into nyoci to guarantee no re-entrant use, but I'm having trouble with a limitation of the event-loop/timer implementation.

• I've got one thread that's running the event loop, i.e. repeatedly calling nyoci_plat_wait and nyoci_plat_process. To avoid consuming unnecessary CPU with polling, I pass a very long timeout into nyoci_plat_wait.
• This thread has called poll with a long timeout, because there are no timers waking up soon.
• Meanwhile, another thread does something like starting a transaction, which creates a new timer that will fire in (say) 10ms.
• Unfortunately the event loop thread stays blocked in the poll system call, so the new timer doesn't fire. The system basically locks up until either an incoming packet arrives, or the very long timeout expires.

It looks like starting or rescheduling a timer needs to abort any current poll system call. I'm no expert on POSIX I/O, but I know how to search StackOverflow; the answer seems to be to create a pipe and add its output file descriptor to the set being polled; then you can write to the pipe and the poll call will exit. (The alternative is to raise a signal, but apparently this is subject to race conditions.)

(Obviously the solution will be platform-dependent since not every platform uses the POSIX poll or select calls. But it happens that I'm only using the POSIX code so far.)

[snej]

Oh, I should mention for others' benefit that the workaround is simply to pass a short timeout, like 100ms, to nyoci_plat_wait. It's just a trade-off between latency and CPU usage.

[darconeous]

Yeah, your analysis is correct. This is a fairly common problem when dealing with async-io and the solutions all tend to be platform-specific. This isn't so much a bug as it is just an inherent limitation of nyoci_plat_wait(): that method was only intended to make writing simple loops easy and straightforward. I've intentionally limited the scope since stuff like this is inherently platform-specific anyway.

If you are on a unix-y platform, then creating a pipe and writing to it to wake up the thread will work fine. If you are using something that has full thread management (that can queue tasks, monitor socket, etc. all automatically) then you could use that to make sure that any calls to the API simply get queued to run on the "nyoci" thread. On iOS and macOS, dispatch queues would be the way to go. (God I miss using GCD...)

If you are on an embedded OS that is emulating the posix sockets (Like LWIP), then you might not have the pipe trick available to you; but it might have other tricks that you could use to wake the thread up.

Note that you'll not only need to wake up the thread, you'll also need a way to wait for it to go back to sleep. A reasonable solution would involve two methods, something like snej_coap_thread_lock() to wait for the nyoci service thread to go to sleep (and temporarily prevent it from servicing timers or packets) and snej_coap_thread_unlock() to wake up the nyoci service thread and have it resume processing.

But yes, your work-around will do the trick in a pinch, assuming you have a way to lock the service thread while you are calling into libnyoci from other threads.

[darconeous]

The more I think about it the more I think it might make sense to actually add lock/unlock methods to the posix platform to allow nyoci_plat_wait() to be used in the way you describe. I'll have to sleep on it though.