Async state preserving reentrant Kernel

[imrn]

Currently, Kernel.run() blocks the thread until it's done with the tasks at the hand. If curio is embedded in an other application you should either:

• Feed the kernel with coroutines with short running times. That, is you can not have long lived and state preserving embedded async apps.
• Or run curio in a seperate thread. But this time you need to take care of thread handling and most importantly guard against thread state dependent cross thread accesses and route them through thread safe interfaces/queues.

Here we'll discuss about providing semantics for:

• Returning from Kernel.run() on demand, preserving all async state: Coroutines and other threads should be able to trigger this action. The trigger is provisionally named as a pause() function. But suggestions for other possible mechanisms are also welcome.
• Providing re-entrant Kernel.run() so that subsequent calls will make curio continue from its previous freezed/paused state.

You can also find previous discussion at #111.

[dabeaz]

Of course, maybe my head is just fried from yesterday (always a distinct possibility). The main thing is that run() only has those two operations.

[imrn]

No problem. May be it should have a better presentation on what is currently covered and what can be possible discussion points. I think daemon removal thing is related with the whole picture. I'll post a more digestible version. Meanwhile, you can post item wise questions if there are any.

[imrn]

Yes.. The main theme is always like that: With a coroutine and without. No deviation.

[njsmith]

That seems like two pretty different meanings for timeout= overloaded onto the same argument?

On Feb 10, 2017 4:46 AM, "David Beazley" notifications@github.com wrote:

There are things in the list that appear to attach different semantics to different values of the timeout when, in fact, the timeout is just passed directly to select() and there's no other interpretation of it. run(coro, timeout=secs) is the same as run(timeout_after(secs, coro)).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dabeaz/curio/issues/172#issuecomment-278934204, or mute the thread https://github.com/notifications/unsubscribe-auth/AAlOaHrw4CXSaZ0uPK6QgJNa8I42Cxzfks5rbFwcgaJpZM4L3gG5 .

[dabeaz]

How do you figure? If given no coroutine, the timeout is the maximum amount of time that will be spent waiting for something to happen. If given a coroutine, the timeout is the maximum amount of time it's allowed to run before being cancelled. Seems fairly consistent to me unless I'm missing something.

[dabeaz]

The run(coro) option above describes non-existant behavior. run(coro) always runs the supplied coroutine and non-daemonic tasks to completion. It returns the result of the coroutine. There are no options to detach from it or have it operate in the background. If this is wanted, then use run(spawn(coro, daemon=True)) instead.

I don't anticipate any of those options (a) - (c) being supported.

[dabeaz]

This is not so much a comment on the specifics of run(), but more of a general thought about repeatedly invoking Curio to make it do things...

In the big picture, the primary use case of Curio is in problems related to I/O handling--especially network handling. Potentially, this could involve a substantial number of I/O operations under load. Because of that, I'm not sure it would make sense to drive Curio by repeatedly invoking run(). For example, if you invoked run() on a periodic timer (say 10ms), you would introduce a 10ms latency into nearly single I/O operation. The only way to get around that would seem to be repeated calls to run() as fast as possible---except that it leads to busy waiting and a 100% sustained CPU usage. Neither option seems particularly desirable to me.

For something like a GUI, the I/O problems are largely alleviated if you put Curio in charge and you make periodic callouts to the GUI on a time interval. Alternatively, if you put Curio and the GUI in separate threads and hook them up via queues, the I/O performance is also solved.

I'm not sure this makes much sense or not. I guess I'd be more interested in knowing more about the use case for structuring things in a way where the Curio run() method is repeatedly triggered like this. Are there are a lot of I/O operations being performed in the background? If not, what is Curio being used for in this environment?

[imrn]

• Embedding curio into sync applications which would not mind some blocking (short or long). That is; blocking indefinitely or for sometime waiting Curio return back. The sync app will block anyway if it does the same work itself. You may even adjust the blocking time depending on the application, creating a "time shared thread". Note that fine control of blocking time time is crucial for adaptation. "Long or short Blocking" is the key. Without it, continous calling will lead to 100% cpu -a useless method.

• And there is run wise slicing. Depends on if your workload can be divided into small calls or not. Parent application sends small work loads as it see fits. Or a big one with oneshot/timeout combinations. Infact blocking time and runwise slicing are two sides of the same coin.

These are the reasons for requesting fine controlled run().

[imrn]

libevent and libuv both have similar functinality:

https://pyuv.readthedocs.io/en/v1.x/loop.html http://www.wangafu.net/~nickm/libevent-book/Ref3_eventloop.html