New Release. Also, thinking about project scope

[dabeaz]

Just a heads up that I will be making a new Curio release fairly soon. With that in mind, I've also been doing a bit of thinking about the overall scope of the project and the whole library vs. framework issue. It has always been my view that Curio should be more of a low-level library that provides a basic set of functionality related to concurrency and I/O from which one could build more interesting things (as opposed to being a budding framework that trying to do everything). Thus, I've been doing some project cleanup that involves removing some of the current functionality, moving some other things over to the examples directory and generally trying to make Curio leaner.

I don't have an exact ETA for the new release, but it will happen sometime in February.

[Fuyukai]

What (if) anything else is going to be removed in cleanup pre-next release?

[ghost]

I’d be interested if it’s possibke to get a heads up in that.

I what I can tell you specifically in regards to the question of the future is this:

I use the subprocess, ssl, the aopen function and socket module a lot.

I like the idea behind Kernel as the more powerful way of controlling events/loops. Is this intended to be subclassible?

I’d love if there was a way to subclass a TCP server tho I suspect limitations around the std libs docket module might be more to blame here.

I am always game for more transparent ways to handle sockets.

I use zeromq extensively professionally and their official library runs with curio fine. I dunno if you need to maintain yours? I just mention that only because I love what you have done with Channel and I think it’s far more interesting.

For the moment that’s what i got I may come back and respond with more as I think about this

[dabeaz]

Probably the big, and maybe controversial, removal is that I'm going to nix Locals. The implementation is already problematic in light of curio's mix of async, threads, and processes. There is also a PEP and the addition of contextvars in Python 3.7. So, all things equal, I'd prefer to push people towards that as opposed to maintaining a curio-specific version of it.

Basic sockets and I/O will stay, but anything that resembles an "application level" protocol (i.e., HTTP, ZeroMQ, Redis, etc.) would be moved to separate projects. At the moment, that only applies to the ZeroMQ submodule I cooked up. I'm more interested in trying to keep a "stable" core of basic functionality as opposed to "Curio" being an umbrella project for every possible protocol or application.

On subclassing TCP servers, I'm game for thinking about ways to customize that further. Same goes for Channel objects.

On the Kernel, one of the initial goals of the project was to make the Kernel as invisible as possible--partly in reaction to the complexity and oversized role of the event loop instance in asyncio. But also viewing it more like an opaque environment much like the kernel in an operating system (it's in charge, but async tasks don't really get a hook to look inside or interact with it directly). I'm also trying to do as little as possible in "kernel space" as possible. Aside from basic task switching and scheduling, there's not much going on there.

I'm a bit curious to know what you might be thinking on subclassing the kernel. There are certain hooks by which one can do custom things with the kernel. I have also implemented different kernels as experiments. For example, I've written a version of the Curio kernel entirely as a C extension (which was interesting, but not interesting enough for me to actively pursue it further at this time).

[ghost]

@dabeaz

Edited Heavily from previous post.

Edit2: fixed Run class example so it can handle multiple arguments for a function

Edit3: for clarity

okay, so when looking at say socket sub-classing I'm just thinking we could have something that easily proxies the underlying Socket architecture? I am looking to see if this can be achieved easily with the Socket class of course, since its preexisting, but I think there are limitations there verses proxying socket.socket() and attaching attributes post inception perhaps.

In as so far as the Kernel sub-classing goes, I am really looking at run. I mocked up a working (albeit not very pretty) Run class that takes the instantiating arguments of the Kernel + those of run, and runs the tasks immediately.

class Run:
    def __init__(self,
                 coroutine_function,
                 *args,
                 shutdown=True,
                 selector= None,
                 debug=None,
                 activations=None):
        self.__obj = Kernel(
            selector=selector, debug=debug, activations=activations)
        self.__call__(coroutine_function, *args, shutdown=shutdown)

    def __call__(self, *args, **kwargs):
        return self.__obj.run(*args, **kwargs)

Super simple, and I think would be cleaner to use instead of the run function. It also allows for a clean way of passing debug, selector, and activations settings to the Kernel object without having to import it as well.

I also wonder if shutdown=True shouldn't be the default, instead of False? Be interested in your thoughts on that as I'm assuming the false is to keep it open for long running tasks or is it just cleanup?

I'm not seeing any downside to do it this way as a standard, as it avoids having to remember when exactly you should use Kernel.run() and when you don't.

Something similar for TCP/UDP server might also be par for the course as well, I'm trying to mock such a thing up today.

Here's another proxy-ish mockup I was thinking about, since the Kernel is a complex beast after all :)

# TODO: Should a separate class should exist for when you are running it as a context method?
# TODO: How should kwargs to a function be handled? Must they be declared explicitly only?\
# explicitly defined kwargs work fine.

class Run:
    _selector = None
    _debug = None
    _activations = None

    def __init__(self,
                 corofunc,
                 *args,
                 shutdown=True,
                 shutdown_funcs=list()):

        self.coro = corofunc
        self.shutdown = shutdown
        self.shutdown_funcs = shutdown_funcs
        self._kernel = Kernel(
            selector=self._selector,
            debug=self._debug,
            activations=self._activations)
        self._kernel._shutdown_funcs = shutdown_funcs
        if monitor or 'CURIOMONITOR' in os.environ:
            m = Monitor(self._kernel)
            self.shutdown_funcs.append(m.close)
        self._kernel.run(corofunc, *args, shutdown=shutdown)

    @property
    def activations(self):
        return self._activations

    @activations.setter
    def activations(self, value=None):
        self._activations = value

    @property
    def selector(self):
        return self._selector

    @selector.setter
    def selector(self, value=None):
        self._selector = value

    @property
    def debug(self):
        return self.debug

    @debug.setter
    def debug(self, value=None):
        self._debug = value

   @classmethod
    def run(cls, coroutine, *args, shutdown=True, shutdown_funcs=list(), monitor=False):
        cls(coroutine, *args, shutdown=shutdown, shutdown_funcs=shutdown_funcs, monitor=monitor)

The TODOs are also things I'm thinking about. It'd be pretty easy to run something in context for synchronous functions with a @classmethod similiar to run.

Anyways, let me explain the idea behind the new Run class:

I'm trying to keep it simple. I figure most of the time someone will just want to run their async funcs, so you can just call the class with Run(coro, *args) and be done with it. You can also register shutdown funcs here as well (or is that not suppose to be an open thing?, its unclear. Looking at the Kernel code I'd think this is for me to hand it functions as the end user, like closing a db connection)

If you instead need to set a property on the Kernel, you simply assign Run without using the init, so the eager call to run never gets called. Then you can set the properties, and use Run.run to run the funcs instead.

Does that appeal to anyone? Not sure. Makes sense in my head I suppose, but I do wonder what others think of this idea. I'm going to try and refine it as much as possible.

[ghost]

@dabeaz should I pull request a Run class or is this something we still want to discuss?

[dabeaz]

Need more time to review and discuss.

[dabeaz]

Can you show me some usage examples of what this would look like? At a high level, I don't want curio to have a lot of extra plumbing involving objects just to get things running. Right now, there are two basic modes of operation. One option is to run a single coroutine. This creates the kernel, runs the coroutine, shuts down the kernel.

 run(coro)

Or you can submit multiple coroutines to a single kernel:

with Kernel() as kern:
        kern.run(coro1)
        kern.run(coro2)
        ...

Both are pretty simple in my opinion. What extra thing is the Run class giving me?

[ghost]

I'm planning on mocking up a more complete version of this later. In essence it boils down to having 1 entry point to the Kernel without having to think about whether I should use a context manager or not. A run class would allow you to do like Run(func, *args) or Run((func1, func1, func3...),*args)

Just less for the end user to have to keep track of. I'm still working out the argument kinks but thats what it boils down to. If you feel like its not worth pursuing though I understand.

LATE EDIT:

I couldn't really come up with a better solution than one that you have. I'll be on this drawing board for awhile. My gut feels like there is some room for optimization here I just gotta think longer about it. Looking forward to future releases!

[Fuyukai]

Has this been pushed back to this month? The task group changes are blocking a new ver of one of my projects currently.

[dabeaz]

I'd like to make a new release before next week because I'm going back into more of a teaching schedule. However, tell me more about this "task group change" issue. Is something in the current Github curio broken?

[Fuyukai]

The opposite; I need #239's fix but it's only available on the current git release, and not the stable release.

[dabeaz]

I just pushed a new release. There are a variety of things I still want to do, but I'm going into a period of teaching and won't much time to work in the short-term. Curio currently passes all tests on Python-3.5, 3.6, and 3.7b2 so now seems as good a time as any to push something.