search

omnilib / aiomultiprocess

Take a modern Python codebase to the next level of performance.
https://aiomultiprocess.omnilib.dev
MIT License
1.75k stars 99 forks source link

Consider aiomultiprocess with stackless Python #19

Open misterpilou opened 5 years ago

misterpilou commented 5 years ago

I'm maybe being a little devil here, but is there a way of speeding up more aiomultiprocess with stackless python and the stackless module. I would do some tests, i'm building a crawler using aiomultiprocess which have pretty good performance thanks to @jreese. Is there anyone would already think/implement that?

amyreese commented 5 years ago

I'm not familiar enough with Stackless Python to know whether there is any performance gains to be had, but I do know that an easy win is to try uvloop as a drop-in replacement for the standard library's asyncio event loop. There are still a few small incompatibilities with certain use cases, but it's pretty comprehensive these days, and for network related tasks, it gives a significant performance boost.

That said, if you or anyone else wants to test adding features to aiomultiprocess to support Stackless or similar alternatives, I'm more than happy to review pull requests.

misterpilou commented 5 years ago

Sorry for the late reply i was considering Dask as a framework for my crawler. Anyway i would be glad to test stackless python with aiomultiprocessing. It should be very fun. I will first try to use uvloop to see how it improve the benchmark. Then i will try to thread aiomultiprocess. I'm just a little scare to deal with your library code cause i sometimes find it too complicated for me from what i saw.

misterpilou commented 5 years ago

So stackless, from what i understood, would be irrelevant. I read the documentation, and it is a kind of asyncio but without much of the benefit. I think it would be funny to do some tests with it. Anyway, i thinked that it could leverage the power of threads in python but tasklets is more a old asynchronous task. You can do some tests, i think it would be very funny if you got time. Anyway, what i meant was to wrap aiomultiprocess in ThreadPoolExecutor. I don't know if you already tried to do that.