exxeleron / qPython

interprocess communication between Python and kdb+
http://www.devnet.de
Apache License 2.0
152 stars 89 forks source link

Asyncio-friendly non-blocking I/O #60

Open nirvana-msu opened 6 years ago

nirvana-msu commented 6 years ago

Hello,

Is it possible to use qPython with asyncio without resorting to things like run_in_executor (i.e. ThreadPoolExecutor)? Both synchronous sync and asynchronous receive calls are blocking, making it impossible to use with asyncio (without running in a separate thread).

Ideally, I would like to do something like the following. Note that I've made up methods sync_async (yeah, not a good name) and receive_async:

async with qconnection.QConnection(...) as q:
    result = await q.sync_async(query, **query_kwargs)

and/or

async with qconnection.QConnection(...) as q:
    q.async(query)
    result = await q.receive_async()

In fact, there'd be no need in two methods (?) - we just need a method returning a future which we could await:

async with qconnection.QConnection(...) as q:
    future = q.async(query)
    result = await future

Another related problem I currently face is that all socket reads are blocking, which makes my program very slow (I want to read multiple large results concurrently). I'm not certain of technical details, but it seems like it should be possible to do non-blocking concurrent socket I/O (e.g. asyncore seems like it does the job?).

Thanks!