how to use curio with redis/mysql or other databases?

[iffybug]

I would like to use redis in my curio program.

IMHO, if I use like redis_conn.set('foo', 'bar') in a coroutine function, the whole thread will block until the operation is complete. I could use curio.run_in_thread(redis_conn.set, 'foo', 'bar') instead, but is this the best way at the time being? is there a curio-native library?

[dabeaz]

Right now, there is no curio-native library so threads might be your best bet. This is definitely something that I want to address in the future though.

[iffybug]

Thanks for your reply!

A further question, do we have to invent all the wheels around curio?

For day to day use, I would need web frameworks, message queue wrappers, database connectors and so on. Most of these popular synchronous libraries are NOT sans-io if I'm not mistaken, so it's not easy to port them directly into curio. And although there is a sans-io library like h11, it takes a lot effort to write a library like asks(requests for curio and trio).

As of asyncio lib, there seems to be a lot libraries reinvented, such as sanic for web frameworks, aiohttp for http client library, aredis for redis client and many others in the aio-libs org

AFAIK, for curio, I only got asks for http client. There are not many other libraries I could use. So my question is:

For the curio ecosystem to emerge, do we have to reinvent all the io-related libraries we need? is there a possibility that we could put a wrapper around the synchronous or asyncio-based libraries and then magically use them with curio? How could I help?

I'm trying to write a web crawler with curio. Right now, the networking part is the only part I could use curio(thanks to asks). When I need to output the result to database, I use a UniversalQueue to connect curio with normal synchronous thread.

[imrn]

On 9/15/17, Yifei Kong notifications@github.com wrote:

For the curio ecosystem to emerge, do we have to reinvent all the io-related libraries we need? is there a possibility that we could put a wrapper around the synchronous or asyncio-based libraries and then magically use them with curio? How could I help?

The problem is not specific to Curio. There is no magic to make sync libs to work seamlessly with asyncs. As David mentioned async threads are your best option.

Support for Asyncio is an interesting topic for which I'd personally down vote. Of course, this is my subjective view. I far as I know (correct me if I'm wrong) python-trio tries to provide something like that.

[dabeaz]

Support for things like database access in Curio is an interesting one. In a perfect world, we'd write a Curio-native library for this and have it "work." But I'm also wondering what is actually gained by doing that in the big picture. For example, accessing a database server isn't the kind of thing that seems like it would really require the supposed scaling benefits of async. By that, I mean, it doesn't seem likely that you'd be sitting there with 5000 open connections to a database server. You'd more likely have a small pool of connections and multiplex requests across that. And for that, it would probably work just as well to use threads.

With that in mind, I wonder if one route to Curio-friendly libraries might be to simply provide a nice "async API" that acts as a kind of proxy between Curio and an existing synchronous library. This isn't the approach I'd want to take for everything--for example, I think HTTP should be native. However, I bet it would work fine for a lot of other things.

[quantverse]

look here (had not tried yet): https://github.com/living42/curio-redis