What does it take to transform a library to use cl-async

[rogersm]

I would like to have access to postgres from cl-async, what is needed to develop a pure lisp library (like pg-dot-lisp) to be used with cl-async?

[PuercoPop]

One would have to make all i/o calls through cl-async, which supports tcp and unix sockets. From perusing [pg]() that would mean rewriting/adapting the code in low-level.lisp and the code that established the connection in the corresponding v2 and v3 files.

You may want to see how the cl-rethinkdb driver does it or how mongo-cl-driver does it. The latter even supports multiple ways to do the i/o: cl-async, iolib and usocket.

Do you intend to update pg or start from scratch?

I'm sure @orthecreedence has more detailed commentary.

[orthecreedence]

Yeah that about covers it (thanks, Javier :smile:).

Ideally, the driver you want to convert will completely separate the protocol implementation from the transport. This makes it easier, because the part that handles the protocol can be wrapped in an async layer, while ignoring the usocket transport stuff (because you're building your own in cl-async). So you could have a pg-async package that loads pg and basically piggyback the protocol handling in pg and replace the transport stuff with cl-async sockets.

If you understand the Postgres protocol and have some experience with async programming, then converting a well-written driver shouldn't be too bad.

An async Postgres driver is one I've wanted for a long time but have hit my capacity for projects I can realistically maintain. I'm also somewhat clueless to the protocol itself. To start, I would pick apart pg-dot-lisp or cl-postgres until you get a good feeling for how they work, then little by little start implementing the async sockets around the protocol. If you're lucky, the project authors will not assume blocking sockets in the protocol parsing. If you are unlucky, you will probably end up rewriting a lot of the protocol parsing yourself. If you do have to do any rewriting, my suggestion is to compartmentalize it into a low-level protocol parsing package (like we did with http-parse/fast-http) so others can build on top of it if needed. Hopefully you won't have to go that far :smile:

Also, I'm around to help answer questions or bounce ideas off of. I've done a few async lisp drivers in cl-async now so can hopefully help out if you hit any snags.

[defunkydrummer]

Hi @orthecreedence,

Asking this question because I was looking for the same question and ended up easily at this issue page, so I bet other people will land here as well.

One solution is the one proposed by @PuercoPop; but I was thinking, on the other hand that perhaps an alternative solution is using threads.

Let me explain in depth: Typical "async" web development relies on having non-blocking I/O. However, as one knows, the non-strictly-IO code runs on the main thread so it is always "blocking" the event loop for a tiny amount of time (which can get bigger depending on the complexity of the operations perfomed).

So in reality one could not only ensure that the IO is done non-blocking, but that the whole operation is done in a non-blocking way. A way of doing this is by having the whole operation run on its own thread, so the main (event loop) thread is never blocked.

Thus, a possibility for transforming any library into a non-blocking "driver" is to have calls to the library run on their own thread. On practical terms this would involve a thread pool of n threads, where each call to the library is then executed on the first "free" thread available. If no thread is available then either the thread pool is enlarged (new thread) or the call fails (your choice...

Threads could be created using, perhaps Bourdeaux threads. I am new to the CL ecosystem (and thus, totally new to threads/processes/concurrency on CL) so please advice if my idea has some pitfalls or isn't as easy as the "rewrite the lib so it uses async I/O calls" idea.

Of course, making this work would mean also using locks/mutex/semaphores/etc to make the threads run "safely". Also, i am not sure this is simpler than rewriting the lib to use async I/O calls; however at least in this "thread pool" your API call can perform as much CPU-intensive stuff as it needs without blocking the main thread at all. The event loop is still there, with its advantage of higher throughput, but HTTP requests that require complex operations can make use of such thread pool for specific library calls, like for example an ORM + database driver, which not only requires I/O but some CPU time for doing the O/R mapping itself, query building, etc.

What do you think?

[defunkydrummer]

Update: @orthecreedence: I see you already have a lib for doing exactly what I was thinking :)

https://github.com/orthecreedence/pretend-event-loop

A package that mimics a non-blocking event loop by using queues and threading. Allows an application to implement non-blocking operations without converting all the libraries to be non-blocking.

Nice !! See, i'm just new to the CL ecosystem...

[orthecreedence]

Glad you found that! It has been a while since I've updated it so it might be dusty, but then again it's not like common lisp changes a whole lot =]. One thing that could be useful would be to integrate pretend-event-loop with the cl-async event loop using the notifiers so you could have a mix of a real event loop with a threaded queuing-based system for background tasks.

[defunkydrummer]

Thanks for your quick reply. To be honest, i'm new to developing in Common Lisp, just a few weeks into CL (but 25+ years using the Blub programming languages ;)), so i am first having a slow survey of the state of the package ecosystem. I'm really impressed by how many libraries are out there! And finding such a specialized but handy thing like pretend-event-loop was surprising to me. I guess that everything that is nice to have, will eventually be made reality for CL.

Thanks for pointing out about your notifier class. "A notifier is an object that lets you trigger a callback in your event loop thread from any other thread in a safe way. " ---> excellent.