Closed jd closed 11 years ago
You are entering down the long and difficult road of converting a synchronous driver to be asynchronous. Welcome to my world the past few weeks =].
Unfortunately, libevent provides an abstraction function for making a socket _non_blocking, but not one to make it block (I think because they block by default). In other words, there's not an easy, portable way for me to add it into the library.
Don't think I don't feel your pain, though. I worked on a solution via coroutines would be the best of both worlds, but the implementation maintainers I talked to basically said that coroutines are going to be a lot of trouble to program, and only a handful of lispers want them, so "no, unless you want to do it yourself."
You have a few options:
evutil_make_socket_nonblocking
function (see below) and use cffi
to mimic it (you can directly call C functions via lisp). This will let you do an "easier" conversion of cl-postgres
, but a solution that only allows one call to the db per thread from your app at once (kind of defeats the purpose, and is another reason I'm hesitant to allow making sockets blocking).// fd can be accessed via le:bufferevent-getfd
int evutil_make_socket_nonblocking(evutil_socket_t fd)
{
#ifdef WIN32
{
u_long nonblocking = 1;
if (ioctlsocket(fd, FIONBIO, &nonblocking) == SOCKET_ERROR) {
event_sock_warn(fd, "fcntl(%d, F_GETFL)", (int)fd);
return -1;
}
}
#else
{
int flags;
if ((flags = fcntl(fd, F_GETFL, NULL)) < 0) {
event_warn("fcntl(%d, F_GETFL)", fd);
return -1;
}
if (fcntl(fd, F_SETFL, flags | O_NONBLOCK) == -1) {
event_warn("fcntl(%d, F_SETFL)", fd);
return -1;
}
}
#endif
return 0;
}
Sorry if this is bad news, but being the early adopter of a library that forces switching to continuation-passing style has its bad parts =[. My recommendation is to not make your socket blocking since in the end, I think you'll hate this option more than you realize, but to either wait for a cl-postgres
conversion or have a try at doing it yourself.
Thanks @orthecreedence! Indeed, it's a tough situation, but I'm really getting used to teething troubles nowadays. :-)
I'm using cl-postgres
into a second thread now to avoid blocking, but everything's not thread safe, so I'll have to add some simple locking mechanism. Something I would have love to avoid by using cl-async
in this precise case, but I'll wait for the conversion to happen!
I don't think I could help that much about converting, I'm still discovering CL every day, but count on me to try and review any implementation you'll have! :-)
Much appreciated, Julien. I'll do my best to let the world know when the conversion is completed, both via the cl-async homepage blog and reddit's /r/lisp.
In the meantime, good luck with the threaded approach, that seems like a fine pursuit. One thing I forgot to mention which may make your life a bit easier is pretend-event-loop, a library I wrote a while back which mimics an event loop using a thread pool. It has one active thread (the main thread that code runs in) and a configurable number of passive threads, used to run blocking operations. The idea is that whenever you have to do I/O, you schedule it in a passive thread and bind the result(s) to a variable available in the active thread such that the passive threads can be many more than your CPU cores since they are mostly doing blocking operations.
Right now it's kind of weak on error handling, as most of it has to happen in the passive thread the blocking operation runs on which is annoying since the passive operations should be as lean as absolutely possible to avoid context switches. It also adds a bit of cruft to the interface it uses. You might still find some use for it though, or even some patches if you can think of a better interface, which I'd gladly accept. If you'd like to give it a try, feel free to ask me any questions you run into while using it!
I'm trying to integrate
cl-postgres
andcl-async
together. I've opened a bug and I am working on an implementatino, see https://github.com/marijnh/Postmodern/issues/28My current problem, is that all of
cl-postgres
is synchronous. That means I need to be able to put the socket into synchronous mode so it can write some requests, read its expected replies, and then give back the hand to the loop.Currently, I provide a callback function to
cl-postgres
for it to call when it wants a stream to communicate with PostgreSQL. The problem is that it calls it and gets a stream fromtcp-connect
. Then it tries to authenticate right away, and it gets "end of file" really soon. I think this is due to the fact there's nothing to read at this precise moment, but I'd prefer theread
function to block, in this case.