poller:add() doesn't honor callback

[dvv]

Hi!

Namely, the subj. So far, I treat polling as the only possibility to go async, say for luvit, which has the only thread and the only event loop. When do you plan to implement it? Or how could I else make lua-zmq not block the thread? I didn't find ZMQ.NOBLOCK exposed (==1, I believe, though).

TIA, --Vladimir

[Neopallium]

zmq.NOBLOCK is exported to Lua. What do you mean by poller:add() not honoring the callback?

I have commited a fix that corrects the exporting of some C functions to the FFI bindings for the poller interface, but this bug shouldn't have caused a problem with how you use lua-zmq in luvit.

Also I added a version of the perf/local_thr.lua script that uses the zmq.poller interface.

I just ran some tests using perf/local_thr_poll.lua and it shows that zmq.poller is working correctly.

[dvv]

I tried :add(s, events, fn) yesterday and fn had never been called. Lemme retry after the fixes

[dvv]

Please, consider https://github.com/dvv/luvit-zmq/blob/master/tests/poll-new.lua . Am I doing it wrong?

I get:

/home/dvv/LUVIT/luvit/lib/luvit/luvit.lua:183: /home/dvv/LUVIT/zmq/tests/poll-new.lua:38: 'struct ZMQ_Poller' has no member named 'start'

---

similar to https://github.com/Neopallium/lua-zmq/blob/master/perf/local_thr_poll.lua#L34 -- i get

require('zmq').poller() [string "REPL"]:1: attempt to call field 'poller' (a nil value)

---

and also

require('zmq').NOBLOCK nil require('zmq').SNDMORE 2

---

The only way I can use poller from luvit is so far ugly -- https://github.com/dvv/luvit-zmq/blob/master/tests/poll.lua

[Neopallium]

Your code shouldn't be using the low-level zmq.ZMQ_Poller object directly. The code for zmq.poller is this file: https://github.com/Neopallium/lua-zmq/blob/master/src/poller.lua

I forgot that you are using zmq 3.1 and that ZMQ_NOBLOCK was renamed to ZMQ_DONTWAIT. Now either zmq.NOBLOCK or zmq.DONTWAIT can be used from lua-zmq for all versions of libzmq.

[dvv]

Oh, I missed poller.lua. It effectively does hide the complexity, yes. Thanks.

However, ZMQ.NOBLOCK is still nil -- logic at https://github.com/Neopallium/lua-zmq/blob/master/src/pre_generated-zmq.nobj.c#L6488 should account for ZMQ_DONTWAIT definition also (to involve 3.*)

[Neopallium]

ZMQ_NOBLOCK is defined earlier in that file to equal ZMQ_DONTWAIT if ZMQ_NOBLOCK is missing: https://github.com/Neopallium/lua-zmq/blob/master/src/pre_generated-zmq.nobj.c#L240

I have moved the #ifndef logic to be outside the #elif VERSION_3_0 encase that was causing a problem.

If that constant is still missing then please check if ZMQ_DONTWAIT is defined in /usr/include/zmq.h

Also what version of libzmq are you using?

[dvv]

NOBLOCK issue solved. Thanks!

I do https://github.com/dvv/luvit-zmq/blob/master/Makefile#L27 -- so I have 3.1.1

The only question now is how to use polling inside luvit, where we do have one thread and one event loop (i suspect one state also)... Have you any kludge whether it's possible to amploy libuv networking primitives for zmq IO?

[Neopallium]

Doesn't look like it. I looked over the libuv interface and they don't provide a way to listen for a read/write event on a fd, like how libev works. I think it is that way because of how IOCP works on windows, where you have to start a read/write request and then be notified when it has been completed.

I just looked over how the zmq bindings [1] for nodejs works and it seems that they use a low-level IOWatcher interface that is only available on posix systems (i.e. now available on windows). IOWatcher provide direct access to libev's ev_io watcher interface.

1. https://github.com/JustinTulloss/zeromq.node/blob/master/lib/index.js
2. https://github.com/joyent/node/blob/master/src/node_io_watcher.cc

[dvv]

I see. So so far polling is the only option, I believe. I needa learn how to not block the main thread with polling... Here's what I have by now -- https://github.com/dvv/luvit/commit/3f2fb1236bf1ae60cc20ae1d7d4fd22027614d3a#L3R1 -- i try to use uv_queue_work() to put blocking calls aside the main loop. I believe I should try to wrap critical raw zmq functions into such calls https://github.com/dvv/luvit/commit/3f2fb1236bf1ae60cc20ae1d7d4fd22027614d3a#L3R20 to make them behaving async... Or, how difficult it is to do with your native objects, if ever possible?

[Neopallium]

uv_queue_work() uses a pool of threads to do the blocking work. zmq socket's can't be accessed from multiple threads. I don't think the work queue is a good idea.

[dvv]

the point is to not share anything with worker thread. i just want to unblock main thread doing zmq_poll(). if i understand it correctly, uv_queue_work(loop, req, work, after_work) executes work(req) in another "context", and after_work(req) in calling context, so real access to the socket (recv/send) is done only in the only main thread.

otoh, libuv is a layer atop libev, and it might be fruitful to consider to unblock zmq_* calls at least for *nix.

[Neopallium]

zmq_poll() has to access the zmq sockets that it polls on. The zmq sockets would be shared between the worker thread and the main luvit thread which is not thread safe.

As far as I can see the zmq bindings for nodejs only work on *nix systems.