Reusing messages in rzmq

[mschubert]

I'm developing an rzmq-based R package for distributing function calls on HPC schedulers (clustermq). For this, I need to send the same message (the function itself and constant arguments; often 100s of MB) to all workers at different times (whenever they start up - so no PUB/SUB sockets).

ZeroMQ allows to reuse messages, i.e. not copying the memory to a new buffer like sendSocket() does.

I'm posting this issue here to see if there is interest in incorporating this functionality in the rzmq package (rather than my package; and to get feedback on my approach). The way I would do this is to:

• Provide a function in rzmq that converts an R object to message_t object and return it via UNPROTECT
• Let R manage the actual object (so the user is responsible not to lose the reference, as with the context object)
• Modify the sendSocket() function to take a message_t object directly, call zmq_msg_copy() on it, and then send it
• Potentially provide a function to zmq_msg_close() the message, but I don't think this is required (as the memory for the message is managed by R and we'd get a double free otherwise)

If you agree with this I will work on a PR (otherwise I'd add low-level code to my package).

[armstrtw]

I think it makes sense as long as we don't complicate the existing codebase too much.

you would use an external pointer for the message_t object?

[mschubert]

I've written a draft: https://github.com/mschubert/rzmq/commit/edcdfd6bee84f431e464631a1a5f4bc1dea922cb

Yes, this is using an external pointer to the message_t object and I've deviated a bit and written a sendMessageObject function. (Still need to check whether the gc will try to collect data_ like this, I'm not an Rcpp expert.)

[mschubert]

And I haven't actually figured out how to get message copy to work yet: https://stackoverflow.com/q/46181044