search

mschubert / clustermq

R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
https://mschubert.github.io/clustermq/
Apache License 2.0
146 stars 27 forks source link

QUESTION: Is ZeroMQ I/O atomic? #116

Closed HenrikBengtsson closed 5 years ago

HenrikBengtsson commented 5 years ago

I don't know much about the ZeroMQ protocol, but I'm curious what happens if an interrupt is signalled (e.g. Ctrl-C user interrupt) during one of the rzmq::send.socket(), rzmq::pull.socket() or rzmq::receive.socket() calls? Will that corrupt the ZeroMQ messaging, e.g. parent or worker falls into a corrupt state? Or, does ZeroMQ take care of that and both ends will rollback to their previous state where they're both ready to resend?

PS. I'm curious at what level you have to start worry about atomicity and corrupt parent-worker communication states, which is a common issue in async programming and often seen, for instance, with the clusters that the parallel package run.

mschubert commented 5 years ago

I realized I never answered this, sorry!

ZeroMQ does very little to protect against corrupt states, and operations are for sure not atomic, but you can build architectures that are robust: http://zguide.zeromq.org/php:chapter4#toc0