search

methodmissing / rbczmq

Ruby extension that wraps the official high level ZeroMQ C API ( http://czmq.zeromq.org/ )
http://github.com/methodmissing/rbczmq
MIT License
125 stars 32 forks source link

readme

= rbczmq - binding for the high level ZeroMQ C API {Build Status}[http://travis-ci.org/methodmissing/rbczmq]

(c) 2011-2014 Lourens Naudé (methodmissing), James Tucker (raggi), Matt Connolly (mattconnolly) with API guidance from the czmq (http://czmq.zeromq.org/) project.

http://github.com/methodmissing/rbczmq

== About ZeroMQ

In a nutshell, ZeroMQ is a hybrid networking library / concurrency framework. I quote the ØMQ Guide (http://zguide.zeromq.org/page:all) :

"ØMQ (ZeroMQ, 0MQ, zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry whole messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fanout, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. ØMQ is from iMatix and is LGPL open source."

== Another ZeroMQ extension ?

This extension bundles both ZeroMQ (libzmq, http://www.zeromq.org/) and CZMQ (libczmq, http://czmq.zeromq.org/) and as such have no third party dependencies other than a Ruby distribution and a C compiler. My goals for this project were :

== Performance

ZeroMQ can have higher throughput than TCP in most cases by using a message batching technique. Please have a look through the Performance section in the ZeroMQ FAQ (http://www.zeromq.org/area:faq#toc2) for further implementation details.

Some notes about these benchmarks :

=== TCP/IP loopback, 100k messages, 100 byte payloads

Lourenss-MacBook-Air:rbczmq lourens$ MSG_COUNT=100000 MSG_SIZE=100 ruby perf/pair.rb
Local pids: 2042
Remote pid: 2043
Sent 100000 messages in 0.3933s ...
[2043] Memory used before: 1128kb
[2043] Memory used after: 3012kb
[2042] Memory used before: 1120kb
====== [2042] transfer stats ======
message encoding: string
message size: 100 [B]
message count: 100000
mean throughput: 227978 [msg/s]
mean throughput: 182.383 [Mb/s]
[2042] Memory used after: 22432kb

=== TCP/IP loopback, 100k messages, 1024 byte payloads

Lourenss-MacBook-Air:rbczmq lourens$ MSG_COUNT=100000 MSG_SIZE=1024 ruby perf/pair.rb
Local pids: 2027
Remote pid: 2028
Sent 100000 messages in 0.641198s ...
[2028] Memory used before: 1120kb
[2028] Memory used after: 12776kb
[2027] Memory used before: 1144kb
====== [2027] transfer stats ======
message encoding: string
message size: 1024 [B]
message count: 100000
mean throughput: 160756 [msg/s]
mean throughput: 1316.919 [Mb/s]
[2027] Memory used after: 189004kb

=== TCP/IP loopback, 100k messages, 2048 byte payloads

Lourenss-MacBook-Air:rbczmq lourens$ MSG_COUNT=100000 MSG_SIZE=2048 ruby perf/pair.rb
Local pids: 2034
Remote pid: 2035
Sent 100000 messages in 0.94703s ...
[2035] Memory used before: 1140kb
[2035] Memory used after: 7212kb
[2034] Memory used before: 1128kb
====== [2034] transfer stats ======
message encoding: string
message size: 2048 [B]
message count: 100000
mean throughput: 123506 [msg/s]
mean throughput: 2023.528 [Mb/s]
[2034] Memory used after: 277712kb

Have a play around with the performance runner and other socket pairs as well - https://github.com/methodmissing/rbczmq/tree/master/perf

== Usage

As a first step I'd highly recommend you read (and reread) through the zguide (http://zguide.zeromq.org/page:all) as understanding the supported messaging patterns and topologies is fundamental to getting the most from this binding. Here's a few basic examples. Please refer to documentation (http://methodmissing.github.com/rbczmq/) and test cases (https://github.com/methodmissing/rbczmq/tree/master/test) for detailed usage information.

=== Basic send / receive, in process transport

ctx = ZMQ::Context.new
rep = ctx.socket(:PAIR)
port = rep.bind("inproc://send.receive")
req = ctx.socket(:PAIR)
req.connect("inproc://send.receive")
req.send("ping") # true
rep.recv # "ping"

ctx.destroy

=== Fair-queued work distribution to a set of worker threads

ctx = ZMQ::Context.new
push = ctx.bind(:PUSH, "inproc://push-pull-distribution.test")
threads = []
5.times do
  threads << Thread.new do
    pull = ctx.connect(:PULL, "inproc://push-pull-distribution.test")
    msg = pull.recv
    pull.close
    msg
  end
end

sleep 0.5 # avoid "slow joiner" syndrome
messages = %w(a b c d e f)
messages.each do |m|
  push.send m
end

threads.each{|t| t.join }
threads.all?{|t| messages.include?(t.value) } # true

ctx.destroy

=== Async request / reply routing

ctx = ZMQ::Context.new
router = ctx.bind(:ROUTER, "inproc://routing-flow.test")
dealer = ctx.socket(:DEALER)
dealer.identity = "xyz"
dealer.connect("inproc://routing-flow.test")

router.sendm("xyz")
router.send("request")
dealer.recv # "request"

dealer.send("reply")
router.recv # "xyz"
router.recv # "reply"

ctx.destroy

=== Send / receive frames

ctx = ZMQ::Context.new
rep = ctx.socket(:PAIR)
rep.bind("inproc://frames.test")
req = ctx.socket(:PAIR)
req.connect("inproc://frames.test")
ping = ZMQ::Frame("ping")
req.send_frame(ping) # true
rep.recv_frame # ZMQ::Frame("ping")
rep.send_frame(ZMQ::Frame("pong")) # true
req.recv_frame # ZMQ::Frame("pong")
rep.send_frame(ZMQ::Frame("pong")) # true
req.recv_frame_nonblock # nil
sleep 0.3
req.recv_frame_nonblock # ZMQ::Frame("pong")

ctx.destroy

=== Send / receive messages

ctx = ZMQ::Context.new
rep = ctx.socket(:PAIR)
rep.bind("inproc://messages.test")
req = ctx.socket(:PAIR)
req.connect("inproc://messages.test")

msg = ZMQ::Message.new
msg.push ZMQ::Frame("header")
msg.push ZMQ::Frame("body")

req.send_message(msg) # nil

recvd_msg = rep.recv_message
recvd_msg.class # ZMQ::Message
recvd_msg.pop # ZMQ::Frame("header")
recvd_msg.pop # ZMQ::Frame("body")

ctx.destroy

== Resources

== Requirements

== Installation

Rubygems installation

gem install rbczmq

Building from source

git clone https://github.com/methodmissing/rbczmq.git
cd rbczmq
git submodule init
git submodule update
rake

Running all tests

rake test

Running a single test file:

ruby -Ilib test/test_context.rb

OS X notes:

If you are installing the package on a new Mac ensure you have libtool and autoconf installed. You can get those with brew packaging system:

brew install libtool autoconf automake

== LICENSE

The rbczmq gem is licensed under the MIT license. See the file "MIT-LICENSE" for details. This gem uses the "zmq" and "czmq" libraries, both of which are licensed under the LGPL license.

== TODO

== Contact, feedback and bugs

This project is still work in progress and I'm looking for guidance on API design, use cases and any outlier experiences. Please log bugs and suggestions at https://github.com/methodmissing/rbczmq/issues