Thoughts about the future of dcell and celluloid-zmq

[Asmod4n]

dcell

cons

• depends on another process for registry needs
• no authentication
• no authorization
• connected peers can execute any ruby code
• encryption setup must be done manually
• not compatible with other languages

  pros

• no need to implement serialization or a rpc protocol, its just marshaled objects send to celluloid mailboxes

  celluloid-zmq

celluloid-zmq can stay as it is right now, the zeromq maintainers said they made some mistakes with zeromq, their answer: https://github.com/zeromq/czmq. czmq is a high level zeromq binding written in c which should be the used for other language bindings. Other notable new projects bourn around czmq: https://github.com/zeromq/zproto, https://github.com/zeromq/zyre. zproto is a protocol generator for server and client classes in several languages around czmq based on finite state machines. zyre is a discovery framework.

czmq is rapidly approaching version 3 which adds a lot new features, i have written a wrapper for it: https://github.com/Asmod4n/ruby-ffi-czmq. I believe it should not replace celluloid-zmq.

czmq v3 api summary: https://github.com/zeromq/czmq#api-v3-summary

possbile solutions for the current cons of dcell

• zyre can be used for node and service discovery and for global storage
• authentication can be done with zauth
• a small DSL can be written for Celluloid Actors to white- and blacklist methods based on capabilities
• automatic configuration of encryption can be handled with zcert and zconfig
• compatibility with other languages can be achieved by using msgpack for serialization and wamp as the RPC protocol (only using wamp as a example, due to the usage of zeromq and zyre it can be reduced to 3 rpc calls)

While capnproto looks nice I haven't yet found a c binding which could be used as a starting point for a ffi wrapper which handles serialization and rpc, the "time traveling" part also gets at least partially solved with zeromqs message batching. capnproto also has the same issue as protobuf for me: you need to learn a new type system and duplicate existing code.

I just don't know where to inject all that code. Wrap everything into a czmq zactor written in c or ruby or better have both options in case there is no c compiler installed? Inject most of it into celluloid actors itself? Expand Celluloid Supervisors?

I for one would go for the czmq zactors in c and ruby so dcell can be the get go for other languages to wrap the c zactor to have a portable rpc implementation.

What would change to how it is now to users of dcell would be rather minimal, the only thing that would be missing are datatypes msgpack can't serialize.

[tarcieri]

Removing the number of possible configurations and moving parts sounds great.

I'd also be fine with you rebuilding Celluloid::ZMQ on top of czmq if you wanted to.

It'd be great if Ruby had at least a good cross-platform capnproto library that at least did serialization, but alas it does not, and its static typing is probably not a good fit for Ruby. It has its own capability-centric RPC system which is great, but not a good fit for 0MQ.

Perhaps standardizing on a base primitive set of types would be a good way to go. msgpack is probably ok.

[niamster]

@Asmod4n I've taken a look at other serialization libraries and IMHO they can't provide what native ruby marshalling can - transparent object serialization. If you do serialization manually for each and every object you want to transfer it begins to add to many complexity(and even the size delta is not that big, just a couple of bytes). Dcell is general purpose library, it can't force user to define serialization method for each object to transfer. Well, it can, but it's loosing its attractiveness. I would tend to agree that this might be an exercise to clean up what is marshalled inside dcell but no more than that. I agree that the registries zoo in dcell is annoying. Personally I would probably keep one simple via shared ram to maintain spec tests and 0-configuration. But DCell should provide a clean and stable API to the users that are willing to attach their backends. Probably it can be a separate repository to maintain them. Encryption is a must, I'm looking forward to get it into dcell soon. The other thing I would like to have is creation of unique remote actors(right now those singletons only), pass actors between cells and dynamic node selection(I want an actor and I don't care where it is located). And of course the stats on node queue, node performance, locality, etc.Basically a node scheduler. I had an idea to store actor states inside the registry. The first free node picks first task for execution, runs it, puts result back and notifies the client. Might be too crazy but you may think about a usecase when you run all nodes on single server(each node bound to a CPU), put your registry into shared ram and do smth like map-reduce. I believe that's one of the goals of celluloid but if I understand correctly it can't be achieved with current MRI due to a LOCK. Rubinius is still too slow in my opinion to run on mid-range servers(8 cores, 32 GB RAM). Jruby is awfully slow =/.

[tarcieri]

@niamster transparent serialization is nice but has a huge costs. Both sides must have the same code loaded or things won't work. Marshal's protocol is complex enough to be Turing-complete in most cases, meaning that remote code execution is easy e.g. remote.instance_eval("system('rm -rf /')"), and while these sorts of problems can be worked around with e.g. $SAFE, this is not a good security mechanism and because of that people have suggested it be removed from the language entirely.

A stricter serialization format could prevent this protocol from resulting in remote code execution, which would be a huge security win.

[Asmod4n]

@niamster when you think about it what you need to transfer over the wire to reconstruct the object on the other side it almost every time comes down to the data types msgpack can serialize, the ruby documentation also states one shouldn't use Marshal when it comes from a untrusted source (the internet).

A node scheduler and stats might be implemented with https://github.com/zeromq/zccp

[niamster]

@Asmod4n @tarcieri I agree that Marshaling is completely insecure. I'm too trustful =| As an exercise I've started to convert dcell to use msg_pack. In fact I removed all celluloid extensions and trickery, just left few bits to memorize local mailbox. You can take a look at niamster/dcell@72c66b2dc90eec7a707ac9efd538298ee97b6d4d and niamster/dcell@fc66aa882ee2a8142718bc9bd5680f52d02ba575 This is a bit dirty implementation but my idea should be clear.

[niamster]

The basic idea is that locally I create a Celluloid actor that "proxies" all requests to the remote actor. I just don't know whether on local node the requests should be delivered asynchronously. Right now those are sent in synchronous manner(a synchronous call to Node.send_request). IMHO I don't see an interest in sending them asynchronously as the client can use "async" call on the local actor.

[Asmod4n]

@niamster could you also check if your patches work with https://github.com/iconara/msgpack-jruby ?

[niamster]

@Asmod4n I've tried that but it doesn't work out of box because msgpack-jruby doesn't support to_msgpack for native types(Hash, Array, etc.). There's a pull request to fix that but seems like the repo is dead.

[Asmod4n]

@niamster the author believes it is better to add to_h or to_a instead of to_msgpack(https://github.com/iconara/msgpack-jruby/issues/13#issuecomment-42399495), the repo also wants to get merged into msgpack-ruby(https://github.com/iconara/msgpack-jruby/pull/15#issuecomment-45710450) but yeah it looks like the owner of msgpack-jruby gave up on the merge (https://github.com/msgpack/msgpack-ruby/pull/33#issuecomment-64178900) but i hope he will finish the merge.

[niamster]

To have a distributed RPC, I was thinking to store requests and responses inside the database. Unfortunately with that zoo of supported registry adapters it's hard to maintain and support some required operations(smth like CAS). For example with Cassandra 1.x it's not possible to have smth like SETGET in Redis(I've managed to do that with mongodb and zk though). I propose to vote for a single official registry that is supported by DCell. The users may use their backend at their own risk. If needed we can use few distributed databases to efficiently implement registry. IMHO, Redis is good enough to efficiently implement DCell registry.

[tarcieri]

I am +1 on a single official registry type, however -1 on Redis: it is, at best, a single point of failure.

I would suggest integrating with the 0MQ primitives for this stuff, since it's already a dependency.

[Asmod4n]

0MQ is basically a big linked queue which makes sure a message arrives at its destination under all circumstances(unless you are using pub/sub), for what would you need redis for rpc? as a queue?

[niamster]

@Asmod4n Ideally it would be the best to drop a binding between a request and a target node. I would like to put a request somewhere and in a worse case send a multicast message to the nodes that know how to handle it(or just notify the master node that does scheduling).

I haven't worked closely with 0MQ, maybe it already provides such mechanisms.

[Asmod4n]

When a receiver goes down a sender will send the message until the receiver gets up again and acknowledges the arrival.

In a typical 0MQ network there is no scheduler or broker. Unless you write one yourself.

[niamster]

That still doesn't solve a problem when server node may be down or busy for a long time. My idea is that a first free server node serves the request(if it knows how to serve it). That's my view of D-RPC.

[Asmod4n]

You can set timeouts for sending and recieving and then select another node which advertises it can fullfill your rpc.

https://github.com/zeromq/zyre can notify you also about a node going offline or which rpc calls etc it can handle.

[tarcieri]

@niamster Celluloid comes from a very different tradition of systems than one which operates that way.

Specifically Celluloid is a stateful system that models the world more along the lines of an object capability system, for example E/CapTP or Cap'n Proto.

In E terminology, each node in a DCell network resembles an E "vat". Actors are localized, persistent (if supervised with a registered name), and stateful (unless they crash).

For the kind of (round-robin?) scenarios you're describing, I've thought about some higher level abstraction (a Service?). I think there are 0MQ primitives that handle this sort of message routing.

There's a lot of questions about how this would work though:

• How do these actors manage state in a distributed system?
• Is there a leader? Should there be a leader?
• What happens when an actor crashes?

There's plenty of work in objcap systems to answer these sorts of questions. One set of principles for answering them is COVR (Composable Output-Valid Resilient Messaging)

http://www.hpl.hp.com/techreports/2014/HPL-2014-14.pdf

I think if you're trying to redesign the messaging semantics, before picking a backend you should describe the new messaging model you'd like to achieve.

[Asmod4n]

I would also recommend to read the 0MQ guide: http://zguide.zeromq.org/page:all

[gregory]

@niamster i'm also looking into a d-RPC mechanism. To solve that, we need some kind of centralized mailbox/queue where nodes could listen for new message(RPC call) and a way for them to publish/stream response to the listener(s)

I also thought about redis as you mentioned, and as of the SPF you mentioned @tarcieri , redis-sentinel could help...

I'm not familiar with zmq, but sounds like it does implement a RPC mechanism, however sounds like there could only be one broker... (but since i've never used that, i'm may talk shit)

[tarcieri]

I also thought about redis as you mentioned, and as of the SPF you mentioned @tarcieri

Redis is just the wrong tool for the job here. In fact Redis's author has been working on a separate tool "disque" for this purpose.

The basic prerequsites you need for a distributed RPC service like this are:

• A distributed directory (e.g. Zookeeper)
• Backends registering themselves in that directory
• A healthchecking system

I know it seems rather attractive to punt that all onto Redis for simplicity's sake, but I would strongly advise against doing that.

Centralizing everything in a single point of failure is pretty orthogonal to the original design goals of this project

redis-sentinel could help...

redis-sentinel is broken-by-design:

https://aphyr.com/posts/283-call-me-maybe-redis https://aphyr.com/posts/307-call-me-maybe-redis-redux

If you really do want to utilize an external message broker, I'd suggest looking into actual message queues like Kafka, RabbitMQ, or ActiveMQ, and perhaps building separate adapters so people can pick-and-choose which broker they want to use instead of coupling to one in particular

[Asmod4n]

@tarcieri https://github.com/zeromq/zyre actually does just that, it is a distributed peer management system.

It can form a network of peers via udp beaconing, tcp peer-to-peer or (e)pgm multicast (i believe). Every zyre node knows who is connected to that network, it does so via "ENTER" messages and heartbeats which can trigger a "EXIT" message, . Each node which connects to that network can set up a list of headers, for e.g. which services it runs, thats effectively a distributed key/value store, but those headers only get send during discovery, e.G. when a node enters a network (at least the documentation says so). Those nodes can then form a group where they exchange messages related to that group, thats called shouting, those are "JOIN" and "LEAVE" messages. What also exists is whispering; shouting and whispering could be used to distribute work, e.G. a parent shouts a work message to a group and says "work done" for example when the first node of that group whispers back that it completed the task.

There is no signle point of failure in a zyre network, every node can act as "broker" and/or "worker".

https://github.com/zeromq/malamute is a "Enterprise Messaging Broker", whatever that means.

[Asmod4n]

Oh, actually https://github.com/zeromq/malamute/blob/master/MALAMUTE.md looks kinda nice, the drawback is: its a broker model, when the broker goes down thats it. There are also no ruby bindings yet.

[Asmod4n]

Apparently you can cluster malamute brokers, which is also cool :)

[MadBomber]

On Wednesday, April 15, 2015 11:21:06 PM Tony Arcieri wrote:

The basic prerequsites you need for a distributed service like this are:

• A distributed directory
• Backends registering themselves in that directory
• A healthchecking system

I have been experimenting with rethinkdb http://www.rethinkdb.com/

The latest v2.0.0 support a large number of distributed real-time problem areas.

[niamster]

@MadBomber not sure rethinkdb is a good choice here as it permits data loss. It's good for collecting lots of data for analysis where you can tolerate data loss.

[dfockler]

I think implementing something like the COVR rules that @tarcieri posted would be a better way to handle messaging without resorting a single point of failure like a broker. Although this may not exist yet in ruby, which is an issue.

[tarcieri]

COVR is just a set of guidelines about how to handle message processing and checkpointing. It could be implemented as a library on top of DCell.

[niamster]

I didn't know anything about COVR before @tarcieri pointed on it but it's similar what I had in mind. Though in the end I believe this should not be a core part of DCell. It might be implemented on top of it(same for Raft, etc.) In fact you may have a KVS and COVR as DCell extensions.

[dfockler]

I'm starting to work on code for a COVR library in Ruby. I'll post it once I get a bit further, but the basic idea is that it is a message pipeline that can implement the COVR rules with a persistent storage backend.

[Asmod4n]

https://github.com/antirez/disque looks cool, there is also a ruby client https://github.com/soveran/disque-rb

[tarcieri]

I'm not sure what problem disque solves that in-process brokers don't already solve better. It's also "alpha quality", and Salvatore doesn't exactly have the best reputation when it comes to building distributed systems.

[niamster]

@Asmod4n looks like DCell now meets all your initial requirements in the cons section ;)

[gregory]

This is really good work @niamster !! could you update the changelog and the readme? :)

[Asmod4n]

@niamster looks great! :)

@tarcieri should this stay open?

[niamster]

@gregory I will update the changelog as soon as release version is ready. I'm waiting for the stable 0.16.1 release of celluloid.

[tarcieri]

@Asmod4n if you're happy with the changes, close it out

[digitalextremist]

Congratz @niamster on covering everything @Asmod4n raised. Well done! :+1: