search

zeromq / libzmtp

Minimal ZMTP implementation in C
Mozilla Public License 2.0
53 stars 26 forks source link

Documentation: What parts of the zmq library does libzmtp support? #40

Open vitiral opened 9 years ago

vitiral commented 9 years ago

This library looks amazing and exactly what I am looking for?

In general, what features are supported and what does the library require to operate?

Thanks a ton!

vitiral commented 9 years ago

From looking at the source code, it seems that it supports the ZMQ_DEALER socket and something called CHANNEL, which I don't see in the zmq socket documentation

How difficult would it be to implement PUB/SUB or XPUB/XSUB?

vitiral commented 9 years ago

Delving deeper into the zmq documentation, it looks like one could build majordomo on top of this library, but pub/sub is completely different. Is that about right?

hintjens commented 9 years ago

The libzmtp is a minimal library meant for clients only, at this stage. That means no multithreading, no asynch I/O, and support for the DEALER socket which is the main client socket. (We'd eventually replace with support for the CLIENT socket when/if that becomes mainstream.)

It should be relatively easy to add SUB socket support. PULL and PUSH are equivalent to DEALER (with half the API disabled each time).

On Fri, Apr 3, 2015 at 5:51 PM, Garrett Berg notifications@github.com wrote:

Delving deeper into the zmq documentation, it looks like one could build majordomo on top of this library, but pub/sub is completely different. Is that about right?

— Reply to this email directly or view it on GitHub https://github.com/zeromq/libzmtp/issues/40#issuecomment-89334935.

vitiral commented 9 years ago

What about a memory manager? Do you use malloced data that you later free? Does the system have to have ways to deal with defragmentation?

hintjens commented 9 years ago

Yes, it uses the heap. You can see this quite simply in the code.

On Fri, Apr 3, 2015 at 9:47 PM, Garrett Berg notifications@github.com wrote:

What about a memory manager? Do you use malloced data that you later free? Does the system have to have ways to deal with defragmentation?

— Reply to this email directly or view it on GitHub https://github.com/zeromq/libzmtp/issues/40#issuecomment-89405244.

vitiral commented 9 years ago

The reason I ask is because I have written a memory manager for microcontrollers, tinymem. It is currently in Alpha state (usable but needs more tests and features), but it would allow microcontrollers to use you library.

I have an overall vision:

I would like to make a branch of libzmtp to use tinymem (as soon as tinymem is in Beta). The branch would be written in such a way that a developer could disable tinymem if they were on a system that supported malloc/free.

I would also like to extend libzmpt to support pub/sub.

Would these things interest you? I will write the code and do a pull request to use tinymem.

hintjens commented 9 years ago

Sounds fun, definitely worth exploring.

On Fri, Apr 3, 2015 at 10:26 PM, Garrett Berg notifications@github.com wrote:

The reason I ask is because I have written a memory manager for microcontrollers, tinymem https://github.com/cloudformdesign/tinymem. It is currently in Alpha state (usable but needs more tests and features), but it would allow microcontrollers to use you library.

I have an overall vision:

  • Microcontrollers that can use pub/sub to provide easy interfacing with larger systems
  • ZMQ implemented on a generic socket structure so that any communication protocol can be used:
    • in particular I want to communicate over TCP/IP, uIP, Zigbee and CAN Bus.

I would like to make a branch of libzmtp as soon as tinymem is in Beta that uses tinymem. The branch would be written in such a way that a developer could disable tinymem if they were on a system that supported malloc/free.

— Reply to this email directly or view it on GitHub https://github.com/zeromq/libzmtp/issues/40#issuecomment-89410899.

vitiral commented 9 years ago

I've looked over how zmq actually does it's filtering, and I'm worried that it is not a protocol that can be realistically ported to microcontrollers.

I've wondered for a while on how they do their matching, and it looks like they use an inverted bitmap to search through a set of strings and determine matches.

I don't think this is an embeddable algorithm. I've actually begun work on a protocol with a much simpler matching scheme that I call TMQ. This uses tokens instead of strings for matching, which means I can use just hash table lookups for everything.

Thoughts?

hintjens commented 9 years ago

That whitepaper describes the matching we used in OpenAMQ, over a decade ago. It's very powerful, and absolutely not for embedded systems.

ZeroMQ uses a very different matching model, see http://rfc.zeromq.org/spec:29. Matching is done by the publisher.

So while TMQ looks fine, it's based on a rather wrong assumption of what ZeroMQ does.

On Sun, Apr 26, 2015 at 9:31 PM, Garrett Berg notifications@github.com wrote:

I've looked over how zmq actually does it's filtering, and I'm worried that it is not a protocol that can be realistically ported to microcontrollers.

I've wondered for a while on how they do their matching, and it looks like they use an inverted bitmap http://zeromq.org/whitepapers:message-matching to search through a set of strings and determine matches.

I don't think this is an embeddable algorithm. I've actually begun work on a protocol with a much simpler matching scheme that I call TMQ https://github.com/cloudformdesign/tmq. This uses tokens instead of strings for matching, which means I can use just hash table lookups for everything.

Thoughts?

— Reply to this email directly or view it on GitHub https://github.com/zeromq/libzmtp/issues/40#issuecomment-96423118.

vitiral commented 9 years ago

That's good to hear. So how much memory do you think a publisher would take under various conditions? Do you really think it is doable on a microcontroller?

I've just had a pretty big breakthrough on my memory manager that will require a bit of an overhaul. I'll let you know when I'm closer :)

hintjens commented 9 years ago

A simple publisher needs no significant memory at all. An embedded device just needs to broadcast its messages; subscribers can filter them if they want.

On Mon, Apr 27, 2015 at 12:14 AM, Garrett Berg notifications@github.com wrote:

That's good to hear. So how much memory do you think a publisher would take under various conditions? Do you really think it is doable on a microcontroller?

I've just had a pretty big breakthrough on my memory manager that will require a bit of an overhaul. I'll let you know when I'm closer :)

— Reply to this email directly or view it on GitHub https://github.com/zeromq/libzmtp/issues/40#issuecomment-96439879.