search

agourlay / omnibus

An HTTP-friendly persistent message bus.
Apache License 2.0
70 stars 4 forks source link

If you publish a message at the "/animals" level, all subtopics will receive it as well. #1

Closed ppurang closed 10 years ago

ppurang commented 10 years ago

Interesting design decision .. I would have said that if one listens for /topics/animals one gets all animal related messages (incl. furry) but if one listens on /topics/animals/furry then one just gets those and not messages from /topics/animals

Thoughts?

agourlay commented 10 years ago

I wanted messages to cascade down the topic tree because it allows easy broadcast on subtopics.

When a message is received it contains the topic on which it was initially pushed in the field "event". This way, the client can still filter those "broadcast" messages.

So subscribing to "/topics/animals/furry/cats" you could receive

id: 11345 event: /animals/furry data: A message for all furry animals! timestamp: 1388250283

and decide dynamically according your use case to do something with it or not.

You are not the first one to ask about this design decision, so I guess it is not really intuitive for a pubsub engine.

ppurang commented 10 years ago

But there is no way of getting to all animals messages right?

SO to be able to listen to all furry messages, I need to discover any newly added topics

/topics/animals/furry/cats /topics/animals/furry/dogs /topics/animals/furry/wombats etc.

to be able to listen to all furry messages. Right?

So now we need a way to discover new topics (recently added). Perhaps there is already some way of doing that. A message up the tree would not require that.

And it seems like other devs have the same model of thinking about it.

agourlay commented 10 years ago

Actually you can listen to "/topic/animals/furry" and you will receive every message pushed to sub topics even if those are created after your subscription. This is done by propagating updates down and up the topic tree.

When a message is pushed to a specific topic it will be visible by all parents and by all children.

But I understand your point, so I will think about something to make the propagation strategy configurable.

agourlay commented 10 years ago

I have two ideas possible to bring more flexibility. Tell me which one you makes more sense to you.

1) The propagation strategy is defined at the producer level :

2) The propagation strategy is defined at the consumer level :

WDYT?

oheidemann commented 10 years ago

You're building an message server whose unique feature is "Topic hierarchies and subscriptions" so you shouldn't aim to rebuild yet another message server with the same semantics. Plus i like the idea of downwards broadcasting.

I like the second approach most, but maybe you could give the subscriber even more freedom. Let him decide from how far upwards the hierarchy he likes to receive messages.

agourlay commented 10 years ago

I really like your idea @oheidemann .

Giving more power to the subscriber is indeed an interesting way of building it, especially if we can come up with a generic way of describing the "how far upwards the hierarchy".

Something like

GET /topics/animals/furry/cat?scope=1

WDYT?

oheidemann commented 10 years ago

probably, i would name it "depth" and make it zero based, but yes basically that's what i was thinking about.

agourlay commented 10 years ago

Ok for zero based :+1:

If you work with trees, depth has the semantic of "going down", here we are "extending" our interest upwards, so I am not entirely convinced.

Of course it is really a detail ;)

oheidemann commented 10 years ago

yeah, although depth has many more meanings, it is somewhat preasigned with going down.

so use level or upscope ( even if it isn't a real word ;-) )

agourlay commented 10 years ago

We could even go further "down" that path. What about an API that could let you define you subscription "scope" in both direction

GET /topics/animals/furry?up=1&down2

Here you specify your interest in :

With default value up=0 and down=infinite

kidi commented 10 years ago

Hi all ;) I join this intersting question to give my understanding. i'd rather opt to 'scope' as we talk with the bidrectional up and down communication;we could think about a restriction like upscope and depth for up and down unilateral communication respectively.

It could offer some interesting power like give for an existing topics/animals/funny/cats a subscription on /animals/funny?upscope=0&depth=1 will give all funny cats, dogs, fish, ... but not the funny/cats/simons by example.

agourlay commented 10 years ago

Great minds think alike ;)

kidi commented 10 years ago

I am not completely sure with the default values; but it could be part of the configuration ;) for me scope is over the up/down configuration.

omnibus { up : -1, // infiinite down : -1 // inifnite }

when a scope=s is defined, it will be equivalent to up=s&down=s.

oheidemann commented 10 years ago

Yes, that's what i was thinking too first, but i couldn't see the use case for limiting the scope downside.

agourlay commented 10 years ago

Sounds good to me. The subscriber API starts to become quite rich, with the 'reactiveMode' we could have things like

GET /topics/animals/furry/+/videos/youtube/cute?up=1&down2&mode=between-id&since=1&from=3

It feels like we are creating some kind of REST SQL :D

ppurang commented 10 years ago

I see a lot of discussion which is great. Here are my 2c

  1. Keeping things simple is a good thing. Aren't we complicating thing? Can we still keeps things scalable? what are the corner cases?
  2. I strongly believe a publisher can't and shouldn't decide on message delivery (except for secure, some eyes only messages). It should only be interested in publishing a message at a given address(es?) (here a path/(paths?) in a tree).
  3. The subscribers should be allowed to target the type of messages/addresses they want to tap. And yet it should be straightforward.
  4. Now if the message bubbles up or down a tree is a design decision. For me (and I don't want to take the word sub-tying in my mouth) furry cat messages are animal messages and not for example e-commerce messages. Given a fictive tree with animals and plants as two nodes up the hierarchy, a zoologist might be interested in all animal messages but not plant ones, and a botanist the other way around. And to keep things simple a zoologist wants to hear about all messages on all topics under animals - present and future. Future additions to the sub-tree here is the key. This makes topic-added-in-the-future discovery redundant.
  5. For simplicities sake I'd go for messages either bubbles up or down and not both. But if you want to allow zoologist like use cases then the messages can only bubble up.

On a lighter note, as NSA I want to hear all messages on all topics in the tree ;)

I hope it all makes sense. Perhaps I am missing something crucial and have gotten the entire concept of tree like addresses/topics all wrong.

agourlay commented 10 years ago

Thank @ppurang for bringing some insights to the thread, you perfectly understand the use case.

1 - totally agree with simplicity 2 - yes, we kind of decided to discard the possibility to offer a rich API on the producer side. 3 - this is the difficult API to design. We are trying to design a ""new"" way of doing pub/sub without getting too far away from the existing solutions. 4 & 5 - I would go for providing a default behavior (without using any extra arguments) being identical to other pub/sub solution semantic concerning bubbling.

Are there any real use case of scoping down?

6 - I already received a backdoor integration request from the NSA ;)

Really happy about the discussion on this thread.

agourlay commented 10 years ago

Following this discussion I have documented explicitly the available subscription models.

Default behavior is the classic pub/sub behavior.

It is also possible to listen to all events occuring higher in the topic path by using a specific http param ?sub=wide when subscribing.

see https://github.com/agourlay/omnibus#subscription-models