Allow a subscriber to be unique

lileio / pubsub

A Publish & Subscribe library, with pluggable providers and middleware for Golang

MIT License

137 stars 23 forks source link

Allow a subscriber to be unique #22

Closed arbarlow closed 4 years ago

arbarlow commented 4 years ago

I think this will work and allow you to a one to many subscriber for the use cases like streaming to clients and updating caches etc

jonbretman commented 4 years ago

So if I understand pubsub correctly then this will mean that if there a number of un-processed messages on a topic and an app restarts then messages may be "lost". An example:

App A makes topic foo
App B (two processes) makes a unique subscription to foo - foo-subscription-1 and foo-subscription-2
App A publishes 1000 messages to foo
A bug in App B means it can't process some messages and so a number of messages sit unprocessed in the two subscriptions
The bug is fixed and App B is re-deployed. As the old version of App B shuts down it deletes its subscriptions.
The new version of App B starts up and makes two new subscriptions foo-subscription-3 and foo-subscription-4
Since these new subscriptions were created after the messages were published to the topic they do not receive the messages.

arbarlow commented 4 years ago

Yes, that's the behaviour in this case, hence why you would only want to use this for "real time" like things where you don't need history

jonbretman commented 4 years ago

Yes, that's the behaviour in this case, hence why you would only want to use this for "real time" like things where you don't need history

Ok cool. In this case ideas for naming the option: CreateNewSubscriberPerProcessAndPotentiallyDropMessages :trollface:

jonbretman commented 4 years ago

Yes, that's the behaviour in this case, hence why you would only want to use this for "real time" like things where you don't need history

Ok cool. In this case ideas for naming the option: CreateNewSubscriberPerProcessAndPotentiallyDropMessages :trollface:

But seriously, I think it needs to be a bit clearer what setting this option to true will do. Not sure if that is naming or docs...

arbarlow commented 4 years ago

We can add something yeah, though that is the default behaviour of most publish and subscribe systems! Nats, etc

Open to naming ideas etc

arbarlow commented 4 years ago

Merging for now and will add some strong docs