Subscribing from the client

[andrewgy8]

AFAIK, the current model of this project is based on a message POST to a client endpoint. Basically, a client opens an api that allows the broker to send messages.

What I would like to see and try to do is subscribe to a topic.

Right now, I can only envision this in one way, but I would love to hear of others.

My idea is to open a websocket in postoffice, in which the subscriber can start listening to. With this, a client can start listening to the websocket and a stream of messages will start to follow. Of course, this leads to issue such as:

• rate limiting
• message ordering
• retry strategy's
• and probably a lot more stuff.

However I think it would be a really good addition to the project since a project with more than a dozen subscriber, would need to manage a dozen or more endpoints. It would be a very passive way to listen to messages/events on the client side.

[jonasae]

What's exactly the benefit of subscribing/opening a websocket instead of exposing an endpoint? I guess it would be more convenient to just subscribe('a-give-topic') than the boilerplate required to expose an endpoint, right?

The cons are how to keep the websocket opened in a resilient way. If that could be addressed, the purposed solution would be a great improvement.

The rest of concerns @andrewgy8 mentioned I think are still there when posting messages to the clients.

[andrewgy8]

Exactly like you describe, the boilerplate could be much less for a traditional model of subscribing to a topic.

how to keep the websocket opened in a resilient way

This is the crux of the problem, and one that we will have to be thought out on the engineering side. Im not sure, if websockets is the right tool, or some other communicaiton protocol that could be a solution from the Phoenix framework.

[lonamiaec]

PO is based on HTTP status codes to mark a message/task as done/delivered. With webhooks, this changes completely and the approach for them should too. Sending tasks via websockets is just sending its payload to a phoenix channel. We can be sure that the payload will be delivered to any client who is listening, but we can not guarantee there will be anyone listening. I would say we need a whole new "mark as done" for websockets.

Once we deliver a task via phoenix's channel, Oban will mark it as completed. But this task won't be removed after being marked, it'll be at least 24 hours in our system.

I was thinking about the following:

• Add a new model to keep track of any message sent through websocket.
• New endpoint to mark those rows from this new model as "delivered". I guess in our case we could attach some relé hook to call it automatically from all the subscriptions.
• Custom plugin for Oban. Check from our new "send but not sure if received" table what messages have been sent from X minutes ago and are not still marked as delivered. For all those, we should update Oban's job to available so it will be delivered again.

The downside here is there is some chance to deliver any message more than once, but the other option is just to lose them so doesn't sound as bad.