Consider changing the approach used for publish handlers

[vishnureddy17]

_Continuing the discussion from https://github.com/vishnureddy17/paho.golang/pull/19#discussion_r1337408392_

I feel it would be better to get rid of the user-defined routers in paho and instead just provide the following methods directly on the paho client: func (c *Client) SetHandlerAllPublishes(func(*Publish)) func (c *Client) UnsetHandlerAllPublishes() func (c *Client) SetTopicHandler(topic string, func(*Publish)) func (c *Client) UnsetTopicHandler(topic string)

This would still allow the same use cases as before.

Currently, the behavior for registering handlers differs depending on the router implementation being used, and this change would make things more consistent.

cc @ashtonian @MattBrittan

[MattBrittan]

I don't really have a big issue with the existing setup. It's relatively simple to use and provides a lot of flexibility (I would be tempted to move the router into a sub-package to fully separate it).

Currently, the behavior for registering handlers differs depending on the router implementation

The process for creating a handler instance does vary (i.e. NewStandardRouter() vs NewSingleHandlerRouter(h MessageHandler)) but is this really an issue? (if you want a single handler router, then setting the handler at the time of creation makes sense as there will be one, and only one, handler).

One issue with the proposed interface is reconnections. Do we provide a way to retrieve the existing router config and pass it into a new instance of the client? (ideally autopaho should be able to reconnect without any need to touch the router config).

I also feel that adding this into paho locks in some functionality that may make sense in many cases, but will sometimes surprise users e.g.:

• Does adding a handler for a topic mean that the new handler will be called as well as, or instead of, existing handlers.
• UnsetTopicHandler may unset multiple handlers (which could be confusing if data from a topic is used in separate packages).

In some ways I think going the other way might be preferable. By this I mean simplifying paho such that Router is replaced with a callback (i.e. func(*packets.Publish) or perhaps func(*packets.Publish) error). My rationale for this is that the paho library only calls a single function in the Router (Route(*packets.Publish)) so it's really not necessary for us to dictate the router structure (and the current structure requires SingleHandlerRouter to implement RegisterHandler and just ignore the topic!). Having said that, the advantage of the current setup is that if you have aClientyou know you can callclient.Router.RegisterHandler` (and this simplifies things for new users).

[vishnureddy17]

The current setup would make things more difficult with dependency injection.

If I have package that accepts an paho.Client, it will not know how to subscribe and listen for publishes because that would depend on how the user set up the Router. That being said, I understand this is an uncommon scenario.

One issue with the proposed interface is reconnections. Do we provide a way to retrieve the existing router config and pass it into a new instance of the client? (ideally autopaho should be able to reconnect without any need to touch the router config).

Perhaps the direction I propose is more appropriate in autopaho?

[MattBrittan]

Perhaps the direction I propose is more appropriate in autopaho?

It's probably simpler to add there (autopaho was intended to be both an example of how to use paho and also a default starting point for those with relatively basic requirements). Another option would be to flip this and have a Router with a link to the Publish/Subscribe functions in autopaho (or paho) and pass that Router around instead? (with autopaho you should be able to call NewConnection and, once that returns, Publish/Subscribe without worrying about the underlying connection).

[MattBrittan]

Note: Issue #71 also needs to be considered (can handle this requirement with the current setup but not the proposed one).

[vishnureddy17]

Another option it to allow multiple func(*packets.Publish) callbacks to be set. All the callbacks would be called when the client receives a publish from the server.

[MattBrittan]

Just wanted to summarise my understanding of the issue here:

If you have a paho.Client then you have access to its client.Router but, because the router is customisable, you cannot be certain that calling router.RegisterHandler(topic, callback) will actually register a callback for the specified topic (e.g. SingleHandlerRouter does not do this). This creates an issue for packages that accept a paho.Client (or autopaho.ConnectionManager) because they have no reliable way to register a handler (if the user sets up a SingleHandlerRouter then the code will fail in a confusing manner).

That being the case here are some options (including those above):

• Integrate the router into Client - I'm not keen on this because we cannot predict all possible requirements (e.g. #71).
• Multiple func(*packets.Publish) callbacks (e.g. Router Router becomes Router []Router and messages are processed by each router in turn).
• change Router Router to Router func (pb *packets.Publish). This means that SingleHandlerRouter would no longer need to implement Router which would reduce confusion (we could leave the Router interface in the code with the requirements documented as a model (this is fairly standard in Go; e.g. you can implement io.Writer, and not follow the requirements in the docs "must return a non-nil error...", but doing this means your Writer will break other code).
• Add Chain(Router) to the Router interface requiring routers to allow chained routers to be added.

I'm quite keen on a change here because SingleHandlerRouter does not sit well with me (it does not follow the interface documentation at the top of the file!).

[vishnureddy17]

Yep, we have the same understanding of the issue here

[alsm]

Although the core client doesn't use the full Router interface the rpc extension does call RegisterHandler and that could be useful in other extensions that people might write. And while there is an interface being used there is no way to guarantee that the implementor does what is implied. Maybe it would be simpler to just remove the SingleHandlerRouter code from the library but leave the rest as is?

[MattBrittan]

OK - How about #184; this adds a default handler to the Standard router which would make it easy to remove SingleHandlerRouter (I like the simplicity of SingleHandlerRouter but the fact is that it breaks the "contract" set by the interface). This is also something I implemented last week as a custom router as I needed it anyway! Hopefully this would provide @vishnureddy17 with the confidence that the Router docs will be honoured ( so can be trusted when an instance is injected).

[alsm]

I didn't put it in but I had the same thought when writing my previous comment, so yeah that sounds good to me.

[alsm]

@MattBrittan do you want to update your PR to remove the single handler router at the same time?

[MattBrittan]

@MattBrittan do you want to update your PR to remove the single handler router at the same time?

Can we make that a second pr? (Need to remove all references to it in the examples too).

[vishnureddy17]

How should users be expected to new handlers on a client? Should they be directly accessing the Router field on the paho.Client? How would they be able to add new handlers to an autopaho.ConnectionManager? It looks like there's no way to do that currently except in the original config of the ConnectionManager.

[MattBrittan]

How should users be expected to new handlers on a client?

Currently you just use the router (if you created it yourself) or, for paho only, access it from the client (client. Router).

In terms of modifying this to support use with dependency injection we need to know what functionality you need. I guess we could add a Router() paho.Router call into autopaho.

[vishnureddy17]

I guess we could add a Router() paho.Router call into autopaho.

That would solve my scenario, at least. There just needs to be some way to add handlers after the ConnectionManager is created.

[MattBrittan]

I've thought about this further and do still have some concerns about the router design.

The obvious solution to issue #204 (other than not using overlapping subscriptions!) is to route based on the Subscription Identifier. I thought a good enhancement would be to add a SubscriptionIdentifierRouter to show how this is done; but implementing the Router interface for this is going to require some nasty code (would need to pass the identifier as a string to RegisterHandler).

As a result I'm back to thinking that a better option would something along the lines of:

• replace ClientConfig.Router with something like OnPublishReceived(args) bool, error (boolean to be true if a handler was found/called). args is to be defined but would include AlreadyHandled bool which would be true if a previous handler returned true.
• Store this in paho/autopaho as onPublishReceived []func(args) bool,error and add a AddOnPublishReceived/RemoveOnPublishReceived functions (to meet the requirements re dependency injection).

I believe this could be done in a non-breaking way (if Router is set then it's added to onPublishReceived).

This approach would increase complexity but allow more flexibility when it comes to non-standard routing (routing using the Subscription Identifier seems like a reasonable requirement).

One further concern; we currently have RegisterHandler(string, MessageHandler) and UnregisterHandler(string). In StandardRouter calling RegisterHandler with the same topic, but differing handlers, results in multiple handlers being registered (a reasonable thing to want). Unfortunately there is no way to remove just one of these handlers (calling UnregisterHandler always removes all handlers for the topic); this may be an issue for requirements such as @vishnureddy17 mentions above.

Thoughts?

[vishnureddy17]

I assume args includes the actual publish packet as well? If so, I think that's a good approach. It seems flexible enough for almost all use cases at that point. It would definitely meet the requirements my scenario.

Based on how you described it, it seems like the functions stored in the slice would be called one at a time. So if the user provides a function that blocks forever, the other handlers will not be called. I think this is fine, but it is a potential pitfall a user can encounter.

[MattBrittan]

I've mocked up a way that this could be done that will not break all that many programs (it will only break code where the router is set after paho.Client has been created - in a separate issue I'm suggesting the config should be private anyway). Interested in feedback on this approach; I feel that it will simplify usage in some cases (no need to create a Router if all messages go to a single handler) and should be support dependency injection (a library can accept paho.Client and then register to receive all messages received).

Note: if we take this approach a few small changes will be needed in autopaho.

[MattBrittan]

So if the user provides a function that blocks forever, the other handlers will not be called.

This is the case currently; it's also a common issue in the V3 client (where I have recommend calling .SetOrderMatters(false), so that handlers are called in a Goroutine, due to the number of issues raised).

Unfortunately I think this is unavoidable; in order to maintain message ordering (as per the spec) we cannot call the handlers in a goroutine. I think this will need to be very clearly documented (TODO).

[MattBrittan]

Note: I'm undoing the [Router()](https://github.com/ChIoT-Tech/paho.golang/commit/25e8bfd98b7fcbcf38c021aa9beefce56fea63ab) change due to the move to callbacks rather than routers. Will close this issue with my next PR as the Router is effectively removed (it's still there for backward compatibility but should be removed before v1).