Generic topic in the tree

[teharris1]

I have been looking at your module for an application that I am refactoring and your module looks very good. One thing that would make my application much easier to write is if I can have a generic subscipriton in the middle of the topic tree. For example if I had a topic tree:

a1:
  b1:
    c1
  b2:
    c1

And if I had the ability to subscribe generically at the b level so that I would get a1-b1-c1 and a1-b2-c1. The obvious answers in the current model are: 1) Move c up a level and b down a level but that will only flip the issue and not really solve it (i.e. I want a generic c selection as well. 2) Stop at the b level and handle c in code. But this means more application logic rather than subscription logic. 3) Subscribe to a1-b1-c1 and a1-b2-c1 which means having more permutations to generate and manage.

My thought is if the topic manager could handle a iterable of hashable objects and each level would then just be a lookup of a hash plus a look up of a generic (like the Any object in pydispatcher.) Thoughts?

[schollii]

@teharris1 IIUC what you are saying, given two topics a1.b1.c1 and a1.b2.c1, you would like to have a listener subscribe to a1.b1, such that it would get messages for both "subtopics"? Pypubsub already does that:

pub.subscribe(your_listener, 'a1.b1')
pub.sendMessage('a1.b1.c1')
pub.sendMessage('a1.b2.c1')

You can do various things like make pypubsub automatically include the actual topic sent, and you can use **kwargs in listener so it will also get the lower level args.

Subscribing to 'a1', same idea. You can also subscribe to all topics via ALL_TOPICS.

[teharris1]

Thank you for your quick reply. Yes, your response is my option 3 above. What I am hoping for is something like this:

pub.subscribe(my_listener, ['a1', pub.ALL_TOPICS, 'c1'])
pub.sendMessage('a1.b1.c1')    # or pub.sendMessage(['a1', 'b1', 'c1'])
pub.sendMessage('a1.b2.c1')    # or pub.sendMessage(['a1', 'b2', 'c1'])

And receive callbacks on both (i.e. only 'a1', any 'b' and only 'c1' would match). Of course all topic values would have to be hashable so they can act as a key.

[schollii]

Subscription to topic 'a' achieves this. If you need to filter just use AUTO_TOPIC to receive topic in listener and check topic name.

[teharris1]

Right but then I need to handle topics b and c, which is what I am trying to avoid. I believe this change does 2 things for your uses:

1. It allows for any hashable object to be a topic (similar to how pydispatcher works)
2. It allows for a more flexible use of subtopic subscriptions meaning the sender does not need to send differently while the subscriber can more easily select which subtopics it cares about.

[schollii]

Sorry not sure I follow why current functionality is insufficient. There is a pypubsub channel on gitter, can we chat there?

[schollii]

@teharris1 So based on the extra info you provided on gitter, I concluded that you are trying to use topics as data, rather than as types of messages that carry data. I proposed 2 solutions:

class Data:
    def __init__(self, a=None, b=None, ... m=None):
        self.a = a
        ...
        self.m = m

def listener(data: Data, b1: int):
    ...use data.a, ... data.m based on value of b1

pub.subscribe(listener, 'A')

...
pub.sendMessage('A', b1=b1,       data=Data(b=..., f=...))
pub.sendMessage('A', b1=other_b1, data=Data(c=..., k=..., n=...))

And here is another, which I find simpler and more expressive:

def listener1(b, f, **kwargs): # listener for b1=1
    ...

def listener2(c, n, k, **kwargs): # listener for b1=2
    ...

pub.subscribe(listener1, 'A')
pub.subscribe(listener2, 'A')

# when b1=1:
pub.sendMessage('A', b=..., f=...)

# when b1=2:
pub.sendMessage('A', c=..., k=..., n=...)

If you could let us know if either of these approaches works for you.

[teharris1]

First, thank you for reaching back out.

Yes, either of those options are very viable solutions. But they defeat the purpose of the original question.

What I am trying to figure out is a way to allow a sender to always send messages the exact same way and route to a command object that will take the appropriate action. The sender has as little implementation logic as possible.

What you are describing above is esetially to have the command object (or handler object depending on the implementation) manage the logic below the highest level. This certainly will work, no doubt.

What i am hoping to acomplish is to simplify the command object logic (and have fewer command objects) by allowing the publisher to route messages to specific command objects rather than have a top level command object that routes messages to a specific command obect.

I can accomplish this, as you described on gitter, by having mutiple subscriptions for all of the permutations. This is also a viable option but requires a lot of subscriptions. I was hoping to take advantage of the subtopic concept in your implementation (which most pub/sub implementations don't have, i.e. pydispatcher). So I like your module and feel the idea of allowing a generic subscription in the middle of the topic tree is not breaking your module's design but rather enhancing it to be more flexible.

I also like that your implementation does not meaningfully degrade the speed of pub/sub with the subtopic concept and see that my request would potentially have a negative impact on speed (i.e. when looking for 'a1-b*-c1' you will have to look for 'c1' in 'b1', 'b2' and 'b3' which may only produce 1 result.) However, this impact would only be realized if the use of the module values flexiblity over speed as I do.

If this is not in scope of your module I understand and have no hard feelings. Thanks for your help.

[schollii]

It would add considerable complexity to the sendMessage implementation, which currently does not track "topics down below". Plus there is yet another approach: create a wrapper for your star listeners:

class StarListener:
    def __init__(self, listener, topicFilter):
        self.__listener = listener
        self.__topicFilter = topicFilter.split('.')
        pub.subscribe(listener, pub.ALL_TOPICS)
        self.__topicCache = []

    def __call__(self, topic=pub.AUTO_TOPIC, **kwargs):
        if self.__checkMatch(topic):
            self.__listener(**kwargs)

    def __checkMatch(self, topic):
        if topic.getName() in self.__topicCache:
            return True

        actualTopicName = topic.getNameTuple()
        # actual: [a, b, c, d, e]
        # filter:   [a, *, c, *, e]
        for actualName, filterName in zip(actualTopicName, self.__topicFilter):
            if filterName == actualName:
                continue
            if filterName == '*':
                continue
            if filterName.endswith('*') and actualName.startswith(filter[:-1]):
                continue
            return False

        self.__topicCache.add(topic.getName())
        return True

# test:

def yourListener(arg1, arg2):
    print(arg1, arg2)

listener = StarListener(yourListener, 'a.*.c.de*.f')
pub.subscribe(listener)
pub.sendMessage('a.b1.c.dea.f', ...) # yes
pub.sendMessage('a.b2.c.dea.f', ...) # yes
pub.sendMessage('a.b1.c.dda.f', ...) # no

In the above simple implementation,

• the listener wrapper subscribes to the root (ALL) topics, because you don't want a topic a,b,c to match "d,f,*, a, b,c". This could be improved somewhat since the portion before the first star is (I think) adequate, thus saving some cpu.
• listeners are weak-referenced in pypubsub so you must keep a strong ref to listener. But this could (testing needed) prevent in some cases listener from being garbage collected (I'm thinking in the case where yourListener is a method, and so listener is a member of the instance of that method, thus leading to a circular reference which could prevent garbage collection). For that, you could store weak ref to yourListener in StarListener.
• The above design lends itself to more sophisticated filtering (just add if ... continue blocks), perf improvements.
• You should make the call have as specific kwargs as possible. For example if you know that all listeners want abc and dfg, and then more args based on topic, then use __call__(self, abc, dfg, **kwargs) as pubsub does some checks for you, helps with debugging events.
• Subtopics will be included, but this could be changed, e.g. if you want exact match then add a rule if len(self.topicFilter) != len(actualTopicName): return False
• You could easily cache the topics matched so you don't have to go throught the set of rules more than once for a given topic (this is shown, but not essential)
• You loose the automated message data specification checking feature of pypubsub.

[teharris1]

That is a really good idea. I was also able to breakdown two of the fields into unique combinations and found that the permutations are smaller than I thought (i.e. hundreds, not thousands). This will allow me to create a dict of these combinations and turn them into a single topic, which meaningfully simplified my problem. I am comfortable now using your base library for the issue without the original request.

Thanks so much for all your help.

[schollii]

That's great to hear!

Some day I'd love to understand better what you are using this for. Could be nice to put something in https://pypubsub.readthedocs.io/en/v4.0.3/about.html#pypubsub-users, I haven't updated it in a while.

I'm closing this now.