New Publishers

[paddymccrudden]

As noted at pubsub.readthedocs, the expression

from pubsub import pub

... creates a “default” instance of pubsub.core.Publisher and binds several local functions to some of its methods and those of the pubsub.core.TopicManager instance that it contains. However, an application may create as many independent instances of Publisher as required (for instance, one in each thread; with a custom queue to mediate message transfer between > threads)."

As noted in this quote, the pub module does quite at bit of work to expose methods and attributes, and it is not immediately obvious how to build a new Publisher() object with the correct attributes exposed. I'd be really interested in knowing if there is a standard way to do this. Alternatively, perhaps we could add a new module that provides this.

[schollii]

Why do you need this, what are you trying to do?

[paddymccrudden]

I am building a system for financial modelling. The objects include typical financial entities such as portfolios, accounts, orders, executions, order management systems, exchanges etc. An event-based program architecture works well for this. However, there are (at least) two levels at which the messages are appropriate. For example, there are "public" messages that are shared between entities such as Portfolios and Exchanges, and there are "private" messages that occur within those entities.

I believe it would be reasonable to simply have a base topic such as "public" for the general messages, and another topic for within the entities. However, with a view to multi-threading, and distributed calculations, it would seem more natural to have multiple Publish() objects.

(Thanks for the speedy response!)

[schollii]

Thanks for the additional info. Plaease clarify:

• how you think multi-threading affects the topic hierarchy, or vice versa
• distributed how: on a network? does that mean you will have multiple Python processes (one for account, another for orders, etc, for example), packaging messages and sending them over a network? PyPubSub does not do the latter, the network aspect is something you would have to do yourself or use something like mqtt, kafka, etc. PyPubSub is for components within the same process.

[paddymccrudden]

Thanks for you prompt response.

• In multi-threading - that is a good question, and I admit I hadn't thought too hard about it. If the topics are the same, the value of the multiple publishers is diminished.
• With regard to the network issue: I have chosen to use pypubsub rather than a more industrial messaging service because I really like the passing of objects in memory, rather than serialising them first. There are significant speed issues if I have to serialise (pickle) before messaging.

I am comfortable right now with one message queue for my application, and so you may close this issue. Thanks again!

[paddymccrudden]

I might just add one more comment. When running unit test, the pub module is imported, and the pub object persists between tests when using pytest. This means that if you modify the topics, for example, in one test, then those modifications will persist into the next test. This makes it quite challenging to have effective unit tests. I would argue that this is a problem with pytest, not pubsub having only one import, but creating new pub objects would be a way to avoid this problem.

[schollii]

Excellent, glad to hear that!

[paddymccrudden]

Hi @schollii. I'm pleased to report that my program works well with pypubsub, and it is very convinient. However, in order to implement my machine learning algorithm with Hogwild! (https://github.com/pytorch/examples/tree/master/mnist_hogwild) I need to run it in a multi-thread environment so that I can run in parallel. This re-opens the multiple publisher issue for me. Again - any thoughts on creating a new publisher for each thread?

[schollii]

There are many different aspects to this, probably easier to do in a chat forum like gitter. Let me know if you would like to discuss.

[schollii]

I did not mean to close this yet :/

[schollii]

In any case the main considerations are:

• if your listeners "live" in the same thread as your senders, then just create one Publisher per thread; just ensure that sendMessage() is called from the same thread as listener (listener can check this by doing assert threading.current_thread() is assign_thread where assigned_thread is something you define at beginning of your app run
• otherwise, you need to either
  • synchronize access to resources; in event based system, deadlocks are likely I would say
  • OR use a thread-specific queueing mechanism (basically to ensure that a listener receives a message on its assigned thread)

Note that "assigned thread" is a design construct; python does not assign functions to threads, it's up to you to determine whether some functions always run in same thread etc.

Feel free to re-open.