Subscribe from another Script

[matteopantano]

Hi,

I am trying to use pypubsub in order to subscribe to a topic from another python script. Is this feasible?

My setup is: One python script continuously looping with pub.sendMessage('rootTopic', arg1=id, arg2=tvec, arg3=rvec) and another script as follows.

from pubsub import pub

def listener0(arg1=None, arg2=None, arg3=None): 
    print("listener0: ", arg1, arg2, arg3)

if __name__ == '__main__':

    while True:
        pubListener, first = pub.subscribe(listener0, 'rootTopic')
        print (pubListener.name())
        assert first == True
        assert pubListener.isDead() == False   

I cannot see any message in the console apart from:

listener0_3800
listener0_3800
Traceback (most recent call last):
  File "main.py", line 12, in <module>
    assert first == True
AssertionError

So seems that first is true for the first is true for the first two instances and then is false.

PS: Before splitting the code in two parts I have tried the sub/pub on the same script and it works

Let me know,

Matteo

[FractalWire]

Hey Matteo,

I'd be surprised if you could send messages across different processes with this library. To communicate between processes, the usual method is to exchange data via socket or via a shared file. However, you can't exchange python object, only bytes can be exchanged that way.

You can, however, leverage the power of threading to run your two scripts simultaneously from a master python script.

Here is a simple example:

script1.py:

from pubsub import pub
import time

def my_listener(arg1=None, arg2=None, arg3=None):
    print(f"mylistener received:", arg1, arg2, arg3)

pub.subscribe(my_listener, 'my_topic')

def infinite_listen():
    while True:
        time.sleep(0.5)

script2.py

from pubsub import pub
import time
import random

def infinite_send():
    arg_choice = [None, 1, "bla", (1, 2, 3)]
    while True:
        arg1, arg2, arg3 = [random.choice(arg_choice) for _ in range(3)]
        pub.sendMessage('my_topic', arg1=arg1, arg2=arg2, arg3=arg3)
        time.sleep(1)

main.py

import threading
import script1
import script2

th1 = threading.Thread(target=script1.infinite_listen)
th2 = threading.Thread(target=script2.infinite_send)

th1.start()
print("th1 started")
th2.start()
print("th2 started")

print("waiting for th1 and th2 to finish")
th1.join()
th2.join()

print("bye-bye!")

In this case, it would probably be cleaner to define the MDS of my_topic in main.py, but I don't really know to do that, as I just started with this library myself.

To answer why the second call to subscribe returned False in your case, you can look at the doc :

Returns (pubsub.core.Listener, success), where success is False if listener was already subscribed

[schollii]

Sorry but pypubsub was designed for intra-process events, not cross-process.

If the approach described by @FractalWire isn't suitable, you might want to try the multiprocessing module: if you start several processes via this module, you can make them share memory. You could in principle share pubsub but I have never tried it. At very least you would be able to pass message data. However this would only work if all processes are on same machine.

If you need distributed messaging, then there are many options like kafka, zeromq, mqtt, google pub/sub, etc. But they all require a server process (a hub of sorts), which can be more or less feasible depending on your needs.

There's also google protobuf for peer-to-peer comms.

[matteopantano]

Great thanks @schollii @FractalWire for the quick replies! I will start working on the integration and let you know about the chosen method!